Robot Learning Infrastructure · Updated Nov 2026
I built this Franka Research 3 manipulation cell at TASL almost entirely from scratch: the 80/20 bench, the real-time control stack, the cameras and gripper, a GELLO leader arm, and the data-collection and rollout pipeline on top. This page walks through the build in the order it happened, what the finished cell has produced since — 60 zero-shot rollouts of a generalist policy and the lab's 10-task demonstration dataset — and what I learned about where capable policies quietly break.
Autonomous rollouts of a π₀.₅-DROID vision-language-action policy, deployed zero-shot on the finished cell — language in, tabletop manipulation out. No teleoperation, no scripting.
At a glance
01 · The bench
Before any robot, there was a frame to build. The cell is an 80/20 aluminum-extrusion bench on casters — squared up, leveled, and bolted together by hand so the arm would have a rigid, repeatable base to mount to and the cameras would stay put between calibrations.



02 · Arm & control plane
With the bench up, the Franka Research 3 (firmware 5.9.2) went on top. The arm is driven by a separate real-time-kernel NUC that owns the FCI link at 1 kHz and exposes a small robot server to the GPU workstation over the robot network — first through franky, later through Polymetis (§3). The NUC has no GPU and does nothing else: it is the robot's real-time hands.
Getting here meant working through the parts of robotics that never make it into papers: firmware compatibility, the right libfranka version, motion-generation gotchas, and a controller that recovers cleanly instead of dropping the FCI session mid-run.
03 · The control stack
Exactly one process may speak libfranka to the arm, so choosing that layer is a real decision. The legacy option — DROID's Polymetis fork — was frozen on libfranka 0.13, out of spec for the FR3's 5.9 firmware (protocol 10 needs 0.18+), and would not even compile against our setup. So I first built the controller on franky: it tracks libfranka through 0.18, supports FR3 5.9.x officially, and installs with plain pip. Within a day the bench was running real 1 kHz control behind a small HTTP server.
Two things later pulled me back to Polymetis. The policy I wanted to deploy, π₀.₅-DROID, was trained on the DROID stack and expects its exact action and observation conventions, so running it faithfully — and comparably to published numbers — was easiest inside DROID's own container, which I rebuilt with libfranka swapped up to 0.18.1. And the lab's direction moved toward RLinf for real-world data collection and online RL, whose environment I gave a Polymetis controller backend. The result is a Docker image on the NUC that owns FCI and serves the robot over zerorpc; everything above it — teleop, data collection, policy inference — runs at 15 Hz on the workstation.
NUC1 (RT kernel, no GPU) FR3 (fw 5.9.2)
container droid-nuc-fr3:0.18.1 ▲
polymetis driver + libfranka 0.18.1 ── FCI 1 kHz ┘
zerorpc server :4242 ◄── workstation: RLinf env / openpi / dashboards @ 15 Hz
Robotiq 2F-85 driver (RS-485)
rule: exactly one libfranka client. The franky robot_server must be stopped
before the container takes FCI, or both hang. The launchers enforce it.So it wasn't “franky wrong, Polymetis right.” franky made the hardware controllable quickly when the legacy stack couldn't even build; Polymetis came back once the goal shifted from make the arm move to deploy this specific DROID-trained policy and plug into this specific RL framework. A control stack is chosen by what the rest of the pipeline already speaks, not by which library is best in the abstract.

04 · Perception & end-effector
The end-effector carries a Robotiq 2F-85 parallel gripper and a wrist-mounted ZED Mini; two ZED 2i cameras observe the workspace from fixed viewpoints. That mirrors DROID's observation layout — one exterior view plus one wrist view, each resized to 224×224 for the policy — so a DROID-trained checkpoint sees roughly what it was trained on.
This part was less about adding sensors and more about making all of them agree with the robot. Each camera had to be calibrated into a consistent frame so that image observations, gripper poses, and tabletop geometry referred to the same physical world. The residuals came out around 3 mm, limited more by marker detection than by the arm's repeatability. The recurring nuisance was physical: the wrist camera's USB link dropped under motion until it was reseated and the launchers learned to probe camera readiness before starting a run.
This layer quietly decides whether everything downstream is possible. If the cameras and the arm disagree about where the table, the objects, or the gripper are, even a strong policy fails for reasons that look like “bad intelligence” but are infrastructure errors.
05 · Teleoperation & data
To collect demonstrations, I built a GELLO leader arm — a low-cost, 3D-printed kinematic twin whose joints map directly onto the FR3, so a human can puppet the real arm one-to-one. It is the bridge between “the hardware works” and “the hardware can learn”: every teleoperated episode becomes training data.
Collection runs through RLinf's real-world environment with the Polymetis backend from §3. A joint-space wrapper turns the leader's joints into the robot's 8-D joint-velocity action at 15 Hz — the same vector π₀.₅-DROID predicts — and each episode is written straight into a LeRobot v2.1 dataset: two 224×224 views, joint state, per-step intervention flags, plus full-HD stereo recordings on the side so frames can be re-cropped later without re-collecting.
Around it I built what a one-person collection session actually needs: a browser dashboard that starts and stops runs, marks an episode as a success with one key, shows the live camera feeds, and browses datasets with per-episode playback; and a pair of launch scripts that bring the NUC controller up, gate on FCI activation, home the arm, and tear everything down in the right order afterwards.
250
episodes across 10 tabletop tasks in the lab's FR3 dataset, 66,463 frames, collected on this cell with the GELLO arm
15 Hz
8-D joint-velocity actions, recorded directly as the dataset action
2 × 224²
camera views per frame, plus full-HD SVO recordings for re-cropping
Pilot datasets (stack a cube, assemble a lamp) came off the pipeline on June 22, 2026; the 10-task dataset was collected by lab members over the following months.
06 · The cell, end to end
The finished cell is three machines with strict roles. The NUC is the real-time hands, the workstation is the brain and eyes, and the laptop only orchestrates. Teleop collection and policy evaluation share the same robot and camera infrastructure; only the process at the top changes.
LAPTOP ── ssh ──┬──────────────────────────────────┬────────────────────────────────┐
▼ ▼ │
DESKTOP TASL-1 (RTX 4090) NUC1 (RT kernel, no GPU) │
2× ZED 2i + ZED Mini (USB 3) container droid-nuc-fr3 │
RLinf real-world env ── zerorpc :4242 ──▶ polymetis + libfranka 0.18.1 │
├ GelloJointIntervention ◄── GELLO Robotiq driver │
├ CollectEpisode ──▶ LeRobot v2.1 + SVO │ FCI 1 kHz │
└ PolymetisController ▼ │
openpi serve_policy :8000 (eval) FR3 arm ◄── Robotiq 2F-85 │
collect dashboard :8004 / eval dashboard :8003 │
teleop.sh / teleop-stop.sh (bring-up, FCI gate, home, teardown) ◄────────────────┘Two constraints shape everything: FCI is exclusive, so exactly one controller process may exist and the launchers stop the other before starting; and the cameras and the robot socket are shared, so the collection and evaluation dashboards never run at the same time. Training never touches the bench — datasets go to the GPU cluster, checkpoints come back for evaluation.
07 · Zero-shot results
With the cell finished, I used it for a small in-the-wild study: deploying the π₀.₅-DROID generalist policy (openpi) zero-shot — no fine-tuning on our robot — across six tasks and 60 rollouts, 10 trials each, in June 2026. I went in expecting zero-shot transfer to real hardware to be brittle. The surprising part was how much did work: through a compliant control stack the arm moved smoothly and solved common pick-and-place tasks it had never seen on our exact setup.
The more interesting part was where it broke. Object grounding was usually correct — the arm went to the right thing — but precise grasping was the bottleneck: the policy hovered or froze above small objects, and a gentle push downward by a human was often enough for it to finish. Spoken ordering was not always respected, and a simple human handover failed every single time.
| Task | Success | Partial | What happened |
|---|---|---|---|
| Bread → plate | 10/10 | 0 | Reliable large-object pick-and-place. |
| Pen → plate | 10/10 | 0 | Reliable despite the pen being thinner than the bread. |
| Pen → plate, then bread → plate | 8/10 | 2 | Often completes both, but sometimes grabs the bread first. |
| All objects → basket | 10/10 | 0 | Robust when the target is large and the order is free. |
| Black pen → red cup | 3/10 | 7 | Grounds the pen, then hovers or freezes before a stable grasp. |
| Bread → outstretched hand | 0/10 | 0 | Does not treat a human hand as a valid target. |
Six tasks, 10 trials each. “Partial” means the policy grounded the right object and made progress but did not finish without help. The pattern that stuck with me: the control stack matters as much as the policy, capable models fail silently at the last few centimeters, and “it understood the instruction” is not the same as “it can do it.”
08 · Lessons