Robotics Engineering Intern · Stealth robotics startup, San Francisco · Jul – Sep 2026

Building the Learning Loop for a Bimanual YAM Cell

At an early-stage robotics startup I ran a bimanual YAM bench on my own and built the loop that turns it into a learning system: teleoperate with leader arms, record clean demonstrations, train ACT and Multi-Task DiT policies, roll them out on the real arms, and let a human take over to record corrections. Alongside it I built an Isaac Lab replica of the cell that generates its own demonstrations. This page walks around that loop, including the negative results that taught me the most.

2× YAM Pro + 2× leader armsRealSense D435 + 2× D405LeRobotACT · Multi-Task DiTReal-time chunkingHuman takeoverIsaac LabRTX 5090
ENV · D435WRIST L · D405WRIST R · D405
TELEOP DEMO · EP 031 · 1×FRAME 000 / 526

One of the 50 teleoperated demonstrations in the public dataset, all three cameras in sync: the overhead D435 on the left, the two wrist D405s on the right. The task: stack the two purple cubes on the orange one.

At a glance

HardwareTwo YAM Pro follower arms and two leader arms with teaching handles, an overhead RealSense D435 and two wrist D405s, four CAN adapters, and one RTX 5090 workstation.
SoftwareUpstream LeRobot with my YAM robot and teleoperator plugins, ACT and Multi-Task DiT policies, and Isaac Lab for the simulated replica.
OutputA public 50-episode dataset, plugins merged into the company's repositories, two open pull requests to Hugging Face LeRobot, and 3,014 simulated demonstrations generated in two hours.
InsightThe last two centimetres decide everything. Every policy, and even the scripted expert in simulation, grasped reliably and then missed the placement, which is exactly where human corrections pay for themselves.
On this page1The cell2Data3Policies & RTC4Human takeover5Sim data factory6Lessons

The loop

Collect, train, evaluate, intervene, and collect again

Each part of the cell exists to feed the next. The sections below follow the loop in order; a simulated replica of the cell (section 5) adds demonstrations from outside it.

01 · The cell

Two arms, two leaders, three cameras

The bench pairs two YAM Pro follower arms with two leader arms. Each leader ends in a teaching handle: a trigger that drives the gripper and two buttons that I later wired into recording and takeover. One RealSense D435 looks down at the table, and a D405 rides on each wrist.

LeRobot had no YAM support, so I wrote it: robot and teleoperator plugins for a single arm, a bimanual pair composed from two of them with a failure-safe disconnect and a rollback when connecting fails, and layered launch configs so a session starts with one command. When the stack moved to upstream LeRobot, a renamed module broke every YAM import in CI; that fix was one of four pull requests of mine merged into the company's repositories.

The cell from the side: the left follower arm grasping a cube, the overhead camera on its pole, and the right follower arm parked
The cell from the side, mid-rollout: the overhead D435 on its pole, the left follower stacking, the right one parked. The leader arms sit out of frame.

First light: a zero-shot VLA

AUTONOMOUS · MOLMOACT 2 · ZERO-SHOT · 1×
MolmoAct 2 packing a vegetable into the crate with no training on this cell. Phone footage at 1×.

One week after bring-up, the cell ran its first policy, and it wasn't mine. I deployed MolmoAct 2, an open vision-language-action model with a checkpoint for bimanual YAM, behind a camera server and an inference server that returns a 30-step action chunk in about 230 ms. A web panel starts, stops and labels each rollout.

Once the gripper zeroing and the start pose were fixed, it picked up a vegetable and packed it into the crate zero-shot: a useful baseline before training anything on my own data.

02 · Demonstrations

Fifty clean episodes, released publicly

I recorded the demonstrations with the leader arms, using the handle buttons to save or redo an episode without reaching for the keyboard. I checked every episode by hand and pruned the failures rather than keep them as noise. The result is a three-camera cube-stacking dataset at 30 fps, published on Hugging Face.

Streaming encoding writes each episode while it is recorded, so saving is instant. When a wrist camera crashed mid-session four times, no recorded data was lost.

TELEOP · LEADER → FOLLOWERS · 1×
Bring-up week: the followers on the left mirror the leader arms I am moving on the right.

50

episodes kept of 52 recorded; the 2 failures were pruned

44,699

frames, about 25 minutes at 30 fps

3

synchronised cameras: overhead D435, wrist D405 × 2

99.8 %

of frames held at 30 fps during recording (1,797 of 1,800)

03 · Policies & real-time chunking

No pause between action chunks

ACT and Multi-Task DiT both predict a chunk of future actions at a time. ACT is fast enough to run inside the 30 Hz control loop. Multi-Task DiT is not: run synchronously, the arm stalls while the next chunk is generated and then jumps to it. On the robot, that jump knocked the cube away.

Real-time chunking (RTC) generates the next chunk while the current one is still executing, and guides it to agree with the actions already committed. I ported RTC to Multi-Task DiT; on the robot, the same checkpoint went from knocking the cube over to completing the stack. Every checkpoint also had to pass an offline gate first: replay training frames through the full inference pipeline and keep the arm error under 3° before it could move the robot.

AUTONOMOUS · MTDiT + RTC · 1×
Multi-Task DiT with real-time chunking, driving the left arm on its own. Phone footage at 1×.

Error at each chunk hand-over, in degrees

Public checkpoint · 35 episodes · 1,474 hand-overs · lower is smoother

SynchronousReal-time chunking
Seam · mean
1.82°0.73°
Seam · p95
4.35°1.42°
Jump · mean
1.87°0.91°
Jump · p95
4.43°2.11°
Chunk hand-over error in degrees, synchronous versus real-time chunking
MeasureSynchronousReal-time chunking
Seam · mean1.820.73
Seam · p954.351.42
Jump · mean1.870.91
Jump · p954.432.11
Seam: how far the new chunk disagrees with the old one at the moment it takes over. Jump: the change between the last action executed from the old chunk and the first from the new one. Measured over the five arm joints.

Upstream, as open pull requests to Hugging Face LeRobot: RTC for Multi-Task DiT (#4394) and RTC observation history at policy ticks (#4638). The benchmark runs on a public checkpoint so anyone can reproduce it.

04 · Human-in-the-loop

Take over mid-rollout, keep only the fix

Corrections are the fastest way to fix where a policy fails, but grabbing a moving arm is jarring when the leader sits somewhere else. In my takeover mode the leader arms shadow the followers at soft gains while the policy drives, so the operator's hands are already in the right pose. One press of the yellow handle button hands control to the human, another hands it back, and only the corrected stretch is saved as a new episode.

I validated it on the hardware end to end, through a closed loop of three corrections. One gap remains: the handle's trigger is a passive spring with an encoder, so it cannot follow the follower's gripper while the policy drives. If the policy is holding a cube at the moment of takeover, the gripper jumps to wherever the trigger happens to be. The fix I designed waits until the trigger matches the gripper before switching.

05 · Simulation

A digital replica that writes its own demonstrations

To see whether simulation could stretch fifty real demonstrations, I rebuilt the cell in Isaac Lab: the arms from their MJCF models, the table and fixtures from measured geometry, and the cameras from the real intrinsics, aligned by overlaying renders on real frames until they matched. Drag across the image to compare.

Overhead view, real cameraOverhead view, simulated replicaREALSIM
Overhead camera: the real cell (left of the divider) and its Isaac Lab replica.
SIM · SCRIPTED EXPERT · 1×
The scripted expert stacking cubes in the replica, seen from the simulated overhead camera.

The expert moves at human pace: each joint's speed cap is the 95th percentile measured from fifty real teleop demos, with cosine easing in and out, and compensating the grasp offset lifted its success rate from 65 % to 78 %. Sixteen environments run in parallel with tiled cameras, and every episode randomises lighting, colour temperature, camera pose and surface tint. The export matches the real dataset's LeRobot schema field for field.

16

environments in parallel on one GPU

8×

throughput, about 1,600 attempts an hour

3,014

successful episodes in 2 hours on one RTX 5090

1.34 M

frames exported across 1,642 episodes

06 · What worked, what didn't

Results and lessons

Real-time chunking was a clear win.

Run synchronously, Multi-Task DiT stalled the 30 Hz loop at every chunk, about 85 slow-loop warnings per session. With RTC the stalls were gone and the same checkpoint finished the task.

Fewer cameras worked better.

Dropping the idle right-wrist camera cut offline error by 14 % for ACT and 27 % for Multi-Task DiT, made grasps decisive on the robot, saved a third of the vision compute, and removed the camera most prone to cable interference.

Naive sim co-training hurt, for textbook reasons.

Mixing 48 clean simulated episodes 1:1 with real data raised the error on real frames (ACT 0.84° → 1.13°, Multi-Task DiT 2.10° → 2.41°), and the robot behaved worse. The setup hit four known anti-patterns: far too little sim data, the wrong mixing ratio, no domain randomisation, and shared normalisation statistics. The 3,014-episode randomised set was built to fix exactly those.

The last two centimetres are the bottleneck.

Every policy, and even the scripted expert at a 12 mm tolerance, grasped reliably and then missed the placement. That is the case for targeted corrections rather than more full demonstrations.

Hardware failures look like policy failures.

A wrist camera that dropped off USB whenever the arm moved (interference from motor power lines), a handle encoder that could freeze at exactly 1.000, motors that refused to reconnect within 15 s of the last session: each looked like a model problem at first, and each got a probe and a written procedure.

Dataset on Hugging FaceLeRobot #4394LeRobot #4638