Robotics Engineering Intern · Stealth robotics startup, San Francisco · Jul – Sep 2026
At an early-stage robotics startup I ran a bimanual YAM bench on my own and built the loop that turns it into a learning system: teleoperate with leader arms, record clean demonstrations, train ACT and Multi-Task DiT policies, roll them out on the real arms, and let a human take over to record corrections. Alongside it I built an Isaac Lab replica of the cell that generates its own demonstrations. This page walks around that loop, including the negative results that taught me the most.
One of the 50 teleoperated demonstrations in the public dataset, all three cameras in sync: the overhead D435 on the left, the two wrist D405s on the right. The task: stack the two purple cubes on the orange one.
At a glance
The loop
Each part of the cell exists to feed the next. The sections below follow the loop in order; a simulated replica of the cell (section 5) adds demonstrations from outside it.
01 · The cell
The bench pairs two YAM Pro follower arms with two leader arms. Each leader ends in a teaching handle: a trigger that drives the gripper and two buttons that I later wired into recording and takeover. One RealSense D435 looks down at the table, and a D405 rides on each wrist.
LeRobot had no YAM support, so I wrote it: robot and teleoperator plugins for a single arm, a bimanual pair composed from two of them with a failure-safe disconnect and a rollback when connecting fails, and layered launch configs so a session starts with one command. When the stack moved to upstream LeRobot, a renamed module broke every YAM import in CI; that fix was one of four pull requests of mine merged into the company's repositories.
D435 · overhead cameraLeft followerD405 · wrist cameraRight followerOne week after bring-up, the cell ran its first policy, and it wasn't mine. I deployed MolmoAct 2, an open vision-language-action model with a checkpoint for bimanual YAM, behind a camera server and an inference server that returns a 30-step action chunk in about 230 ms. A web panel starts, stops and labels each rollout.
Once the gripper zeroing and the start pose were fixed, it picked up a vegetable and packed it into the crate zero-shot: a useful baseline before training anything on my own data.
02 · Demonstrations
I recorded the demonstrations with the leader arms, using the handle buttons to save or redo an episode without reaching for the keyboard. I checked every episode by hand and pruned the failures rather than keep them as noise. The result is a three-camera cube-stacking dataset at 30 fps, published on Hugging Face.
Streaming encoding writes each episode while it is recorded, so saving is instant. When a wrist camera crashed mid-session four times, no recorded data was lost.
50
episodes kept of 52 recorded; the 2 failures were pruned
44,699
frames, about 25 minutes at 30 fps
3
synchronised cameras: overhead D435, wrist D405 × 2
99.8 %
of frames held at 30 fps during recording (1,797 of 1,800)
03 · Policies & real-time chunking
ACT and Multi-Task DiT both predict a chunk of future actions at a time. ACT is fast enough to run inside the 30 Hz control loop. Multi-Task DiT is not: run synchronously, the arm stalls while the next chunk is generated and then jumps to it. On the robot, that jump knocked the cube away.
Real-time chunking (RTC) generates the next chunk while the current one is still executing, and guides it to agree with the actions already committed. I ported RTC to Multi-Task DiT; on the robot, the same checkpoint went from knocking the cube over to completing the stack. Every checkpoint also had to pass an offline gate first: replay training frames through the full inference pipeline and keep the arm error under 3° before it could move the robot.
Error at each chunk hand-over, in degrees
Public checkpoint · 35 episodes · 1,474 hand-overs · lower is smoother
| Measure | Synchronous | Real-time chunking |
|---|---|---|
| Seam · mean | 1.82 | 0.73 |
| Seam · p95 | 4.35 | 1.42 |
| Jump · mean | 1.87 | 0.91 |
| Jump · p95 | 4.43 | 2.11 |
Upstream, as open pull requests to Hugging Face LeRobot: RTC for Multi-Task DiT (#4394) and RTC observation history at policy ticks (#4638). The benchmark runs on a public checkpoint so anyone can reproduce it.
04 · Human-in-the-loop
Corrections are the fastest way to fix where a policy fails, but grabbing a moving arm is jarring when the leader sits somewhere else. In my takeover mode the leader arms shadow the followers at soft gains while the policy drives, so the operator's hands are already in the right pose. One press of the yellow handle button hands control to the human, another hands it back, and only the corrected stretch is saved as a new episode.
I validated it on the hardware end to end, through a closed loop of three corrections. One gap remains: the handle's trigger is a passive spring with an encoder, so it cannot follow the follower's gripper while the policy drives. If the policy is holding a cube at the moment of takeover, the gripper jumps to wherever the trigger happens to be. The fix I designed waits until the trigger matches the gripper before switching.
05 · Simulation
To see whether simulation could stretch fifty real demonstrations, I rebuilt the cell in Isaac Lab: the arms from their MJCF models, the table and fixtures from measured geometry, and the cameras from the real intrinsics, aligned by overlaying renders on real frames until they matched. Drag across the image to compare.

⇆REALSIMThe expert moves at human pace: each joint's speed cap is the 95th percentile measured from fifty real teleop demos, with cosine easing in and out, and compensating the grasp offset lifted its success rate from 65 % to 78 %. Sixteen environments run in parallel with tiled cameras, and every episode randomises lighting, colour temperature, camera pose and surface tint. The export matches the real dataset's LeRobot schema field for field.
16
environments in parallel on one GPU
8×
throughput, about 1,600 attempts an hour
3,014
successful episodes in 2 hours on one RTX 5090
1.34 M
frames exported across 1,642 episodes
06 · What worked, what didn't
Real-time chunking was a clear win.
Run synchronously, Multi-Task DiT stalled the 30 Hz loop at every chunk, about 85 slow-loop warnings per session. With RTC the stalls were gone and the same checkpoint finished the task.
Fewer cameras worked better.
Dropping the idle right-wrist camera cut offline error by 14 % for ACT and 27 % for Multi-Task DiT, made grasps decisive on the robot, saved a third of the vision compute, and removed the camera most prone to cable interference.
Naive sim co-training hurt, for textbook reasons.
Mixing 48 clean simulated episodes 1:1 with real data raised the error on real frames (ACT 0.84° → 1.13°, Multi-Task DiT 2.10° → 2.41°), and the robot behaved worse. The setup hit four known anti-patterns: far too little sim data, the wrong mixing ratio, no domain randomisation, and shared normalisation statistics. The 3,014-episode randomised set was built to fix exactly those.
The last two centimetres are the bottleneck.
Every policy, and even the scripted expert at a 12 mm tolerance, grasped reliably and then missed the placement. That is the case for targeted corrections rather than more full demonstrations.
Hardware failures look like policy failures.
A wrist camera that dropped off USB whenever the arm moved (interference from motor power lines), a handle encoder that could freeze at exactly 1.000, motors that refused to reconnect within 15 s of the last session: each looked like a model problem at first, and each got a probe and a written procedure.