The UR5e arm and gripper my policies ran on, at RoboTouch Lab.
I built a real-world imitation-learning pipeline on the UMI stack, on my own rig and data, and trained diffusion policies for a UR5e arm; the work won the URSA Best Research Poster Award (2026).
On the real arm I traced three failures from the logs and fixed them; after the fixes, policies I trained on the lab’s teleop datasets grasped an Allen key and turned a knob.
What I did
From handheld clips to trained policies
Collected 500+ demonstrations, handheld and in simulation, and built the data path that turns my GoPro clips into metric 3D trajectories and training episodes.
Tracked my recordings with ORB-SLAM3 + IMU and built a SLAM QA tool to check them: all 197 cylinder grasps tracked start to finish.
Trained diffusion policies for ≈180 GPU-hours on one RTX 5080, after porting the stack to Blackwell, and wrote an HDF5 → zarr converter with validation for the lab’s datasets.
Pretrained a visuo-tactile CLIP encoder (FreeTacMan method), fixed its scheduler and normalization bugs, and integrated it into the lab’s Diffusion Policy fork.
From logs to working rollouts
Wrote the ROS 2 node that runs the policies on the UR5e, smoothing 10 Hz plans into 60 Hz commands, and a step-mode harness that logs every input and predicted plan.
Fixed the gripper snapping shut (it opens wider than UMI’s convention) and the robot’s speed (actions were 0.3 s apart, not 0.1 s).
Measured the arm’s lag at 0.13–0.15 s, found a backward jerk at every plan hand-off and made each new plan start further ahead.
Figures
Data
Geometry
Learning
Robot
logsfixesreplays
How the pieces fit: my handheld demonstrations and the lab’s teleop datasets train diffusion policies, which run on the UR5e through my ROS 2 node; the robot’s logs drove the fixes.Hover over a box, or tab to it, for how that step works.Tap a box for how that step works.Handheld demonstrations. I recorded 487 clips in 5 sessions on my own UMI rig: a handheld gripper with a fisheye GoPro, its IMU and side mirrors.SLAM + tag mapping. UMI’s monocular-inertial ORB-SLAM3 tracks the camera; ArUco tags on the fingers give the gripper width, and a board on the table fixes the world frame.Metric 6-DoF trajectories. Each demo becomes an end-effector path in meters. My SLAM QA tool checks every session: all 197 cylinder grasps tracked start to finish; 338 training episodes in all.Lab teleop datasets. The lab’s teleop datasets for Allen-key grasping (50 episodes) and knob turning (65 episodes). I trained policies on both and ran them on the UR5e.HDF5 → zarr converter. My converter brings them into UMI’s training format: quaternion → axis-angle, gripper scaling, JPEG-XL streaming and a natural-sort fix, each checked.Diffusion policy. A CLIP ViT-B/16 encoder and a ConditionalUnet1D denoise 16-step action chunks: ≈180 GPU-hours on one RTX 5080, after I ported the stack to Blackwell.UR5e deployment. My ROS 2 node turns 10 Hz plans into a 60 Hz command stream (linear position, SLERP rotation). Policies I trained on the lab’s teleop datasets grasped an Allen key and turned a knob.Log-driven debugging. Every input and plan is logged. From the logs I fixed the gripper scale, the action spacing and the jerk at plan hand-offs, and measured the arm’s 0.13–0.15 s lag.
One demonstration as the gripper’s GoPro saw it: fisheye view, ArUco finger tags at the bottom.
The wrist camera during a knob-turning rollout of a policy I trained on the lab’s teleop dataset.
