The benchmark layout, rendered from the printed tag’s pose and blended over the live RealSense view.
A printed AprilTag tells the camera where it is, and my ROS 2 tool draws the benchmark layout over live video.
Whoever sets up a benchmark scene sees, on the live image, where each real object has to go. I built it as sole author.
What I did
Built the ROS 2 tool end to end: it finds a printed AprilTag, recovers the camera pose and blends the benchmark layout over live video.
Made the render line up with the real camera: an off-axis OpenGL frustum built from the intrinsics K.
Wrote a mesh converter that splits multi-material meshes into per-material URDFs and verifies the geometry is preserved.
Designed a hand-editable YAML scene format and moved capture from ROS topics to direct pyrealsense2.
Figures
K
From a printed tag to a see-through guide: the tag gives the camera’s pose, and a camera matched to the RealSense intrinsics renders the layout over live video.Hover over a box, or tab to it, for how that step works.Tap a box for how that step works.RealSense frame. Color frames come straight from the camera through pyrealsense2 (I moved capture off ROS topics), with the intrinsics K.Tag → camera pose. pupil-apriltags finds the printed tag on the table, and the tag gives the camera’s pose relative to it.Matching camera. An off-axis OpenGL frustum built from K, with the OpenCV-to-OpenGL frame conversion, keeps the render aligned when the principal point is off-center.Benchmark scene. Placements come from ManipulationNet scenes or my hand-editable YAML; my converter splits multi-material meshes into per-material URDFs for PyBullet.Render the layout. PyBullet renders the benchmark objects from the tag’s pose through the matching camera.Blend over live video. The render is blended over the live frame at 0.6 alpha, so whoever sets up the scene sees where each real object goes.
A second benchmark layout on the same table: the sugar box, mustard bottle and ball are renders, placed from the tag’s pose over the live frame.
Built with
ROS 2 ·RealSense (pyrealsense2) ·AprilTag (tag36h11) ·PyBullet ·OpenGL camera math
Credits
Builds on ManipulationNet: its benchmark scenes and the tag-to-render math I re-implemented as a standalone tool. YCB object meshes (CC BY 4.0). With Prof. Wenzhen Yuan.
