A walk past the Palace of Fine Arts rotunda during Cal Hacks 13, rebuilt by my pipeline from one Ray-Ban Meta clip: the place is a Gaussian splat, and the people crossing it are moving point ghosts.
Record a moment on Ray-Ban Meta glasses, then step back inside it with friends. memos rebuilds the clip as a 3D scene that moves over time: the room as a Gaussian splat at real scale, the people as moving ghosts, the small things as objects you can grab and throw.
Everyone relives it on one shared clock, on a Quest 3S, a phone or a laptop. Pause, throw something, and the edit becomes a new take; the original recording stays as it was. We built it at Cal Hacks 13 as a team of three: I built the pipeline from video to scene and brought our parts together into one app.
What I did
Built the pipeline that turns one clip into a scene: NVIDIA ViPE for the camera path and depth, a level metric world frame, a gsplat room, people as moving ghosts and props you can throw.
Ran it on all nine Ray-Ban Meta clips we recorded at the event, indoors and out, and scored every scene against its own video.
Brought our three parts into one app: the adapter that ties the viewer to SpacetimeDB, the party panel and avatars, and the GPU worker that publishes each scene.
Built the ways to relive a memory: follow the recorded walk, guided exploration, the wearer’s view, and VR grabbing that freezes time.
Ran a true 4D Gaussian splatting track alongside: 30.5 dB PSNR on held-out frames, with sharp moving people.
Figures
Made once
Relived together, on any device
every device downloads the scene once
Ship the scene, sync the state: a GPU builds each memory once, every device renders it itself, and only the clock, takes, grabs and poses travel live.Hover over a box, or tab to it, for how that step works.Tap a box for how that step works.Ray-Ban Meta. The glasses record a short clip with sound. We recorded nine at the event, from 7 to 60 seconds, indoors and out.Phone app. An iPhone app on the Meta Wearables Device Access Toolkit pulls the clip off the glasses and uploads it through a resumable queue.Vision pipeline. I built this pipeline. A GPU worker on Runpod runs it on each upload and publishes the scene when the memory is ready.Camera and depth. NVIDIA ViPE recovers the camera path, the lens, metric depth for every pixel, and masks of the people and objects.Level, metric world. The whole clip is placed in one level world frame at real scale, with the floor at zero.Room, people, props. A gsplat room trained with depth supervision, people as moving point ghosts, and small props carved out as grabbable splats with colliders.Scene files.memory.json carries the room splat, the props, the people, the camera path and the audio. Each device downloads it once.One web viewer. The same web app runs on a Quest 3S, phones and desktops. I built its follow and guided modes, the wearer’s view and VR grabbing that freezes time.SpacetimeDB. One shared clock, the takes, who holds what and every player’s pose. Pause, throw something, and the edit becomes a new take; the recording stays as it was.
