RoboMove

Community-driven robot training through social interaction.

RoboMove is an interactive robot playground built on the CodecFlow stack. It livestreams robots in simulation environments and lets people control them through social commands.

The project uses Fabric and optr to connect social input, simulation, policy inference, video streaming, and reply generation into one loop.

How it works

1. A user tweets a command and tags @RoboMove.
2. The system interprets the instruction.
3. The robot attempts the action in simulation.
4. A short video reply shows the result.
5. The run logs data that can improve future policy behavior.

Technical architecture

MuJoCo physics runs in a dedicated subprocess, separate from streaming and HTTP layers. The main loop steps physics, renders a frame, pushes the frame to the GStreamer pipeline, and records the run.

Frame rendering and streaming run on the same tick. Physics and rendering are decoupled through a frame queue, which keeps the simulation loop from blocking on video output.

GStreamer handles video rendering and output, including RTMP for broadcast and shared memory for low-latency local consumption.

Cloudflare WebRTC SFU streams real-time simulation state data to connected clients, including joint positions, velocities, and sensor readings.

Actions arrive through HTTP requests, enter the callback manager queue with duration and priority, and execute inside the simulation loop on the next available tick.

Robot model and policies

The first robot model is a Unitree G1 humanoid with 23 degrees of freedom: 6 left leg, 6 right leg, 3 torso, 4 left arm, and 4 right arm.

Joint stiffness, damping, default positions, and torque limits are defined per joint in the scene XML.

• Balance mode: an ONNX policy uses IMU data, velocity commands, joint angles, joint velocities, last action, and phase signal. It runs inference every 10 simulation steps and applies output through PD control.
• Motion tracking mode: a PyTorch policy loads reference motion trajectories such as walk, dance, kick, and squat, then maps target joint positions to motor torques at each timestep.

optr graph

RoboMove is structured as a real-time optr graph:

social listener -> command parser -> sim worker -> frame renderer -> stream pusher -> reply poster

Each stage can run as an optr operator. Inference-heavy steps can dispatch to Fabric GPU workers.

What it proves

• Fabric at work: GPU compute for physics-heavy simulation and policy inference, provisioned on demand per session.
• optr runtime: a multi-node real-time graph that turns social input into physical output across the see-think-act loop.
• MuJoCo on Fabric: full humanoid physics with contact-rich locomotion running cloud-side and streamed back through GStreamer and WebRTC.
• Community as data source: social commands create training episodes. Each run can log to a dataset, letting the policy improve from crowd interaction.

Stack

• MuJoCo: physics engine with XML scenes and per-joint tuning.
• GStreamer: video rendering and RTMP or shared-memory output.
• Cloudflare WebRTC SFU: real-time simulation state streaming.
• ONNX Runtime and PyTorch: balance and motion-tracking policies.
• Fabric: GPU workers for simulation and inference.
• optr: orchestration runtime.