Data · Post 3 of 6

Recording teleoperated datasets with two cameras

A top camera and a wrist camera, a leader arm, and lerobot-record.

With both arms calibrated, we recorded demonstrations by teleoperation: a person moves the leader, the follower mirrors it, and LeRobot logs joint states, actions and camera video at every step into a LeRobotDataset v3.

The task and the scene

Our task instruction is “put the blue box in a bowl”. Two cameras watch the scene: a fixed top camera for the overall layout and a wrist camera mounted on the gripper.

See the LeRobot camera guide for how OpenCV cameras are discovered and configured.

The recording command

lerobot-record \
    --robot.type=so101_follower \
    --robot.port=/dev/ttyACM1 \
    --robot.id=my_awesome_follower_arm \
    --robot.cameras="{ top: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30},
                       wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 15} }" \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM0 \
    --teleop.id=my_awesome_leader_arm \
    --dataset.repo_id=<your-username>/so101_dataset_test \
    --dataset.num_episodes=5 \
    --dataset.episode_time_s=40 \
    --dataset.single_task="put the blue box in a bowl" \
    --dataset.streaming_encoding=true \
    --display_data=true

Two choices are worth explaining:

• Different frame rates per camera. The wrist camera runs at 15 fps to reduce USB bandwidth and encoding load while the top camera keeps 30 fps, which is also the dataset's control rate.
• streaming_encoding=true encodes video while recording rather than after each episode. See LeRobot's streaming video encoding page for the trade-offs, including how visualization lag can accumulate.

What we collected

Across our test sessions on 17–19 July, 17 runs of the script created a dataset folder, and 6 of them saved at least one episode. In total that is 7 episodes and about 8,700 frames at 30 fps. The remaining runs ended before an episode was saved, which is typical while working out ports, cameras and timing. These are debugging-scale datasets, not yet enough to train a good policy.

Every episode stores a 6-dimensional observation.state and action (shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper, each .pos) plus the two video streams observation.images.top and observation.images.wrist.

Next steps in LeRobot's workflow are to visualize and replay episodes and then train a policy such as ACT or SmolVLA.