IIT Delhi — AI & Robotics Program · Project Submission

Gesture-Teleoperated SO-101 Arm

A single RGB webcam and real-time hand tracking replace the joystick: an open palm arms the robot, a moving hand drives its inverse kinematics, and a pinch closes the gripper — mirrored live into a MuJoCo digital twin and, on explicit opt-in, onto the physical arm's servos.

Authors: Sruthi Mohan & Reena Catherine J & Harishankar R

Platform: SO-101 · 5-DoF arm + gripper

Stack: ROS 2 Jazzy · MuJoCo · MediaPipe

Base project: SO101-Gesture-Teleoperation

servo axes (5 arm + gripper)

hand landmarks / frame

1 Mbps

servo bus baud rate

:5000

TCP teleop channel

≤12

IK iterations / tick

5 mm

IK convergence tolerance

01 — Overview

Project description, problem statement, outcome

Replacing a joystick or leader-arm rig with the operator's own hand — tracked through a webcam and turned into bounded, safety-gated motion commands for a real 6-axis arm.

1.1Problem statement

Teleoperating a multi-degree-of-freedom robot arm conventionally means buying and learning an extra piece of hardware — a joystick, a keyboard mapping, or a mechanically-identical leader arm. Each adds cost, setup time, and an indirect mapping between what the operator wants and what the arm does. This project asks a narrower question: can a single consumer RGB webcam, with no depth sensor and no wearable, supply enough signal to drive a 6-axis arm's inverse kinematics directly and safely, in both a physics simulation and on physical hardware?

1.2Defined outcome

The delivered system lets an operator stand in front of a laptop webcam and, with no controller in hand, do the following:

Show an open palm to arm gesture control; the arm stays inert until this explicit gesture.
Move the hand through space to drive the SO-101's end effector in Cartesian (task-space) mode, or drive each of the five arm joints individually in joint-space mode — switchable live with one key.
Pinch thumb to index finger to close the gripper; release the pinch shape or re-open the palm to open it.
Make a fist, at any time, to instantly hand control back — the single universal "let go" gesture.
Trigger a latched software emergency stop from the keyboard that requires a deliberate two-step reset (clear the stop, then re-present an open palm) before motion resumes.
Watch every command land in a MuJoCo digital twin first, and — only behind an explicit, separately-gated flag — mirror the same joint targets to the physical arm's Feetech servos.

Built on prior work

The gesture-recognition and Cartesian-mapping core is adapted from Reena Catherine J's SO101-Gesture-Teleoperation, originally split across a Windows host and a WSL receiver. This submission collapses both halves onto one machine over loopback, and closes a functional gap in the receiver: in the upstream version the main loop never called the IK solver and never wrote the arm actuators, so only the gripper actually moved. The receiver documented in §3 now runs inverse kinematics on Cartesian targets, passes joint-space targets straight through, and drives all six actuators.

02 — Assembly

Mechanical and electrical assembly

A 3D-printed 5-DoF arm plus parallel-jaw gripper, driven by six daisy-chained smart serial servos on a single bus — and a host machine split across two cooperating processes so a webcam pipeline never has to share a Python environment with ROS.

2.1Mechanical structure

The arm is the open-hardware SO-101: six revolute joints in series — base → shoulder → upper_arm → lower_arm → wrist → gripper — each link a 3D-printed part described by the project's URDF and its matching STL meshes. Joint 6 actuates the parallel jaw; the jaw body itself is the point inverse kinematics targets when solving for a Cartesian position (§3.3).

Joint travel limits, as declared in the URDF and enforced identically by the gesture controller and the receiver
Joint	Link pair	Min (rad)	Max (rad)	Function
1	base → shoulder	−1.920	+1.920	base rotation
2	shoulder → upper_arm	−1.745	+1.745	shoulder pitch
3	upper_arm → lower_arm	−1.745	+1.571	elbow
4	lower_arm → wrist	−1.658	+1.658	wrist bend
5	wrist → gripper	−2.793	+2.793	wrist roll
6	gripper → jaw	runtime¹	runtime¹	gripper open/close

¹ read from MuJoCo's actuator_ctrlrange at startup rather than hard-coded, so it always matches the compiled model.

Modeling note

The URDF's visual and collision meshes are the same non-convex, 3D-printed geometry, designed to mate with small intentional overlaps. MuJoCo approximates each as one convex hull for contact, so neighboring links (e.g. the jaw against its wrist mount) register large false penetrations. The digital twin therefore disables collisions on every robot geom and the floor — gravity, joint limits, and the position servos are still fully simulated, only contact dynamics is not.

2.2Electrical assembly

All six joints use Feetech STS3215 smart serial servos, daisy-chained on one half-duplex TTL bus at 1,000,000 baud (protocol version 0, the STS/SMS register map), addressed by servo ID 1–6 and reached from the host over a USB-serial adapter (default /dev/ttyACM0). Each servo reports its own position as a raw tick in [0, 4095] with no relationship to the URDF's zero pose, so a one-time calibration pass records the raw tick every servo reports with the arm physically posed at the URDF's zero — the six numbers that map raw ticks to radians for the rest of the system.

Fig. 1 — Electrical assembly. One serial adapter drives all six servos over a shared signal bus; power for the servo rail is deliberately separate from the USB signal path. The host runs the vision pipeline and the ROS 2 / MuJoCo receiver as two processes talking over a local TCP loopback, not ROS topics.

Why two processes, two environments

The controller runs from its own virtual environment because MediaPipe pins an OpenCV build that would otherwise displace the one the rest of the ROS 2 workspace relies on. Splitting the pipeline across a TCP socket rather than a shared process — or a ROS topic — means the webcam/MediaPipe half can be recreated, restarted, or swapped independently of the simulation/hardware half:

python3 -m venv --copies ~/ros2_ws/.venv-gesture
~/ros2_ws/.venv-gesture/bin/pip install mediapipe opencv-python

The real hardware bridge is deliberately reused rather than reimplemented: FeetechBus is joint-name-for-name and radian-for-radian compatible with the simulated model, because both read the same URDF and the same calibration reference — a joint target computed for the MuJoCo actuators is sent to the physical servos completely unchanged (§3.3, §4).

03 — Algorithm design

Robot function, decision logic, and kinematics

Every video frame runs the same pipeline: classify the hand shape, update a small control state machine, turn hand position into a target, and stream that target to a receiver that solves inverse kinematics before anything moves.

3.1End-to-end data flow

Fig. 2 — End-to-end flow from a video frame to servo motion. The receiver runs IK on a scratch scene (never the live one) and only ever writes to real servos on the explicit, doubly-gated --real-hardware --no-dry-run path.

3.2Gesture classification & decision logic

Classification runs on the 21 normalized hand landmarks MediaPipe returns per frame, checked in a fixed priority order — pinch first, since a pinch can occur mid-gesture regardless of finger count:

✊ Fist

Zero fingertips above their knuckle joint. The universal disable — works from any state.

extended_count == 0

✋ Open palm

Four or more fingertips extended. Arms gesture control (unless keyboard already has it).

extended_count ≥ 4

🤏 Pinch

Thumb tip and index tip closer than 8% of frame width apart. Closes the gripper.

‖P4 − P8‖ < 0.08

— Other

1–3 fingers extended, or no hand in frame. No state change; last command holds.

no action

finger[i]_extended = tip[i].y < pip[i].y for i in {index, middle, ring, pinky} landmarks 8/6, 12/10, 16/14, 20/18

Fig. 3 — Control state machine. Note the asymmetry: keyboard control never returns directly to gesture control — an operator must clear to DISABLED (fist, or a mode switch) before an open palm re-arms gesture input, so a hand drifting back into frame can't silently override manual keyboard positioning.

Hand-to-target mapping

In Cartesian mode, wrist position in the (mirrored) camera frame maps to robot X/Y, and the distance between wrist and middle-fingertip landmarks — hand tilt toward or away from the camera — maps to robot Z (depth). In joint mode, the five arm joints each take a different signal from the same 21 landmarks: horizontal wrist position for base rotation, vertical wrist position for shoulder pitch, hand depth for the elbow, palm-edge tilt angle (landmarks 5→17) for wrist bend, and wrist-to-index-fingertip angle for wrist roll. Every target — in either mode — is exponentially smoothed toward the raw reading and clamped to the soft workspace/joint limits before it is ever put on the wire.

3.3Kinematic calculations

Fig. 4 — The five arm joints plus gripper, in kinematic order. Inverse kinematics solves for J1–J5 to place the jaw body at a requested (x, y, z); J6 is set directly from the gripper open/closed bit, never through IK.

The receiver solves IK numerically rather than with a closed-form solution: a damped, pseudo-inverse Jacobian iteration, run fresh every control tick and seeded from the arm's own current joint state — a continuous resolved-rate controller, not a one-shot plan-and-execute solve. It reasons about position only (3 rows), not orientation, and always runs on an isolated scratch MjData so a bad iteration can never write into — or destabilize — the live simulation or the real arm mid-solve:

e = target_xyz − x_jaw(q) if ‖e‖ < 0.005 m: break J = ∂x_jaw / ∂q (3×5, mj_jacBody) Δq = J⁺ e (Moore–Penrose pinv) q ← clamp(q + 0.5·Δq, q_min, q_max) repeat ≤ 12×

Soft Cartesian workspace bounds enforced client-side before every send
Axis	Min (m)	Max (m)	Note
X	−0.20	+0.20	lateral
Y	−0.32	−0.06	reach — arm extends toward −Y from the base
Z	0.05	0.30	height above the base plane

Whichever mode is active, the joint target it produces — solved via IK, or passed straight through in joint mode — lands in exactly one place: the position actuators' data.ctrl array. The simulation then steps physics 20 substeps per outer loop tick under MuJoCo's implicitfast integrator, and the same array is what gets read out and forwarded to the real servo bus when hardware mirroring is on.

04 — Control & safety

Control system and safety feature design

No single safeguard carries the whole system — each layer of the pipeline (perception, control logic, kinematics, network, hardware bridge) independently refuses to pass along something it shouldn't.

Defense-in-depth: hazard, mitigation, and the layer that owns it
Layer	Hazard	Mitigation
Perception	Gesture control arms itself accidentally	Control only enables on an explicit open palm; a fist from any state disables both gesture and keyboard control immediately.
Control logic	Operator needs to halt motion instantly	SPACE latches a software emergency stop from any state; no outgoing command is sent while latched. Resuming needs an explicit R reset and a freshly re-shown open palm — no silent auto-resume.
Control logic	Keyboard and gesture inputs fight each other	The state machine (Fig. 3) makes the two mutually exclusive and gives keyboard input no direct path back to gesture control — it must pass back through `DISABLED` first.
Kinematics	Hand tracking asks for an out-of-reach or unsafe pose	Cartesian targets are clamped to soft X/Y/Z bounds and joint targets to per-joint limits on both the controller and the receiver — independently, so neither side depends on the other enforcing it.
Kinematics	A diverging IK iteration corrupts the live scene	IK iterates on an isolated scratch `MjData`; only a finished joint target is ever copied out; the live simulation's `qpos` is never written mid-solve.
Network	A malformed or truncated TCP message	Newline-delimited parsing wrapped in `try/except` — an invalid line is logged and dropped without killing the receiver thread or the connection.
Network	The controller disconnects mid-session	`recv()` returning empty (or raising) flips a `running` flag that stops the simulation loop and drives a clean shutdown — socket, server, and servo bus all closed in `finally`.
Network	Torn reads of a multi-field target	A threading lock guards the shared Cartesian/joint/gripper/mode state between the network-receive thread and the physics/control loop.
Hardware bridge	The real arm moves unexpectedly	Two independent opt-ins: `--real-hardware` alone still connects read-only (no torque); the physical arm only moves with the additional, explicit `--no-dry-run`.
Hardware bridge	An out-of-range gripper target aborts the whole write	Arm joints and the gripper are written to the servo bus in two separate calls, so a gripper `ClampExceeded` can't also block the five arm-joint writes.
Hardware bridge	Simulated and real gripper ranges disagree	The gripper target is pre-clamped to the real servo's own live calibrated range before writing — it closes as far as mechanically possible instead of raising and stalling.
Perception	A stale or re-enumerated camera index runs blind	Camera open is verified with an actual frame read, not just `isOpened()`; failure exits immediately with an actionable message (override via `SO101_CAMERA`).

./run_receiver.sh --real-hardware # dry-run: logs intended writes, no torque ./run_receiver.sh --real-hardware --no-dry-run # LIVE — moves the physical arm

Known limitation

Execution success on the real hardware path means the target was sent, not that the servo has physically arrived — the conservative velocity/acceleration ramp means a large move can still be travelling for several seconds afterward. This is a property of the underlying lerobot_hardware bridge, documented as a limitation there rather than papered over here.

05 — Demonstration

Working demonstration

A one-minute recording of the system operating end to end, against the outcome criteria defined in §1.2.

demo.mp4 — pending

Video to be added: open palm → arm → move → pinch → gripper close → fist → disarm, shown on both the digital twin and the physical arm.