SmolVLA trained to move a chess piece with an
SO-101 arm in one fixed simulated scene (MuJoCo). It is the simple baseline of the
chess robot's "hand": first get reliable closed-loop success in the easiest setting, then add
variety back one change at a time. Trained fresh from lerobot/smolvla_base, not from the
earlier Machanize/chess_phase_smolvla.
Status: not yet reliable. In a closed-loop test it completed 36.7% of 60 moves (details below). It has not been run on a real arm.
What it does
- Input: two 640x480 camera images and the arm's joint positions, at 30 Hz.
observation.images.overhead: a webcam 65 cm straight above the board, 44 degrees top to
bottom, the arm at the top edge of the image. Fed to the model as camera1.observation.images.wrist: the official SO-101 wrist camera. Fed to the model as camera2.- The piece to move has a red square with an opaque outline drawn over its square; the
target square has a blue one. Both are drawn on the images before they reach the policy.
observation.state: 5 arm joints in degrees and the gripper 0-100 (LeRobot's SO-101
follower units with use_degrees=True).- The instruction is always "move the piece on the red square to the blue square".
- Output: the next joint targets in the same units, in chunks of 50 steps.
- The training video was stored at high quality (H.264, CRF 18, full 4:4:4 colour), so the
markers keep their colour. For the closest match, pass live frames through the same encode
and decode (
training_look(image, 18, "yuv444p") in the project's sim/camera_effects.py).
The scene (all fixed)
- Board with 25 mm squares, square to the robot and flush against its 6 cm deck; the robot
plays black. One Staunton set, black and white pieces, centred in their squares. White arm.
- Tray on the right, one table, the same two lamps, no clutter, a clean camera image.
- Ten moves from the start position: e7e5, d7d5, g8f6, b8c6, c7c5 (the robot's side) and e2e4,
d2d4, g1f3, b1c3, c2c4 (the far side). The start is the same every time and only the red and
blue squares change, so doing different moves correctly shows it reads the markers.
Test in simulation
60 closed-loop episodes, each a random one of the ten moves from seeds not used in
training. An episode counts as a success if the piece ends within 6 mm of the target square,
upright, and no other piece moved more than 2 mm. 20 s limit.
Success: 36.7%
- Gripper within half a square of the marked piece: 95.0% of episodes
(median closest approach 3.3 mm).
- Marked piece lifted: 68.3%.
| move |
success |
episodes |
| b1-c3 |
16.7% |
6 |
| b8-c6 |
14.3% |
7 |
| c2-c4 |
0.0% |
5 |
| c7-c5 |
66.7% |
6 |
| d2-d4 |
33.3% |
6 |
| d7-d5 |
60.0% |
5 |
| e2-e4 |
60.0% |
10 |
| e7-e5 |
33.3% |
6 |
| g1-f3 |
33.3% |
3 |
| g8-f6 |
33.3% |
6 |
What goes wrong
- It reads the markers. The gripper reaches within half a square of the marked piece in
95% of episodes, for every one of the ten moves.
- It fails at the grasp. A grasp inspection over 20 episodes compared the pinch point, at
the moment the policy starts closing the jaws, with the grasp the scripted expert would make
on the same piece. Grasps that held were 0.8-0.9 mm off; failures were 4-6 mm off. With the
jaws off-centre it pushes the piece over (7 of 20) or closes beside it (6 of 20).
- Executing 10 steps of each 50-step chunk before predicting again, instead of all 50, did not
help (4 of 15).
- ACT trained on the same data reached 14%.
- A first fine-tune on correction demonstrations of its own mistakes reached 20%. In those
corrections the teacher rose above the piece and paused before grasping again, and the
model learned to hover over pawns. That model was not kept.
- Next: corrections that line up from where the arm is and go straight down, with no rise and
no pause.
Training data
Machanize/playful, folder chess-sim/datasets/baseline_v1 (private): 300 successful
episodes (89008 frames) of a scripted inverse-kinematics expert at 30 fps, about
30 per move. The expert's start pose and speed vary slightly between episodes.
Training
From lerobot/smolvla_base, 10000 steps at batch 64 on one RTX 4090 in 1.9 h, LeRobot
0.4.4, default SmolVLA settings (vision encoder frozen, action expert trained). Overhead and
wrist cameras are renamed to camera1 and camera2.
Use
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
policy = SmolVLAPolicy.from_pretrained("Machanize/chess_phase_smolvla_baseline")
Load the pre- and post-processors saved with the model using make_pre_post_processors. They
apply the training normalisation and rename the camera keys.
Limitations
- Simulation only, one scene, ten moves. Other moves, board positions, camera views,
lighting or piece sets are untested and expected to fail until the variety is added back.
- The overlay is required: without the red and blue squares it does not know what to move.
