Text-to-motion planner for Reachy Mini: a prompt goes in, and a ready-to-play head, antenna and body trajectory comes out.
This repo is a complete serving bundle:
| path |
what |
planner/ |
Qwen/Qwen3.5-0.8B fine-tuned (LoRA, merged) to write motion recipes; includes the base model's MTP head for speculative decoding |
generator.pt |
21.8M-parameter flow-matching transformer: plan -> 25 Hz 9-DoF motion, trained only on real Reachy Mini motion |
serve.json |
the settings the service was tuned with (FP8, MTP drafts, 32 diffusion steps, 2 Hz plan expansion) |
python -m inference.server --bundle binhpham/reachy-mini-motion-planner-0.8b # from the reachy-motion-generator project
curl -s localhost:8000/generate-dense -H 'content-type: application/json' \
-d '{"prompt": "sneezing. You build up and then sneeze loudly.", "n": 2}'
# all three planners in one process (one GPU, shared generator), picked per request with "effort"
python -m inference.server --bundle high=binhpham/reachy-mini-motion-planner-27b \
--bundle medium=binhpham/reachy-mini-motion-planner-4b --bundle low=binhpham/reachy-mini-motion-planner-0.8b
curl -s localhost:8000/generate-dense -H 'content-type: application/json' -d '{"prompt": "startled. A door slams.", "effort": "medium"}'
The response holds the recipe, a one-line idea, and moves: Reachy Mini recorded-move dicts
({"time", "set_target_data": [{"head": 4x4, "antennas", "body_yaw"}]}), already projected onto the robot's reachable set.
Serving effort: "low" runs this model. Speed: ~0.18 s median per prompt on one RTX PRO 6000 (FP8 + fine-tuned MTP; planner 0.14 s), the fastest tier; weaker on multi-phase motions (yawns, bows, build-up events). GPU memory: ~6 GB.
How the planner is prompted
Compact system prompt (units, recipe grammar, 4 motion rules, 3 examples), user message = the prompt
(word. one sentence of context. works best). The answer is {"idea", "recipe"} with thinking off.
Allow at least 400 output tokens.
Training
- Data: 5,872 teacher rows: hand-written recipes (Claude), build-up/release events (×3), 287 seeds, and 5,000 scenarios
(Astra) re-authored in a lively style by Codex (gpt-6-astra). Each prompt is also trained as its bare word and its
sentence alone. Rows near any evaluation prompt are removed (embedding filter plus a keyword blocklist for the
out-of-distribution probes). The data is published as
binhpham/reachy-mini-massive-motion-library.
- Training: LoRA r = 32 on the attention and MLP projections, loss on the answer only, best checkpoint by held-out loss.
The MTP head was then fine-tuned on the planner's own outputs, which makes speculative decoding faster without changing outputs.
Evaluation (prompts never seen in training, 12 samples per probe)
|
|
| out-of-distribution physical probes (sneeze releases downward, sleepy toddler droops and recovers, …) |
0.66 |
| skill probes (nodding, bowing, looking up, …) |
0.57 |
| plan agreement with held-out teacher recipes (mean r) |
0.41 |
| identification among 12 held-out real Pollen clips (top-1 / mean rank, chance 8% / 6.5) |
17% / 4.93 |
| "sneezing" release direction correct |
12/24 |
Try it in the browser: binhpham/reachy-mini-motion-generator.
