spiral-so101-rlt12-stage1-beta-teleop57-dagger44-rm1fix12-fresh-1500

SPIRAL on rlt12 stage 3: stage-1 (round 1) beta sweep 5 / 15 / 70 / 100 on π₁'s own data (57 teleop + 44 DAgger RLT_1, RM1 fix12), 1,500 steps

SPIRAL round 1 as Algorithm 1 stage 2b defines it (π₁'s own fine-tuning data, labelled by RM1), repeated at four values of beta_bc, one run per beta, everything else fixed. A residual actor and a 5-critic ensemble on top of a FROZEN π₀.₅ + RLT policy; actor and critics both start fresh.

Not robot-evaluated; the offline critic probes have not been run.

The four checkpoints

All four are identical except beta_bc (checked in each rl_config.json).

Caveat for beta 70 and 100: with a fresh actor the residual starts large (random init) and has to shrink to the value set by the critic. At beta 5 and 15 it settles (residual × beta ≈ 0.035, the critic's slope on this data), but at beta 70 and 100 it was still falling at step 1,499: the critic-set value would be about 0.0005 and 0.00035. Part of their residual is therefore left over from initialisation, not the critic's direction. Both are still tiny corrections.

The critics of the four runs are the same to 3 decimals (at step 1400: critic loss 0.0017, Q min / mean / max ≈ 0.48 / 0.851 / 0.979, best-of-4 selection gain 0.0005).

Data

local/r1b_teleop57_daggerrlt1_rm1fix12: 101 episodes / 37,813 frames, 15 fps; the same data r1b used and π₁ (the base) was fine-tuned on:

Same episodes and states as π₁'s training set local/teleop57_daggerrlt1_v21 (checked frame by frame), minus the last frame of each episode, which the labeller drops.

Labels: RM1 (sarm2_gripper58), fix12: opensarm/tools/label_for_spiral.py --stride 15 --smoother-mode none --write-progress --task "Fold the long sleeved T-shirt on the table" --rm-instruction "Fold the T-shirt on the table". Final progress 0.90–0.97 in every episode; MC target 0.871 ± 0.056; 4.2% of per-step rewards are negative. There are no failed endings in this data, so the critic's target is nearly constant.

Training

Training numbers, beta 15 run (from the log)

Suggested robot order (the openspiral author's procedure): base (dry_run=True) → beta 100 → 70 → (r1b, beta 30) → 15 → 5; stop once a beta is clearly worse than the base.

Serving

Identical to rounds 1b, 2 and 4; only the checkpoint path changes.

1. Get everything

hf download Nadeem-Shoukath115113/pi05-so101-rlt12-lehome-s3-daggerrlt1-15fps-b32-10k --local-dir ckpt/base
hf download Nadeem-Shoukath115113/spiral-so101-rlt12-stage1-beta-teleop57-dagger44-rm1fix12-fresh-1500 --local-dir ckpt/spiral_s1beta

You need code from branch lehome-noise-norm (commit f79f191 or later): the lehome normalisation and the 12-dim model. A code-only copy is in the HF bucket Nadeem-Shoukath115113/test, folder spiral_rlt12_serve/ (hf sync hf://buckets/Nadeem-Shoukath115113/test/spiral_rlt12_serve ./spiral_rlt12_serve). Run from the code root. The loader falls back to assets/rlt12_s3_dagger_rlt1/local/teleop57_daggerrlt1_v21/norm_stats.json relative to that root. That file is included in the bucket copy and is byte-identical to the base checkpoint's 9999/assets/local/teleop57_daggerrlt1_v21/norm_stats.json.

2. Load

scripts/serve_policy.py has no entry point for this policy. Build it with the loader and wrap it in the websocket server:

import os
from openpi.policies.residual_rl_rlt_policy import load_residual_rl_rlt_policy
A = os.path.abspath   # ALL paths must be absolute -- orbax rejects relative ones
policy = load_residual_rl_rlt_policy(
    pi0_config_name="residual_rl_pi05_so101_rlt12_demos57",   # same model/transforms; see note
    rl_checkpoint_dir=A("ckpt/spiral_s1beta/beta15/step_00001499"),   # or beta5 / beta70 / beta100
    rlt_checkpoint_dir=A("ckpt/base"),                       # latest step inside must be 9999
    pi0_checkpoint=A("ckpt/base/9999/vla_params"),
    best_of_k=1,                                             # or 4, see below
    noise_cholesky_path=A("ckpt/spiral_s1beta/action_cholesky.npy"),
    dry_run=False,                                           # True = base actions only (no residual)
)

On the config name: the round-1 preset residual_rl_pi05_so101_rlt12_demos57 is in the committed code, and its model, normalisation and assets are identical to this run's preset. The two differ only in the training repo_id, which serving does not use. With the uncommitted preset available, residual_rl_pi05_so101_rlt12_r1b_rm1fix12 works too.

3. Choose K and the noise

K=1 fix. At commit f79f191, _FusedRLTInferenceModule.infer_actions (src/openpi/policies/residual_rl_rlt_policy.py, about line 122) draws plain noise and ignores the Cholesky. Only the best-of-K path uses eps = L @ z. For K=1: - give infer_actions a noise_cholesky=None argument and draw eps = einsum("ij,...j->...i", L, z).reshape(B, 20, 12) when it is set; - pass noise_cholesky= at warmup and in infer() for both K paths.

The same eps must go to pi0_model.sample_actions(..., noise=eps) and, flattened to 240, to the actor.

4. Observation sent by the robot

The input transforms skip the repack step, so send the post-repack keys:

images.right_front   <- overhead camera  (observation.images.left_top)
images.left_wrist    <- observation.images.left_left_wrist
images.right_wrist   <- observation.images.right_right_wrist
state                <- 12 joint values, robot units
prompt               <- "Fold the long sleeved T-shirt on the table"   (exactly)

5. What must match training (it does, if you use the loader above)

• Actions: all 12 dims are deltas from the state, then z-scored per chunk step with per_timestamp_mean/std (20 × 12). The residual is added in that normalised delta space, then: Unnormalize (per step) → AbsoluteActions → BiSO101Outputs. A plain global mean/std normaliser gives wrong actions with no error.
• State: min/max mapped to [-1, 1]. π₀.₅ and the actor see the same normalised state.
• z_rl: RLT encoder 2048 → 2048, 768 image tokens only, 2 layers, 8 heads, mlp ratio 4.0. π₀.₅ runs in bfloat16.
• Noise: eps = L @ z, z ~ N(0, I_240), L = this repo's action_cholesky.npy. Never use the raw Cholesky stored in norm_stats.json. π₀.₅ uses 10 flow steps.
• Residual scale: the actor already multiplies its output by 0.1. Add it as is.