Dataset
b1k-task55-re-shelving-library-books-official-eval
by Jeremy jermo1120/b1k-task55-re-shelving-library-books-official-eval
Rollouts of the Pi0.5 fine-tune jermo1120/b1k-task55-re-shelving-library-books (step24999) on BEHAVIOR-1K task reshelvinglibrarybooks, following the 2026 challenge leaderboard protocol. Missing instances count as 0 in Q.
Dataset Card
By Jeremy, published under apache-2.0, revision c25ce1c07f10.
Official-protocol evaluation of jermo1120/b1k-task55-re-shelving-library-books
Rollouts of the Pi0.5 fine-tune jermo1120/b1k-task55-re-shelving-library-books (step_24999) on BEHAVIOR-1K task re_shelving_library_books, following the 2026 challenge leaderboard protocol.
Status: complete.
Result
| checkpoint | instances run | successes | Q (mean q_score over the 10 instances) |
|---|---|---|---|
step_24999 |
10 / 10 | 0 | 0.067 |
Missing instances count as 0 in Q.
| checkpoint | instance_id | success | q_score | steps | norm. base / left / right distance |
|---|---|---|---|---|---|
step_24999 |
301 | No | 0.333 | 14293 | 1.11 / 0.36 / 0.43 |
step_24999 |
302 | No | 0.000 | 14293 | 4.53 / 0.69 / 0.48 |
step_24999 |
303 | No | 0.000 | 14293 | 1.79 / 0.48 / 0.30 |
step_24999 |
304 | No | 0.000 | 14293 | 3.43 / 0.53 / 0.81 |
step_24999 |
305 | No | 0.000 | 14293 | 0.94 / 0.41 / 0.40 |
step_24999 |
306 | No | 0.000 | 14293 | 3.09 / 0.61 / 0.67 |
step_24999 |
307 | No | 0.000 | 14293 | 2.55 / 0.57 / 0.53 |
step_24999 |
308 | No | 0.000 | 14293 | 1.32 / 0.53 / 0.51 |
step_24999 |
309 | No | 0.333 | 14293 | 0.90 / 0.36 / 0.37 |
step_24999 |
310 | No | 0.000 | 14293 | 1.87 / 0.55 / 0.37 |
q_score gives partial credit: 1.0 for a full success, otherwise the fraction of goal predicates that were false at the start and true at the end of the episode. The task 55 goal is only "every book (3) is inside a bookcase", so per-instance scores are 0, 0.33, 0.67 or 1.0. Distances are normalized by the mean human demo (1.0 = same path length as a human).
Per-rollout metrics are in results/<checkpoint>/*.json and results/summary.csv, videos (head + both wrist cameras) in videos/<checkpoint>/, evaluator logs in logs/<checkpoint>/instance_*.log and the policy server log in logs/<checkpoint>/server.log.
Protocol
As specified in BEHAVIOR-1K docs/challenge/evaluation.md:
| Instances | public_test instance indices 0-9 (instance ids 301-310), the leaderboard-reporting half of the split |
| Rollouts | 1 per instance, no reruns or selection |
| Episode limit | default 1.5x mean human-demo length (14292 steps) |
| Videos | --write-video |
| Evaluator | BEHAVIOR-1K v3.9.3-post2 (python -m omnigibson.eval.eval), OmniGibson 3.9.3, Isaac Sim 5.1 |
| Robot config | omnigibson/eval/r1pro.yaml (robot robot_r1), default wrapper (224x224 RGB + proprio) |
| Policy server | wensi-ai/openpi@behavior + the model repo's changes.patch + scripts/robot_config_eval2026.patch; scripts/b1k/serve_b1k.py --robot b1k/R1Pro_eval2026 |
| Hardware | 1x RTX 5090 (simulator and policy on the same GPU) |
Command: OUT_ROOT=out_official INDICES="0 1 2 3 4 5 6 7 8 9" scripts/run_eval.sh 24999.
Checkpoint selection
step_24999 was chosen from a sweep of 9 checkpoints (5k to 25k) on the other half of the split (instances 311-315, indices 10-14), where it was the only checkpoint with nonzero q_score. The sweep is in jermo1120/b1k-task55-re-shelving-library-books-eval.
Details
- Repository
- jermo1120/b1k-task55-re-shelving-library-books-official-eval
- Publisher
- Jeremy
- Task category
- Not stated by the source
- Tags
- robotics, behavior-1k, evaluation
- Size category
- Not stated by the source
- Languages
- Not stated by the source
- Revision
- c25ce1c07f1001dc5fafe710db12c54f7e7e7533
- Last updated
- 2026-10-13
Files
37 files, 273.2 MB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| results/step_24999/re_shelving_library_books_301_0.json | Data | 561 B | — |
| results/step_24999/re_shelving_library_books_302_0.json | Data | 543 B | — |
| results/step_24999/re_shelving_library_books_303_0.json | Data | 544 B | — |
| results/step_24999/re_shelving_library_books_304_0.json | Data | 544 B | — |
| results/step_24999/re_shelving_library_books_305_0.json | Data | 541 B | — |
| results/step_24999/re_shelving_library_books_306_0.json | Data | 544 B | — |
| results/step_24999/re_shelving_library_books_307_0.json | Data | 543 B | — |
| results/step_24999/re_shelving_library_books_308_0.json | Data | 543 B | — |
| results/step_24999/re_shelving_library_books_309_0.json | Data | 557 B | — |
| results/step_24999/re_shelving_library_books_310_0.json | Data | 543 B | — |
| results/summary.csv | Data | 730 B | — |
| README.md | Documentation | 3.4 KB | — |
| logs/step_24999/instance_301.log | Other | 56.6 KB | — |
| logs/step_24999/instance_302.log | Other | 55.8 KB | — |
| logs/step_24999/instance_303.log | Other | 55.8 KB | — |
| logs/step_24999/instance_304.log | Other | 55.8 KB | — |
| logs/step_24999/instance_305.log | Other | 56.1 KB | — |
| logs/step_24999/instance_306.log | Other | 57.4 KB | — |
| logs/step_24999/instance_307.log | Other | 56.1 KB | — |
| logs/step_24999/instance_308.log | Other | 56.1 KB | — |
| logs/step_24999/instance_309.log | Other | 56.9 KB | — |
| logs/step_24999/instance_310.log | Other | 56.1 KB | — |
| logs/step_24999/server.log | Other | 12.7 KB | — |
| scripts/build_official.py | Other | 5.1 KB | — |
| scripts/robot_config_eval2026.patch | Other | 1.1 KB | — |
| scripts/run_eval.sh | Other | 1.7 KB | — |
| videos/step_24999/re_shelving_library_books_301_0.mp4 | Other | 27.3 MB | afea22511d93 |
| videos/step_24999/re_shelving_library_books_302_0.mp4 | Other | 22.9 MB | ef0e7ce24d11 |
| videos/step_24999/re_shelving_library_books_303_0.mp4 | Other | 27.6 MB | 63dbbb94ec7d |
| videos/step_24999/re_shelving_library_books_304_0.mp4 | Other | 28.6 MB | aa8583d764ff |
| videos/step_24999/re_shelving_library_books_305_0.mp4 | Other | 27.4 MB | 0c5230ce1f3c |
| videos/step_24999/re_shelving_library_books_306_0.mp4 | Other | 24.2 MB | 5182f304b7b5 |
| videos/step_24999/re_shelving_library_books_307_0.mp4 | Other | 27.6 MB | 400f81ab22d7 |
| videos/step_24999/re_shelving_library_books_308_0.mp4 | Other | 28.1 MB | 229eb5c8be06 |
| videos/step_24999/re_shelving_library_books_309_0.mp4 | Other | 29.1 MB | 74989f9e02d8 |
| videos/step_24999/re_shelving_library_books_310_0.mp4 | Other | 29.6 MB | 382e1848f819 |
| .gitattributes | Repository | 2.5 KB | — |
License and Download
- License
- apache-2.0
- Access
- No access gate
Released by Jeremy through its official repository on Hugging Face. Read the license.