dynamics-toy

Toy v2.1: independent families and crossed factors

This is the replacement Toy v2.1 release requested by the dataset owner. It supersedes the initial v2.1 extension at commit 2023f1d2254394f469877a4e2af3e6b78fa415a3. The unpublished development name was v2.2; there is no separate active v2.2 release. Pin the new commit SHA for reproducibility, because the v2.1 tag selects this replacement. Preregistered opaque v22_* family/point IDs retain their identity and are not version selectors.

Toy v2.1 preserves the 9,600-family, 207,000-clip v2.0 core and replaces the former small diagnostic extension with 2,304 independent new families. It provides 237,150 reference clips over 11,904 physical families, plus separately counted evaluator futures. All new family counts exclude rerenders, synchronized views and stochastic draws. The 507 preserved core archives are unchanged.

Size and splits

Clip means one reference-view 96-frame video. Synchronized views and evaluator future banks are not counted as extra reference clips. A physical family contains matched variants/rerenders and belongs to exactly one split; cohort memberships overlap. History rerenders reuse 54 preserved families.

Full archive storage: 883.34 GB (decimal), 3,049 archives. This includes RGB, dense oracles and evaluator references. See release.json and the SHA256 asset index for exact bytes and membership. Download a selected cohort/mode rather than unpacking the full dataset by default.

Each main clip is lossless uint8 RGB, 96 frames at 24 FPS, 256 × 256. Observation is 0–35, future is 36–95. New enriched clips have three synchronized same-time views, calibrated camera trajectories, simulator state and persistent material-point tracks. The 36 exact-legacy-renderer bridge clips have one view and no new dense point/depth assets. The preserved core retains its original annotation coverage.

What is independently varied

• Each new dynamic trajectory has matched static/moving camera renders. Camera/appearance/policy streams are separate from physics initialization; cameras never follow future states.
• The 2,304 new families balance object count 3/4/5 × sphere/rigid geometry × D_control/S_policy within each split. D_mass remains in the core; 54 of its original families also have 432 matched enriched camera/appearance renders. Another 54 rendering-fixed control clips use existing new D_control families with the same object count, appearance and camera.
• Sparse matched crosses combine camera motion, rigid geometry, repeated appearance, added partial/full occlusion and stochastic strength. All 96 registered split × geometry × regime × camera × added-occlusion × repeated-appearance cells are present. This is not a full Cartesian product of every nuisance parameter; factor coverage lists every stored cell. Strength sweeps use matched partial panels, while full-panel conditions occur in D_control and high-strength S_policy families.
• Crossing and object-object occlusion use prescribed physical initial conditions, then record realized visibility/crossing events. Individual intended-event failures are retained. Render-only foreground panels are a separately labeled intervention.
• Continuous camera parameters and rig baselines vary. IID camera motion classes are balanced across split, tracking and strength banks. Test OOD covers amplitude, curvature, speed profile and elevation; a metadata-registered combination holdout uses individually seen appearance, motion class and rig-baseline bins.
• Rigid objects include cube, cuboid, capsule and cylinder; asymmetric wedges occur in a registered test shape holdout. Shape changes collision physics. Texture, palette, background, illumination and material settings are rendering factors.
• 216 families (72 per split) provide A/B/C/D static/dynamic scene × static/moving camera groups. 36 sphere families (12 per split) provide exact legacy shader / enriched static / enriched moving renderer bridges on the same physics.

The new dynamics use PyBullet 3.2.7 at 240 Hz, with stored position, velocity, rotation, angular velocity, mass, inertia and shape. The preserved core uses its original ballistic CCD dynamics. New S_policy futures use piecewise random force/torque after the forecast boundary, distinct from the core interactive-feedback policy. physics_engine and policy_kernel label these differences; do not pool them as a single identical transition kernel.

Stochastic generation evaluation

The registered new test subset has 36 families × four strengths = 144 conditions. Each has 128 IID state references, with 32 saved RGB futures from the same registered draws: 18,432 state futures and 4,608 RGB futures in total. Each condition stores a shared 36-frame observation and 32 future sequences of 60 frames. Reference seeds are disjoint from primary saved draws; zero-strength outcomes coincide by design. All strengths and draws share their observed physical/RGB history and full camera trajectory.

RGB reference paths apply only to the canonical moving-camera, primary-appearance, no-panel rendering condition. A static, repeated-appearance, panel or camera-OOD render cannot use that reference bank. The original 1,200 test S_policy families retain their original 128 state references; these are not additional RGB banks.

Hidden strength is unidentifiable from identical observations. Use an externally declared strength track, separate experiments, or the pooled mixture. Do not expect an RGB-only model to infer the hidden per-row strength label.

Generation is the primary method comparison: hold backbone/decoder, observation history, prediction horizon and training budget fixed. Report family denominators and factor-conditioned generation scores. Geometry/correspondence diagnostics help attribute failures but do not replace generation evaluation. This release contains ground truth and validation results, with no learned-model predictions or performance claims.

Access and training sampling

student returns observed reference RGB and timestamps only. teacher returns simultaneous observed multi-view RGB. oracle explicitly loads physical state, camera/depth/segmentation/visibility/point/event assets. Forecast RGB is a separate opt-in training target. Evaluator future banks require oracle mode. Researcher manifests, IDs, filenames, mass, regime and hidden strength are not returned as model features.

The sampler chooses group → family → regime → factor cell → trajectory → render. Default weights: enriched 0.50, tracking_cross 0.15, stochastic_strength 0.15, legacy_core 0.15, history 0.05. Overlapping groups are explicit mixture routes. Stored rerender/view count does not set family probability. Exact static-camera probability on training rows is 92.704% under uniform clips and 57.500% under this sampler. These are computed distribution weights, not fitted model results.

# After materializing the selected data, place loader.py and sampler.py together.
from loader import ToyDataset
data = ToyDataset('/path/to/toy-v2.1', split='train', cohort='training',
                  mode='student', include_targets=True)
sampler = data.training_sampler(draws=1000, seed=0)
example = data[next(iter(sampler))]
# rgb: [36,256,256,3], timestamps: [36], target_rgb: [60,256,256,3]

The default sampler requires its five groups to be materialized. For an enriched-only training selection, pass weights={'enriched': 1.0}. For teacher inputs select cohort='geometry_multiview', mode='teacher' and explicitly choose available enriched groups. Views remain channels of one example.

# Download v2.1/download.py, loader.py and sampler.py from this repository.
pip install numpy huggingface_hub
python download.py --root /path/to/toy-v2.1 --split train --cohort enriched --mode student --limit 3 --revision <commit-sha>
python download.py --root /path/to/toy-v2.1 --split train --cohort geometry_multiview --mode teacher --limit 3 --revision <commit-sha>
# --limit 0 selects the complete cohort. --sample-id selects an exact render.
# --mode oracle --references additionally materializes registered future banks.
# --source volume uses volume://vessl-storage/prj-dynamics/datasets/toy-v2.1/v2.1.

The downloader verifies full archive SHA256 and selected member SHA256, extracts only requested regular members, and records the immutable Hub revision. selection.jsonl records the currently selected rows. Public files are NPZ/TAR/JSONL; this is a custom video dataset, not a Parquet table for automatic load_dataset() decoding.

The asset index selects the active metadata member for each path. All 144 new reference manifests use public version 2.1; their canonical metadata archive supersedes unindexed development members inside immutable render archives. Use the indexed downloader. The metadata migration record records the correction; RGB and physical states are unchanged.

Cohorts and evidence

Do not sum overlapping cohort counts. Within-extension matched comparisons and the renderer bridge have different estimands. Fully-visible clean observations require every reference-view object to remain in frame with at least 99.5% of its isolated silhouette during frames 0–35. Full-clip intervals and observed-only labels are distinct fields. Material-point IDs end at physical identity; learned query ownership is inferred by the model/diagnostic.

The validation report checks all registered jobs, family isolation, point reprojection, RGB history matching, matched crosses and reference-bank conditioning. Shortcut audit reports contingency tables, mutual information and categorical lookup accuracy under uniform clips and the recommended sampler. Lookup is in-sample, not a heldout predictive probe; concatenated purpose-built cohorts are not claimed independent. No future-outcome sample filtering was used.

Actual ground-truth context: two geometry rows × three prescribed scenario columns, fixed test n3 D_control ordinals 0/4/8, reference camera, no added panel. These are different physical families, not matched geometry interventions. Quantitative findings come from exhaustive audit tables, not this animation. Static strip, sample selection, schema, generator, registration, camera assignment audit.

Limitations: synthetic room dynamics and constrained event initialization; finite camera/appearance catalogs sampled from continuous distributions; sparse rather than universal factor crossing; visually ambiguous textureless symmetric surfaces; new physics/renderer domain differs from the preserved core. Full state/oracle availability does not make an unobserved physical identity identifiable from RGB.