hyperscene_traj

hyperscene_v0

Surface-normal control videos of annotated indoor scenes, with per-pixel instance ids, camera trajectories, six-section captions, LynnReal-Omni RGB renders, the scene meshes, and the packed MiniMax-H3 Ref2VA training split. Every clip is 832x480, 24 fps, 124 frames (5.17 s).

Selection

The clips are InsWindowFilt/select_final.json from this repository's earlier layout (commit b9305a2, since replaced), copied as select_final.json: 10,219 clips over 2,950 scenes that passed three filters in turn:

1. black fraction, edge density, inst_win_min >= 3 (24,560 of 46,309 renders);
2. trajectory dedup within a scene: position Hausdorff < 0.02 and mean per-frame heading < 5 deg;
3. first frame: n0 >= 5 instances of at least 0.1% area, c0 >= 0.6 of their area seen later.

One clip has one key everywhere: <scene>-<traj>, e.g. sk_00061-traj_05 (upstream writes sk_00061_traj_05).

Layout

select_final.json                  the clip list above, byte copy of upstream
raw/shards/raw-NNNNN.tar           10,219 clips in 18 shards of ~2 GiB, stock webdataset
raw/samples.jsonl                  key, scene, traj, shard, split, has_render
train/  val/                       Ref2VA training split, 2,764 / 33 clips (H3-Train format)
render/<scene>/<scene>.tar         5,154 LynnReal RGB videos, one tar per scene
scene/<scene>.mesh.meshopt.zst     2,950 scene meshes
tools/                             captioning tools (caption2step, video_caption); Hub only
gpt-6/                             archive of a normal-caption RGB experiment; Hub only

raw

Members of one sample K:

caption is the six-section Ref2VA prompt (template v2) without its trailing newline; null for the 5,065 clips never captioned. train/val were encoded from rewrite_prompt(caption) (H3-Train data/hyper/add_first_frame.py), which adds the <Picture 1> first-frame wording. scores holds every upstream InsWindowFilt field (blk, edge_density, inst_win_min, inst_win_max, inst_mean, inst_frame_min, inst_win, win, hit_frac, frames_scored, path, path_m, step_std, distinct_colors, window_start) plus n0, a0, c0 from the selection. Upstream defines only the filter fields and inst_win_min; step_std and distinct_colors are undefined there. split is train, val or null; has_render says whether render/ holds the clip.

instance.mkv was rasterized from scene/ with the cameras and intrinsics above (nvdiffrast, near plane 0.001). On a 31-clip check it agreed with normal.mp4 on ray misses everywhere except within 2 px of an edge.

train / val

The Ref2VA video-to-video split H3-Train trains on, restricted to the selection: the first-frame version (references: [picture, video]). Train clips also passed 10-50 visible instances and at most 3 per scene; val is 8 whole held-out scenes, unfiltered. Each shard shards/hyper-NNNNN.tar has a .utix index (h3_train.utix); each sample is a folder K/ with

• sample.safetensors: picture f32 (1,390,96), normal f32 (1,14430,96), rgb f32 (1,14430,96), H3 video-VAE rows of the first RGB frame, the normal video and the RGB target; encoder_hidden_states bf16 (1,L,5120), the Qwen3-VL conditioning of the caption, picture and control video;
• meta.json: scene, traj, geometry and source paths at packing time;
• rope.safetensors: the clip's rel_cam_rope coverage (coverage_entries, coverage_instance, coverage_pixels, visible_raw_ids, object_poses, object_extents, camera_extrinsics). Its file-level parameters are the manifest's rope block. Its object boxes are PCA-oriented, from before H3-Train switched to world-axis boxes.

Stock webdataset splits these folder members into separate samples; group them by folder (for example with rename_files) or read through h3_train.utix.

render

render/<scene>/<scene>.tar holds K.mp4 for the scene's selected clips that LynnReal-Omni rendered: the normal video turned into RGB with the clip's caption (weights weight/standard, seed 7, CRF 18), H.264 yuv420p, no audio.

scene

zstd-compressed pickle: meshopt-encoded vertices (V,3) f32 and faces (F,3) u32, instance_ids (K,) and instance_offsets (K+1,): instance k owns faces offsets[k]:offsets[k+1], faces past the last offset belong to none. Metres, Z up. H3-Train's data/hyperscene_process/fetch_instance_mesh.read_instance_mesh decodes it.

Built by

workspace/H3-Train/data/hyperscene_v0/ (render_instances, pack_raw, pack_split, pack_render, pack.sbatch), 2026-10-09.