GiT_hdf5

Grounded in Time（GiT）： a multi-source dataset and benchmark for temporal grounding under perceptual ambiguity in robotic manipulation.

[ Website ] [ GitHub ] [ arXiv ] [ HuggingFace ] [ ModelScope ]

Overview

Grounded in Time is a benchmark and dataset for temporal grounding under perceptual ambiguity in robotic manipulation.

Its tasks are deliberately built so that the object a command refers to cannot be identified from the current frame alone. Each scene contains several perceptually near-identical objects, and what separates them is not appearance but the temporal structure of the episode — which test tube was contaminated first, which container was filled last, which part was handed over just now. A model that grounds the instruction on the present frame alone will pick the wrong object; resolving the reference requires tracking and reasoning over the episode's history.

The benchmark ships three complementary subsets — real, sim, and umi — totaling 4680 episodes across 18 task families, collected on an AgileX dual-arm platform. Every episode pairs a natural-language instruction with the temporal structure of the task:

• an explicit split between a demonstration segment and a subtask segment, so evaluation can be restricted to the ambiguous portion;
• subtask segment annotations (description over [frame_start, frame_end]) giving the temporal ground truth of each stage;
• synchronized multi-view RGB, optional depth, and full proprioceptive state, so the same episodes serve both as a benchmark and as training data.

Because the annotations record which stage disambiguates the reference and when it resolves, grounding can be probed directly rather than inferred from task success alone.

At a glance

Repository structure

GiT/
├── real/                      # 1,260 real-robot episodes (18 tasks × 70)
│   ├── bio1_episode0.hdf5
│   ├── …
│   └── industrial6_episode69.hdf5
│   
├── sim/                       # simulation episodes
│   ├── train/                 # 1,440  (9 tasks × 160)
│   ├── val/                   #   180  (9 tasks ×  20)
│   └── test/                  #   540  (9 tasks ×  60)
│  
└── umi/                       # 1,800 UMI-style episodes (18 tasks × 100)
    ├── train/                 # 1,516
    ├── val/                   #   154
    └── test/                  #   130

On the Hub, the three subsets live under the real/, sim/, and umi/ paths of this repository. The backup_* directories are internal snapshots and are not part of the released data.

Task families

All three subsets use a bio* / house* / industrial* task taxonomy (abbreviated ind* inside sim/):

HDF5 schema

Every episode is a single self-contained HDF5 file named {task}_episode{N}.hdf5. Image streams are stored as encoded bytes ((T,) variable-length uint8 arrays in HDF5; exposed as object arrays by h5py), not decoded image tensors. Decode with cv2.imdecode / PIL.Image.open on demand. The concrete examples below are real/bio1_episode0.hdf5 (T=743), sim/train/bio2_episode100.hdf5 (T=434), and umi/train/bio1_episode1.hdf5 (T=782).

1. real/ — real-robot teleoperation

{task}_episode{N}.hdf5
├── instruction                     scalar, UTF-8 str  # task instruction
├── size                            scalar, int64      # number of frames T
├── timestamp                       (T,)      float64  # Unix seconds
├── arm/
│   ├── endPose/{puppetLeft,puppetRight}           (T, 6) float64  # x,y,z,roll,pitch,yaw
│   ├── jointStatePosition/{puppetLeft,puppetRight} (T, 7) float64
│   ├── jointStateVelocity/{puppetLeft,puppetRight} (T, 6) float64
│   └── jointStateEffort/{puppetLeft,puppetRight}   (T, 7) float64
├── camera/
│   ├── color/{front,left,right}    (T,)      vlen uint8  # JPEG bytes
│   │     decoded: front 1280×720; left/right 640×480, RGB
│   └── depth/{front,left,right}    (T,)      vlen uint8  # PNG bytes
│         decoded: front 1280×720; left/right 640×480, uint16
└── subtask/
    ├── demo_end                    scalar, int64  # last frame of the demonstration segment
    ├── frame_start                 (K,)   int64
    ├── frame_end                   (K,)   int64
    └── description                 (K,)   UTF-8 str

The arm stream keys in the files are puppetLeft and puppetRight. In the supplied real example, K=4, demo_end=115, and the subtask ranges are 116–283, 284–369, 370–562, 563–684; the remaining frames are an unannotated tail. The supplied sim example has K=4, demo_end=56, and ranges 57–160, 161–261, 262–389, 390–433.

Root attributes: alignment_reference ("camera/color/front"), end_pose_fields ("x,y,z,roll,pitch,yaw"), max_time_delta_sec, source_episode_dir.

Segment semantics. Each episode is split in two:

• Demonstration segment — frames [0, demo_end], inclusive. This is the human demonstration that the policy is meant to learn from.
• Subtask segment — frames [demo_end + 1, size − 1]. At least for real, the convention frame_start[0] == demo_end + 1 holds, and subsequent segments are contiguous: frame_end[k] + 1 == frame_start[k + 1]. Some files also carry a short unannotated tail after the last segment.

Temporal downsampling. The demonstration segment has been 4× temporally downsampled: adjacent frames are ≈0.133 s apart (≈7.5 FPS), while the subtask segment runs at the native ≈0.033 s (≈30 FPS). timestamp is authoritative — always derive dt from it rather than assuming a constant frame rate. Take this into account when mixing this data with other 30 FPS corpora.

2. sim/ — simulation

The supplied sim example has the same layout and root attributes as real/, with one data difference:

• arm/jointStateEffort/{puppetLeft,puppetRight} are empty — shaped (T, 0) in sim/train/bio2_episode100.hdf5. Do not consume effort for sim/.

The demonstration segment is 4× downsampled (≈0.133 s), and the subtask segment runs at native rate (≈0.033 s, with some tasks differing slightly — timestamp is authoritative).

sim/ additionally ships a CSV sidecar annotation set at sim/subtask/, covering five tasks (house1, house3, house5, bio4, bio5). These CSVs encode frame ranges as strings such as frame82-frame199. Where both exist, the embedded subtask/ group and the CSV agree in coverage but use different boundary conventions — prefer the embedded group, which is consistent across all subsets.

3. umi/ — UMI-style policy learning

umi/ uses a different, action-rich schema geared toward policy learning rather than replay, and is split by held-out category:

{task}_episode{N}.hdf5
├── instruction                     scalar, UTF-8 str  # task instruction
├── size                            scalar, int64      # number of frames T
├── timestamp                       (T,)      float64
├── arm/
│   ├── endPose/{puppetLeft,puppetRight}           (T, 6) float32  # x,y,z,roll,pitch,yaw
│   ├── gripperDistance/{puppetLeft,puppetRight}   (T,) float32
│   └── gripperWidth/{puppetLeft,puppetRight}      (T,) float32
├── camera/color/{front,left,right} (T,)     vlen uint8  # JPEG bytes
│     decoded: front 1280×720; left/right 1920×1080, RGB
├── observation/
│   └── bimanualRelativePose/rightFromLeft (T, 6) float32
└── action/
    ├── gripperTarget/{puppetLeft,puppetRight}      (T, 16) float32
    ├── relativeTrajectory/{puppetLeft,puppetRight} (T, 16, 6) float32
    └── validMask/{puppetLeft,puppetRight}          (T, 16) bool

Quick start

import io
import h5py
import numpy as np
from PIL import Image

path = "real/bio1_episode0.hdf5"

with h5py.File(path, "r") as f:
    T = f["size"][()]
    print("instruction :", f["instruction"][()].decode())
    print("frames      :", T)

    # --- language ---
    demo_end = f["subtask/demo_end"][()]
    for s, e, d in zip(f["subtask/frame_start"][:],
                       f["subtask/frame_end"][:],
                       f["subtask/description"][:]):
        print(f"  [{s:4d}, {e:4d}] {d.decode()}")

    # --- proprioception ---
    left_pos = f["arm/jointStatePosition/puppetLeft"][:]   # (T, 7)

    # --- images (stored as encoded bytes) ---
    t = 0
    rgb_front = np.array(Image.open(io.BytesIO(f["camera/color/front"][t])))  # (720, 1280, 3)
    depth_front = np.array(Image.open(io.BytesIO(f["camera/depth/front"][t])))  # (720, 1280)

    # --- timing: always read dt from timestamp, never assume a fixed FPS ---
    dt = np.diff(f["timestamp"][:])
    print("demo dt ≈", np.median(dt[:demo_end]))       # ≈ 0.133 s
    print("subtask dt ≈", np.median(dt[demo_end:]))    # ≈ 0.033 s

Citation

If you find this dataset and benchmark useful, please cite:

@article{wang2026grounded,
  title={Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation},
  author={Wang, Yi and Yang, Yang and Xu, Guangqi and Lin, Sumin and Kang, Ning and Lu, Pengxiang and Chen, Xiaotong and Xue, Zeyu and Deng, Ping and Liu, Xing and others},
  journal={arXiv preprint arXiv:2610.04255},
  year={2026}
}

License

Released under the Apache-2.0 license. See LICENSE for details.

Acknowledgments

We thank the ManiSkill-3 team for their open-source simulation platform and the work that makes this benchmark's manipulation environments possible.

Contact

For questions, issues, or collaboration, please open an issue on GitHub.