Hy-Embodied-0.5-VLA-Data

Abstract

We introduce Hy-Embodied-0.5-VLA (Hy-VLA) — an end-to-end Vision-Language-Action system that spans the full robot learning stack: data collection, model design, pre-training, supervised fine-tuning, RL post-training, and real-world deployment. Built on the Hy-Embodied-0.5 MoT backbone, Hy-VLA integrates a flow-matching action expert, a compact memory encoder for multi-frame history, and a delta-chunk action representation decoupled from embodiment-specific kinematics.

Powered by 10,000+ hours of high-fidelity UMI demonstrations collected via a custom fingertip interface with optical motion-capture, Hy-VLA achieves state-of-the-art results on the RoboTwin 2.0 benchmark (90.9% / 90.1% on Clean / Randomized) and demonstrates robust cross-embodiment transfer across four real-world robot platforms. Paired with FlowPRO preference optimization and an asynchronous inference framework, Hy-VLA establishes a scalable paradigm for continuous dexterous manipulation.

Overview

Hy-Embodied-0.5-VLA-Data is a large-scale bimanual manipulation dataset for training Vision-Language-Action (VLA) foundation models. Powered by 2000+ hours of high-fidelity demonstrations collected via a custom fingertip UMI device with optical motion-capture, it spans 70+ manipulation tasks. The dataset is released in Lance format compatible with LeRobot v3.0.

Note: The open-source release contains approximately 20% of the full corpus.

Dataset Statistics

Directory Structure

Each table is a self-contained LeRobot v3.0 dataset root:

table_000/
├── table_000.lance/          # Lance columnar data (5GB shards)
│   ├── _versions/
│   └── data/
│       └── ...-data-0.lance
└── meta/                     # LeRobot v3.0 metadata
    ├── info.json             # Table-level summary
    ├── stats.json            # Per-feature statistics
    ├── tasks.parquet         # Task ↔ index mapping
    └── episodes/             # Episode boundary parquet files

Data Schema

Each row (frame) contains:

Observations

Actions

Metadata

Usage

LanceTableReader reads a single Lance table (local or HF Hub):

from hy_vla.data.lance_dataset import LanceTableReader

# Local directory
reader = LanceTableReader(root="./table_000")

# HF Hub
reader = LanceTableReader(
    repo_id="tencent/Hy-Embodied-0.5-VLA-Data",
    table_name="table_000",
)

# Access
frame = reader[42]                        # single frame dict
episode = reader.get_episode(3)           # all frames of episode 3

Also compatible with raw lance, lancedb, and lerobot-lancedb (LeRobotLanceDataset).

Episode Visualization

# Use the HF Hub dataset, pick table_000 episode 666
python scripts/vis_umi_episode.py -t table_000 -e 666

# Local Lance root
python scripts/vis_umi_episode.py /path/to/Hy-Embodied-0.5-Data -e 0 --no-3d

Downloading Specific Tables

Due to the large total size (~18.8 TB), you may prefer to download individual tables:

from huggingface_hub import snapshot_download

# Download only table_000 (~890 GB)
snapshot_download(
    "tencent/Hy-Embodied-0.5-VLA-Data",
    allow_patterns="table_000/**",
    repo_type="dataset"
)

Citation

If you find Hy-VLA useful for your research, please cite:

@article{zhang2026hy,
  title={Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack},
  author={Zhang, He and Xiang, Lingzhu and Lin, Haitao and Huang, Zeyu and Wang, Minghui and Zhong, Dingyan and Dong, Yubo and Wu, Yihao and Rao, Yongming and Zhang, Dongsheng and others},
  journal={arXiv preprint arXiv:2606.14409},
  year={2026}
}

License

This dataset is released under CC-BY-4.0.