DexCanvas

DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation

Paper (arXiv:2510.15786)

DexCanvas is a dataset of 111,857 complete right-hand trajectories and 55,442,683 frames of rigid hand–object interaction. Every Parquet row is one trajectory with a shared time axis for hand kinematics, object pose, and per-frame contacts. For each trajectory, the repository also provides the complete dual-camera JPEG stream of the corresponding capture and shape-specific kinematic assets for replay.

Camera images are associated at the complete Raw trajectory level. The release does not claim that trajectory frame i corresponds to image i.

Summary

• 111,857 trajectories; 55,442,683 frames; 96,484,004 contact records.
• 32 action_label values spanning Cutkosky-style manipulation primitives, including palmar pinch, prismatic and lateral grasps, sphere grasps, extension, adduction, tripod, writing tripod, ring, distal type, and index-finger extension.
• 22 object categories spanning geometric primitives and household items: banana, bowl, cube1, cube2, cuboid1, cuboid2, cuboid3, cylinder1, cylinder2, cylinder3, cylinder4, cylinder5, cylinder6, cylinder7, iphone, largeclamp, mayonnaisebottle, pitcherbase, powerdrill, scissor, sphere1, sphere2.
• 21,079 deduplicated dual-camera captures; 22,212,754 JPEG frames.
• Eight right-hand MANO shape identities, each packaged as a 28-DoF floating-base URDF bundle; one canonical metre-scale mesh per object category.
• The package contains trajectory Parquet shards, image archives, capture/link metadata, kinematic assets, documentation, checksums, and a local Rerun replay tool.

What is included

• Trajectory data: data/train-*.parquet, with data/manifest.json recording shard coverage, byte sizes, ID ranges, and SHA-256 values.
• Capture images: images/<prefix>/<capture>.tar, WebDataset-compatible archives containing the complete original dual-camera JPEG streams and adjacent public JSON sidecars.
• Indexes: metadata/image_captures.parquet and metadata/trajectory_image_links.parquet describe captures and trajectory-to-capture links.
• Kinematic assets: assets/hands/hand_shape_*/mano_hand.urdf with referenced meshes; assets/objects/<category>/mesh.stl; see assets/manifest.json.
• Replay: replay/ contains a standalone Rerun player that reconstructs URDF link geometry through forward kinematics.
• Integrity: SHA256SUMS, all_rows_report.json, image_validation_report.json, clean_room_report.json, and public_string_audit.json.

Use the data

Streaming is recommended because a row contains a complete trajectory:

from datasets import load_dataset

trajectories = load_dataset(
    "Dex-GEM/DexCanvas",
    split="train",
    streaming=True,
)

trajectory = next(iter(trajectories))
print(trajectory["trajectory_id"], trajectory["num_frames"])

The Hub Dataset Viewer is disabled because complete nested trajectories can exceed its per-row display limit.

Row schema

trajectory_id
action_label
frame_rate_hz
num_frames
timestamps_s[T]

hands[]
  ├── hand_id
  ├── side
  ├── mano_shape[10]
  ├── global_position_world_m[T, 3]
  ├── global_rotation_world_rotvec_rad[T, 3]
  ├── mano_pose_rotvec_rad[T, 48]
  ├── joint_positions_world_m[T, 21, 3]
  ├── urdf_dof[T, 28]
  └── urdf_dof_target[T, 28]

objects[]
  ├── object_id
  ├── category
  ├── position_world_m[T, 3]
  └── rotation_world_rotvec_rad[T, 3]

contacts[T][]
  ├── hand_id
  ├── joint_name
  ├── object_id
  ├── total_force_world_n[3]
  ├── total_force_wrist_n[3] or null
  ├── total_force_joint_n[3]
  ├── total_force_object_n[3]
  └── points[]

timestamps_s is authoritative. frame_rate_hz is calculated from its median interval rather than copied from source metadata. Positions are in metres, axis-angle rotation vectors are in radians, and forces are in newtons.

See DATA_SCHEMA.md for the trajectory contract and IMAGE_DATA.md for capture indexes, WebDataset archives, and the alignment boundary.

Assets

The asset manifest is assets/manifest.json.

• assets/hands/hand_shape_*/mano_hand.urdf: eight public shape IDs covering every hand shape in the data;
• assets/hands/hand_shape_*/meshes/: all visual and collision meshes referenced by each URDF;
• assets/objects/<category>/mesh.stl: one canonical metre-scale object mesh per category.

Public shape IDs are matched to the mano_shape[10] stored in each trajectory. No operator identity is included.

Rerun replay

Install the replay dependencies:

python -m venv .venv
. .venv/bin/activate
pip install -r replay/requirements.txt

Replay by public trajectory ID:

python replay/replay_rerun.py \
  --data data \
  --assets assets \
  --trajectory-id traj_00000001 \
  --spawn

Export a portable recording:

python replay/replay_rerun.py \
  --data data \
  --assets assets \
  --trajectory-id traj_00000001 \
  --save trajectory.rrd

The player reconstructs hand-link geometry through URDF forward kinematics and displays the object mesh, URDF and recorded hand joints, contact points, and scaled contact-force arrows. It uses only this repository's Parquet data and relative assets; Lance and private storage are not required.

Dataset structure and integrity

• Split: train
• Trajectories: 111,857
• Frames: 55,442,683
• Contact records: 96,484,004
• Active hands per trajectory: 1
• Objects per trajectory: 1
• Unique hand shapes: 8
• Object categories: 22
• Deduplicated camera captures: 21,079 (two cameras each)
• Camera images: 22,212,754 JPEGs (approximately 889 GB of TAR archives)

Exact shard counts, byte sizes, ID ranges, and SHA-256 values are recorded in data/manifest.json. Package checksums are in SHA256SUMS. Every published row is covered by the aggregate structural scan in all_rows_report.json, image integrity is recorded in image_validation_report.json, source-independent replay is recorded in clean_room_report.json, and every public string field is covered by public_string_audit.json.

Limitations

• Only right-hand trajectories are present in this release.
• The repository provides one training split and is not a benchmark split definition.
• action_label preserves the source vocabulary; some labels are compact numeric codes while others are descriptive strings.
• Wrist-coordinate contact fields are null when that coordinate representation was not present in the source contract; missing vectors are never replaced with zeros.
• URDF geometry is a rigid-link approximation of the MANO surface. The recorded 21 MANO joint positions remain available for comparison in the data and replay.

License

This release is made available under the Open Data Commons Open Database License (ODbL) v1.0. See NOTICE_ASSETS.md for third-party attribution covering the hand and object assets.

Citation

If you use DexCanvas, please cite:

@misc{xu2025dexcanvas,
      title={DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation},
      author={Xinyue Xu and Jieqiang Sun and Jing Dai and Siyuan Chen and Lanjie Ma and Ke Sun and Bin Zhao and Jianbo Yuan and Sheng Yi and Haohua Zhu and Yiwen Lu},
      year={2025},
      eprint={2510.15786},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2510.15786},
}