pico-robotics-annotated

Pico Robotics Dataset · Annotated Edition

Egocentric multimodal capture from a Pico VR headset + custom tracker rig — Annotated tier

Builds on the Advanced edition by adding coarse action segmentation. Every sequence is divided into labelled temporal segments, so the data can be used directly for action recognition, temporal segmentation, and behaviour-understanding tasks without an annotation pass of your own.

This is a gated dataset. Access requests are reviewed manually; submit one from the dataset page.

Editions

Scenes

from datasets import load_dataset

ds = load_dataset("skycn110/pico-robotics-annotated", "retail-shelf-a")

Directory structure

retail-shelf-a/
└── sample_0001/
    ├── head_left_camera_undistorted.mp4
    ├── head_right_camera_undistorted.mp4
    ├── depth/
    ├── audio_dual_channel.wav
    ├── undistort_camera.json
    ├── video_index.json
    ├── pointcloud/
    ├── pointcloud_world/
    ├── hand_landmarks.json
    ├── recording.mcap
    └── segments.json                       # Coarse action segmentation  ← new in this tier

Everything except segments.json is documented in the Advanced README.

Annotation format

segments.json

{
  "sample_id": "sample_0001",
  "scene": "retail-shelf-a",
  "fps": 30,
  "segments": [
    {
      "id": 0,
      "start_frame": 0,
      "end_frame": 148,
      "start_time": 0.0,
      "end_time": 4.93,
      "label": "approach_shelf",
      "notes": ""
    }
  ]
}

Segments within a sequence are contiguous and non-overlapping; frames that fit no class are labelled other.

Label set

TBD — list the action classes here, with a one-line definition and the frame count for each, for example:

Annotation protocol

Labels are deliberately coarse: boundaries are approximate and the class vocabulary is small. They are intended as weak supervision or as a starting point for finer annotation, not as a precision benchmark.

Quick start

import json

with open("retail-shelf-a/sample_0001/segments.json") as f:
    seg = json.load(f)

for s in seg["segments"]:
    print(f"{s['start_time']:6.2f}–{s['end_time']:6.2f}s  {s['label']}")

Clip a segment with the video:

import cv2

cap = cv2.VideoCapture("retail-shelf-a/sample_0001/head_left_camera_undistorted.mp4")
s = seg["segments"][0]
cap.set(cv2.CAP_PROP_POS_FRAMES, s["start_frame"])

for _ in range(s["end_frame"] - s["start_frame"] + 1):
    ok, frame = cap.read()
    if not ok:
        break

Intended uses

• Egocentric action recognition and temporal action segmentation
• Weakly supervised pretraining for behaviour understanding
• Video–language grounding of first-person activity
• Segment-conditioned imitation learning

Out of scope

Class balance is uneven and follows whatever occurred naturally during capture; some labels have very few segments. Boundaries are coarse and were not adjudicated frame-by-frame. Reporting state-of-the-art numbers on this label set without acknowledging those limits would be misleading.

License

Released under CC BY 4.0. When using this dataset, please attribute:

Pico Robotics Dataset by skycn110, licensed under CC BY 4.0

Citation

@misc{skycn110_pico_robotics_annotated,
  title  = {Pico Robotics Dataset: Annotated Edition},
  author = {skycn110},
  year   = {2026},
  url    = {https://huggingface.co/datasets/skycn110/pico-robotics-annotated}
}

Contact

• Data questions / collaboration: [email protected]
• Access requests: use the form on this dataset page (manually reviewed)
• Issues and Discussions: welcome on this repository