OneStreamer-1M

Adapted from the paper: task taxonomy on the left; streaming caption and QA construction pipelines on the right.

Task coverage

1. Dataset design and construction

A training dataset for perception, memory, and proactive response in streaming video interaction, introduced in OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction.

The curated collections include Streamo, JoyAI-VL-Interaction (JoyVL), MMDuet2, ViSpeak, StreamForest, VideoChat-OL, Seeker, and VST, alongside OneStreamer-Core. The appendix groups upstream sources by collection and task; a source list in the table describes that group rather than assigning every record to all listed sources.

OneStreamer-Core also includes alert/reminder annotations from TimeLens-100K and TimeLens2-93K.

OneStreamer-1M combines newly constructed OneStreamer-Core data with curated streaming-video training collections. Its supervision connects what has been observed, what should be retained, and when a response should be produced.

2. Video Sources

OneStreamer-Core (upload in progress): Video sources are being added to this repository under video_sources/OneStreamer-Core/<source>/, packaged as part-XXXXX.tar shards with matching part-XXXXX.manifest.json files.

For the other collections, please refer to the following media sources:

• Streamo: VideoMind (ActivityNet, QVHighlights, QuerYD, DiDeMo, TACoS, COIN, YouCook2), LLaVA-Video, Ego4D (Ego-TimeQA), HowTo100M (HowToStep).
• VST: VST video sources.
• MMDuet2: EgoExoLearn.
• StreamForest — videos: VideoMind (ActivityNet, Charades, COIN, HiREST, QuerYD), InternVid, AVA, GOT-10k, LaSOT.
• StreamForest — images: COCO (RefCOCO), Visual Genome, SA-1B (AS-V2).
• StreamForest — OnlineIT-Drive: TS_Retrieval, TR_Analysis, TOI_Recognition.
• ViSpeak: ViSpeak-Instruct videos, Social-IQ, FunQA, OOPS media preparation.
• Seeker: LLaVA-Video, LongVideoDB, Video-o3-Selfbuilt.
• JoyVL: Molmo2-VideoCapQA, Molmo2-AskModelAnything, TimeLens, VideoGPT+, Shot2Story, HoloAssist.
• VideoChat-OL: AVA.

3. Metadata release and usage

Annotations are organized by task family, collection, and task in JSONL files. Each line is one training record and may contain multiple inputs and outputs. Records may share a video, and some tasks use images.

The release provides readable annotations and relative media references. See Video Sources for the OneStreamer-Core upload status and upstream media links. Media paths identify the expected dataset layout; processed or cropped assets may require a matching media package or a mapping from their original source videos.

Files and fields

README.md
assets/
  data_pipeline.png
Proactive_Caption_Memory/<collection>/<task>/*.jsonl
Proactive_Interaction/<collection>/<task>/*.jsonl
Proactive_QA/<collection>/<task>/*.jsonl
Perception_and_Memory_QA/<collection>/<task>/*.jsonl

Time representation. silence and standby use interval: [start, end]; adjacent intervals of the same state may be merged. response, observe, summary, and ordinary QA answer outputs use at and text. A response is a single output event. The intervals retain the original state labels; they do not imply repeated response text. Source-specific loss masks and training-only sampling settings are not included.

Times are in seconds and retain the source annotation convention. In this metadata format, at is the endpoint of the associated input observation window; it is not necessarily the precise time of an event described in the answer. null means that an explicit time was unavailable. Answer text, including any timestamps within it, is preserved.

Media references. Video and image paths include their filename extension. Frame paths end at the logical video directory, with a trailing /; video_name is the directory name. Sequences originally stored as one-second MP4 segments use the intended 4 FPS frame representation in this release, with filenames such as 00000_00.jpg, 00000_25.jpg, 00000_50.jpg, and 00000_75.jpg. These frame assets must be prepared separately. Existing frame sequences retain their annotated FPS, including sources at other frame rates.

A proactive QA example, showing selected fields from one record:

{
  "video_name": "ytb_ls8tmkfD7S0",
  "video_path": "LLaVA_Video/1_2_m_youtube_v0_1/liwei_youtube_videos/videos/youtube_video_2024/ytb_ls8tmkfD7S0/",
  "media_type": "sequence",
  "sequence": {"storage": "frames", "fps": 4, "filename_pattern": "{seconds:05d}_{centiseconds:02d}.jpg"},
  "video_ranges": [[0.0, 6.0]],
  "prompts": [
    {"id": "p1", "kind": "text", "at": 5.0, "text": "What event is the person with long blonde hair being interviewed at in front of a poster for 'The Beguiled'?"}
  ],
  "timeline": [
    {"type": "silence", "interval": [0.0, 4.0]},
    {"type": "standby", "interval": [4.0, 5.0]},
    {"type": "response", "at": 6.0, "text": " The person is being interviewed at a red carpet event in front of a poster for 'The Beguiled.'", "prompt_id": "p1"}
  ]
}

To inspect the metadata after downloading the dataset repository:

import json
from pathlib import Path

dataset_root = Path(".")  # downloaded dataset repository
annotation = next((dataset_root / "Proactive_QA").rglob("*.jsonl"))
with annotation.open(encoding="utf-8") as f:
    record = json.loads(next(f))

print(record["prompts"])
print(record["timeline"])

# Resolve a reference against your own media root.
media_root = Path("/path/to/downloaded/media")
asset = record["videos"][0] if "videos" in record else record
media_path = media_root / asset["video_path"]

Different source datasets can be stored under separate roots by mapping their path prefixes.

Citation

@misc{zeng2026onestreamerunifyingperceptionmemory,
  title={OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction},
  author={Xiangyu Zeng and Yuandong Yang and Zhiqiu Zhang and Yuhan Zhu and Xinhao Li and Qingyi Si and Dingyu Yao and Changlian Ma and Haoran Chen and Xinyu Chen and Yansong Shi and Junhao Zhou and Yifei Li and Jun Zhang and Chuanyu Qin and Chenxu Yang and Xinlei Yu and Kun Ouyang and Yuchen Shao and Qianshan Wei and Changhai Zhou and Jun Gao and Jiaqi Wang and Limin Wang},
  year={2026},
  eprint={2610.01762},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.01762}
}