PROWBench

PROWBench Dataset

PROWBench evaluates whether video models follow program-specified motion and interactions. It contains 170 episodes, 200 camera views, and 600 proxy videos, with prompts, recorded world states, and RGB first frames where available.

Paper · Project · Gallery · Code · Evaluation guide

Download

pip install -U huggingface_hub
hf auth login
hf download AlayaLab/PROWBench --repo-type dataset --local-dir PROWBench

Data

Each view has three proxy representations: white (coarse 3D), mixed (semantic proxy), and colored-obb (3D boxes). Multi-view episodes have four synchronized cameras.

The full Viewer split contains 600 rows, one per camera–representation pair; the other splits are subsets. Full prompts are available through prompt_file. preview_kind distinguishes RGB first-frame references from proxy previews.

A typical episode:

episodes/<group>/<episode>/
├── manifest.json
├── scene/                              # Recorded world states
├── cameras/                            # Camera records
└── views/<camera>/
    ├── <representation>/video.mp4
    ├── <representation>/prompt.txt
    ├── prompt-context/first_frame.png   # When available
    └── visibility.json                 # Multi-view

Long-video prompts use JSON files or a prompts/ directory of windowed prompts. Use prompt_file and the adjacent sample.json for exact input paths, including versioned episodes.

This release contains conditioning inputs and annotations; raw depth maps and generated model outputs are not included. Release metadata · Sample metadata.

License

This dataset is released under the MIT License.

Citation

If you find PROWBench helpful for your research, please consider citing our paper:

@misc{huang2026prowbench,
  title={PROWBench: Do Video Models Render What the Program Specifies?},
  author={Zheng-Hui Huang and Guixu Lin and Yu-Ju Tsai and Jian-Kai Zhu and Fengbo Lan and Yu-Lun Liu and Yung-Yu Chuang and Kaipeng Zhang and Zhixiang Wang},
  year={2026},
  eprint={2610.02205},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.02205}
}