agentic-env-index

An index of materials for building agent environments: not a list of environments, and no content is copied here.

It covers 963,778,357 public files and records (4.03 PB) from 88 sources. Each row says where one item can be downloaded, how large it is, which hashes verify it and under which license it is offered. The files themselves stay at their sources.

Use

This repository gets new commits every day. Pin one commit so that the same query gives the same rows later. First note the current commit:

curl -s https://huggingface.co/api/datasets/Battam/agentic-env-index/revision/main | python3 -c "import json, sys; print(json.load(sys.stdin)['sha'])"

Put it in place of <commit> in the examples below.

1. Browse: the dataset viewer has one subset per source.
2. Query with DuckDB, for example the ten largest PDF files across all sources:

SELECT source, uri, size_bytes, lower(hex(sha256)) AS sha256 FROM 'hf://datasets/Battam/agentic-env-index@<commit>/data/*/*/*.parquet'
WHERE ext = 'pdf' ORDER BY size_bytes DESC LIMIT 10;

Single files at the same commit: https://huggingface.co/datasets/Battam/agentic-env-index/resolve/<commit>/meta/sources.json.