terminal-data-pointers

A catalog of 419,170,334 public files (259 TB; size not published for 20,783,849 files) from 24 sources: web documents, Python packages, research data, chess games, forensic scenarios (disk, memory and network images), Q&A posts, mixed file types, mixed documents, presentations, issue tickets, source packages, U.S. government documents, chess engine evaluations, spreadsheets, R packages, emails, forum posts, system logs, security vulnerability records (CVE), chess puzzles, chat logs, JavaScript packages and meeting transcripts. Every row says where one file can be downloaded, how large it is, which hashes verify it and under which license it is offered. This repository holds only the catalog (about 65.4 GB); the files stay with their publishers.

Sources

A file is one download, as the source publishes it; an archive counts once.

Training use and Redistribution summarize each license as yes, conditional, no or unknown (mixed when the files of a source differ); docs/licenses.md defines each tier and where the license of a file is written. This is a summary, not legal advice: check the upstream terms before use.

Quick start

Browse, search and filter every row in the viewer at the top of this page, or load the catalog:

from datasets import load_dataset

catalog = load_dataset("Battam/terminal-data-pointers", split="train")  # all sources
end_of_term_web_archive = load_dataset("Battam/terminal-data-pointers", "end-of-term-web-archive", split="train")  # one source

usable = catalog.filter(lambda row: row["training_ok"] == "yes")
print(usable.num_rows, "files,", round(sum(b or 0 for b in usable["size_bytes"]) / 1e12, 2), "TB")

Then download each uri and check it against size_bytes and the hashes (see docs/retrieval.md).

What is inside

By content

By license

Source details

Using the catalog

• manifests/<source>/<batch>/: the source of truth. collection.json describes the batch and part-NNNNN.jsonl.gz holds its pointers, one line per file.
• catalog/<source>.parquet: a derived view of the manifests, one flat table per source, shown in the viewer above; where it differs from the manifests, the manifests are right. Sources published while they are collected have the table split into files of up to 100 parts instead, under catalog/<source>/<batch>/.
• catalog/curated/ and catalog/starter/: per use tag, the pointers left after curation, and a starter pack drawn from them.
• assets/: the figures on this page.

Every field of the pointers, the batch headers and the catalog tables is described in docs/fields.md.

How to download a file and verify it against its size and hashes, including byte ranges, http_payload and api_body, is in docs/retrieval.md.

The license tiers, and the license of this catalog itself, are defined in docs/licenses.md.

To pick files by use, file type, license or size, start from the curated tables catalog/curated/<tag>.parquet or the starter packs catalog/starter/<tag>.parquet: docs/selecting.md has DuckDB queries and the download command, and docs/curation.md says what the curated tables leave out and why.

Changelog

License

Each file keeps the license of its source: see the license tiers and the license text of each source above, and the license columns of the catalog. This catalog itself (addresses, sizes, hashes and descriptive text) is released under CC0 1.0; that dedication does not extend to the files it points to. See docs/licenses.md.

The catalog itself contains no personal information. Some of the files it points to, such as email archives and Q&A posts, contain names and contact details that their authors published; follow each source's terms when you use them.

Questions and corrections are welcome in the Community tab.