ContextBench

ContextBench measures what an agent keeps, updates, and offloads when its input does not fit in its context window. Each task streams a sequence of operations as ordinary user turns, more than the context budget holds, and the reward is read from the agent's own messages: what survived in its context, not what it wrote to disk.

Paper: https://huggingface.co/papers/2609.37725

Families

The all config holds every instance. This release has 256 instances: 64 kv_store, 56 log_triage, 80 needle_retention, 56 sudoku_sketchpad (32 levels, seeds 42-49), with a context budget of 32768 tokens and a reserve of 2048.

Columns

Each operation is shown to the agent as === {header} ({remaining} op(s) remaining after this) === followed by its body.

Usage

With datasets

import json
from datasets import load_dataset

ds = load_dataset("context-language-model/ContextBench", "kv_store", split="test")   # or "all"
row = ds[0]
ops = row["chunks"]            # list of {"index", "header", "body"}
gold = json.loads(row["gold"])

To evaluate an agent from Python, install the contextbench package, write an instance to disk, and pass your agent to contextbench.run:

from pathlib import Path
from contextbench import run

inst = Path("instances") / row["task_id"]
inst.mkdir(parents=True, exist_ok=True)
(inst / "chunks.jsonl").write_text(
    "".join(json.dumps(c) + "\n" for c in row["chunks"]))
(inst / "gold.json").write_text(row["gold"])
(inst / "meta.json").write_text(row["meta"])

result = run(my_agent, inst, budget=row["budget"], reserve=row["reserve"],
             out_dir=Path("runs") / row["task_id"])

With Harbor

The harbor/ directory holds one Harbor task per instance, and registry.json lists the datasets contextbench (every task) and contextbench-<family> (for example contextbench-kv-store). Download the repository, then run Harbor from inside it:

hf download context-language-model/ContextBench --repo-type dataset --local-dir contextbench-hf
cd contextbench-hf
harbor jobs start --registry-path registry.json -d contextbench \
    -a contextbench.harbor.agents:Terminus2Stream -m <model>

Use -d contextbench-sudoku-sketchpad for one family, -i <task_id> for one task, -p harbor/ to run the directory without the registry, or -a oracle to check a task with its reference solution. Agents that deliver operations into a running conversation ship with the contextbench package (contextbench.harbor.agents).

License

Apache-2.0.