Grid-Bench

Grid-Bench: Can LLMs Predict How Discrete Worlds Change?

Paper · Website · Code

Grid-Bench evaluates complete spatial state prediction. Each instance gives a language model an initial grid, explicit transition rules, and an ordered event history as text. The model predicts the complete final grid. Executable rules provide exact reference states, allowing both complete answers and individual cell values to be checked.

Explore the project website and use the companion code repository to simulate, generate, and score tasks. Maintainer: Zixiang Xu (xzx34). Original data and documentation are licensed under CC BY 4.0. Collected model answers are not distributed; see licensing and release scope.

Contents

Each configuration has equivalent Parquet and JSONL files. There are no training or validation splits. Instances cover PAINT, REFLECT, STENCIL, MIX, RELABEL, LAYER, MOVE, and COMPRESS on two-, three-, and four-dimensional grids. The histories have 0, 1, 8, 16, 32, or 64 events. Each history uses one operation family and may include SET assignments.

The six evaluated models, in fixed display order, are GPT-5.6 Sol, Claude 5 Sonnet, GPT-5.4 Mini, GLM-5, Kimi K2.5, and MiniMax M2.5. All six answer the same benchmark instances. Result rows describe evaluations of these questions, not additional benchmark questions. They contain measurements, not model answers.

Load the dataset

Install the optional loader with python -m pip install datasets, then load either configuration directly:

from datasets import load_dataset

root = "xzx34/Grid-Bench"
benchmark = load_dataset(root, "benchmark", split="test")
results = load_dataset(root, "results", split="test")
print(benchmark[0]["prompt"])

For offline evaluation and summary reproduction, download the full package. Install huggingface_hub with python -m pip install huggingface_hub, then run:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="xzx34/Grid-Bench",
    repo_type="dataset",
    local_dir="Grid-Bench-data",
)

This includes the manifest and supplementary evidence needed by the reproduction script. A downloaded directory can also replace root in the loading example.

The companion Python package uses grid_bench; its command is grid-bench. After installing the code repository, evaluate your own response file directly:

grid-bench evaluate --data /absolute/path/to/huggingface \
  --predictions responses.jsonl --view main --out results

Each response record contains public_id and the delivered response string. The default main view contains 1,152 instances; use workload for the 528-instance workload view or all for all 1,488 instances. Missing responses stay in the selected denominator. Duplicate and unknown IDs are rejected. See the code evaluation guide.

Alternatively, stream data/benchmark.jsonl or data/results.jsonl using Python's json module. JSON-string fields can be decoded with json.loads. The Parquet export preserves the same field values, including nulls.

A small example

Consider a 2×2 world whose nonzero cells are [0,0]=1 and [1,1]=1. A PAINT event centered at [0,0], with radius 1, replaces value 1 with value 2 inside the Manhattan ball. Only [0,0] changes. The complete answer is:

{"cells": [[[0, 0], 2], [[1, 1], 1]]}

Coordinates and axes are zero-based. Unlisted cells have value zero. The canonical output contains every nonzero cell exactly once in lexicographic coordinate order. The supplied prompt explains the applicable rule and includes a worked example. The illustration above is for documentation; it is not an additional test instance.

Benchmark fields

The machine-readable types are in schema.json.

Only prompt is sent to the model in the formal interface. Reference states and result fields are evaluation-side information.

The formal prompt is spatial-worlds-prompt/0.4. Source metadata retains its historical prompt identity, including version 0.3. The exporter renders 0.4 explicitly and verifies its hash against the formal request. Source hash fields follow the frozen serialization convention; per-file export hashes are recorded separately in manifest.json.

Construction, coverage, and sample units

The frozen selection contains 248 trajectories, each evaluated at six prefix lengths. The main design crosses eight operation families and three dimensions, with eight trajectories per family–dimension condition: 192 trajectories and 1,152 instances. Seven additional workload conditions supply 56 trajectories and 336 instances.

The 528-instance workload view combines those 336 instances with 192 reused main instances. Filter the membership fields rather than concatenating the views. There are 1,488 unique instances in total.

Main world shapes are [16,16], [6,6,7], and [4,4,4,4]. Initializers, alphabets, SET policies, and within-family workload targets are preserved in native metadata and the source evidence. All instances are synthetic. The English prompts describe mathematical spatial operations; no human participants or demographic groups are represented by the generated grids.

Prefixes from one trajectory share an underlying history. The paper's uncertainty intervals resample whole trajectories within the reported strata, using 2,000 bootstrap replicates. Neither the six prefixes nor six model answers create six independent reference trajectories.

Result fields and scoring

Join results.public_id to benchmark.public_id. model_id, model_name, and zero-based model_order identify the model. Field names and storage types are in schema.json; the principal groups are:

Identity fields shared with benchmark have the same meaning in both configurations. wrong_cells is null for invalid states; changed/unchanged-cell accuracies are null when the state is invalid or the respective reference subset is empty. The JSON-string fields decode to objects with json.loads.

Semantic exact match is one only when a valid complete state equals the reference everywhere. Incorrect, absent, and invalid answers remain in the denominator and score zero. A fixed extractor may locate a complete JSON answer within surrounding text; it does not repair state values. Strict exact match scores the delivered text without that extraction.

Cell-level error analyses use valid returned states, and error-extent analyses use valid incorrect states. Changed and preserved cells are defined by the initial and final reference values. Empty conditional subsets remain null; stored low-level invalid-state sentinel values should not be treated as valid cell-error observations.

The common collection conditions were one answer, no tools, a 128,000-token combined reasoning-and-answer ceiling, and a 7,200-second deadline. Temperature, top_p, and seed were omitted. Five interfaces requested high reasoning effort; MiniMax M2.5 used native always-on thinking. model-settings.json preserves the scientific settings. The export includes measurements for all final requests without filtering on correctness, validity, or termination. Answer text, predicted states, answer hashes, and free-text parsing errors are excluded.

Provenance and reproduction

provenance.json records source bindings and export transformations. manifest.json records exported member sizes and SHA-256 hashes. The export keeps original instance, prompt, and result identities while omitting provider infrastructure, account information, headers, and billing records. The original answers remain in the private research archive.

The evidence/ directory carries the frozen analysis and diagnostic tables. The companion code repository provides scripts/reproduce_release.py for reference replay and supported summary/plot reconstruction from measurements. It does not rescore the collected answers, which are not distributed. Those steps run offline. The original provider ledger is not needed for these checks and is not part of this dataset.

Citation and maintenance

Please cite Grid-Bench: Can LLMs Predict How Discrete Worlds Change? when using this dataset. The code repository also provides the paper citation in CITATION.cff.

@misc{xu2026gridbench,
  title = {Grid-Bench: Can {LLMs} Predict How Discrete Worlds Change?},
  author = {Xu, Zixiang and Song, Zirui and Zheng, Wanrong and Zhang, Chiyu and Huang, Yue and Peng, Yuyang and Zhang, Kexin and Chen, Jinkun and Chen, Xiuying and Zhang, Xiangliang and Liu, Yan and Itti, Laurent and Shyu, Frank and Luo, Xiaoqiang and Pandey, Sandeep and Liu, Xi and Simon, Luke and Hu, Xiyang and Zhao, Yue},
  year = {2026},
  eprint = {2610.22312},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  doi = {10.48550/arXiv.2610.22312},
  url = {https://arxiv.org/abs/2610.22312}
}

Report a data issue with its public instance ID, expected behavior, and affected release revision. Corrections require a new revision and changelog; frozen instances and result identities should not be replaced silently.