constraintdial

A deterministic, local-EVM benchmark for smart-contract security tasks, built as families of tasks that share one hidden objective and differ only in labeled protocol constraints.

Paper: ConstraintDial: A Lifelong Factory that Builds Smart Contract Exploit Tasks (under review).

Inventory

Every task runs on a private in-memory EVM (chainId 31337, zero gas price, no external network). The chain is rebuilt deterministically from bootstrap.py on every start.

Layout

huggingface/
  task_manifest.csv          # 3,045 rows: task_id, case_id, version, task_type, ...
  tasks/                     # 3,045 task directories
    <id>-case/               # exploit task + shared environment files
    <id>-detect-case/        # detect task
    <id>-patch-case/         # patch task
  infra/revm_node.py         # shared JSON-RPC chain server
  vendor/                    # Linux CPython 3.10 wheels (pyrevm, web3, ...)

Task ids follow 4000-<family>[-<variant>]-<type>-case, e.g. 4000-0002-eq-detect-case (family 0002, equivalent variant, detect). A family's five variants share the hidden objective J and the judge; they differ in the protocol constraints (defense coverage, funding, roles, cross-contract gates — the K1/K2 mutations).

Task types

• exploit — the directory is <id>-case. The agent gets the deployed contracts, an unlocked attacker account (info.json), and the public objective, then must reach a declared end state on the local chain. run.sh serves JSON-RPC; judge.sh grades only final public chain state (balances, supply, reserves, nonce) and accepts any attack path that reaches that state.
• detect — the agent must name the contract file/function that the hidden evaluator treats as the weakness (submission/detect.json). Grading is file-label scoring against the released gold labels (construction/gold/ in the companion repo). The judge/judge.py inside a detect directory is a placeholder that fails cleanly on-chain; it is not the grader.
• patch — the agent edits the Solidity sources so that ordinary protocol use still works and the reference attack no longer reaches its target. The hidden evaluator rebuilds the chain from the edited sources.

Running a task

# Python 3.10 required (vendor/ supplies the CPython 3.10 build of pyrevm)
cd tasks/4000-0001-case
bash run.sh            # terminal 1: serves http://127.0.0.1:8545
bash judge.sh          # terminal 2: writes results/result.json

run.sh and judge.sh locate infra/revm_node.py by walking upward from the task directory, so the tasks tree can be relocated freely. Override the port with PORT / RPC_URL.

Provenance and honesty notes

• Each family is a faithful local reconstruction of one historical on-chain incident (DeFiHackLabs, Web3Bugs/Code4rena, BlockSec). The reconstruction keeps the security-relevant dependencies (roles, approvals, capital, liquidity, callbacks, ordering) as recorded in complexity_fidelity.json; provenance metadata is in seed.json.
• task_manifest.csv records reference_accepted=False and acceptance_status=not_checked for every row: the released inputs have passed construction gates and self-checks, but a full independent reference-and-grading acceptance pass is tracked separately. Treat the inventory as prepared inputs, not certified outcomes.
• Attack helpers, reference solves, private oracles, and evaluator-side source notes are not part of this dataset.
• Solidity contract names and file names are neutral; prompts never name the historical project.

License

Task code and data: Apache-2.0. Third-party runtime packages in vendor/ retain their own licenses (MIT/BSD/Apache-2.0; see each vendor/*.dist-info). The reconstructed incidents derive from public disclosure sources; attribution is recorded per family in seed.json.