MADBench-full

MADBench-Full: Multi-Agent Anomaly Detection with Tool Use

MADBench-Full contains 5,200 fully labeled execution traces from five LLMs solving synthetic escape-room tasks. The updated release adds paired tool-enabled and no-tool runs over two clue domains, making it possible to study tool selection, argument construction, tool execution failures, error recovery, and downstream error propagation in a controlled multi-agent system.

Dataset Overview

Four models have 1,200 traces each: 2 clue domains × 2 tool modes × 3 temperatures × 100 rooms. claude-opus-4-8 has 400 traces at temperature 0.0 only.

Models Evaluated

Clue Domains

Each room embeds one or more reasoning problems inside escape-room clues:

• gsm-hard uses challenging arithmetic word problems.
• livecodebench uses chained Python programs. Agents must distinguish runnable code from syntactically broken distractors and compute the valid program's result.

The resulting number is a delta that must be applied to a physical instrument. Instruments include thermometers, clocks, compasses, and scales; delta units cover temperature, time, angle, and mass conversions.

Tool-Use Experiment

Every model/domain/temperature configuration is evaluated in paired conditions:

• unit_tools: false: the item manager performs unit conversion directly in its response.
• unit_tools: true: the item manager may issue up to three calls per action to convert_time, convert_temperature_delta, convert_angle, or convert_mass.

A tool request is stored as its own UNIT_TOOL_CALL trace step. The step records the selected tool, arguments, execution result or error, expected tool behavior, and an anomaly label. Run-level tool summaries appear under failure_report.unit_tool_summary.

Across the updated release, tool-enabled runs contain 2,900 tool calls: 2,418 successful executions and 482 failed executions. Tool availability does not guarantee that a tool is called in every trace.

Difficulty Levels

Multi-puzzle rooms can include a planner that selects a safe puzzle order under item lock/unlock constraints. A room succeeds only when every required puzzle is solved.

Multi-Agent Trace

Depending on difficulty and configuration, a trace may contain these actions:

Each agent step includes the full input, free-text output, structured output, schema errors, latency/token statistics when applicable, and deterministic verification against ground truth.

Escape Rate Results

Escape rate is the fraction of rooms in which all puzzles were solved. Values aggregate temperatures; claude-opus-4-8 contains temperature 0.0 only. Denominators are 300 per cell for all other models and 100 per cell for Claude Opus.

These are descriptive benchmark results, not estimates of general model capability. Tool and no-tool outcomes can differ because the tool interface introduces both assistance and new failure modes.

Dataset Structure

MADBench-full/
├── metadata.jsonl
├── MADBench-full-croissant.json
├── README.md
├── claude-opus-4-8/
├── claude-sonnet-4-6/
├── deepseek-reasoner/
├── gpt-4.1/
└── gpt-5.4/
    └── {clue_domain}/
        └── run_{timestamp}_{domain}_{tool|no_tool}_{model}_{temperature}/
            └── {timestamp}_EscapeRoom_{room_id}_{escaped|failed}.json

The model folders contain only the 5,200 published JSON traces; local run logs and experiment-tracking caches are not part of the dataset.

metadata.jsonl Schema

The dataset viewer loads metadata.jsonl, with one row per trace:

Trace JSON Schema

Each trace contains the complete execution record for one room:

{
  "config": {
    "system": {"architecture": "sequential"},
    "llm": {
      "provider": "openai",
      "model": "gpt-4.1",
      "temperature": 0.0,
      "max_tokens": 2048
    },
    "agents": ["... per-role prompts ..."],
    "benchmark": {
      "clue_domain": "gsm-hard",
      "unit_tools": true,
      "max_unit_tool_calls": 3
    }
  },
  "room": {"room_id": "room_0000", "difficulty": "easy", "puzzles": ["..."]},
  "trace": [
    {
      "action": "OBSERVE_CLUE",
      "agent": "observer",
      "input": "...",
      "output_message": "...",
      "output_structured": {},
      "schema_errors": {},
      "call_statistic": {
        "duration": 2.87,
        "input_tokens": 549,
        "output_tokens": 147,
        "timed_out": false,
        "superlong_reasoning": false
      },
      "verification": {"status": "correct", "error_type": ""}
    }
  ],
  "failure_report": {
    "schema_failures": [],
    "eval_results": ["correct"],
    "unit_tool_failures": [],
    "unit_tool_summary": {}
  },
  "escaped": true,
  "puzzles_solved": 1,
  "puzzles_total": 1,
  "timing_sec": 19.8
}

A UNIT_TOOL_CALL step additionally contains parent_action_id, call_index, tool_execution, and tool-specific verification fields such as expected/actual tool names, arguments, results, and anomaly_label.

Anomaly Signals

How to Load

import json
from pathlib import Path

root = Path("MADBench-full")

with (root / "metadata.jsonl").open() as f:
    metadata = [json.loads(line) for line in f]

# Paired tool-use analysis for one model and domain.
subset = [
    row for row in metadata
    if row["model"] == "gpt-5.4"
    and row["clue_domain"] == "livecodebench"
]

# Load a complete trace from its metadata row.
with (root / subset[0]["file"]).open() as f:
    trace = json.load(f)

tool_calls = [step for step in trace["trace"] if step["action"] == "UNIT_TOOL_CALL"]

Limitations

MADBench-Full is a synthetic, controlled benchmark. Its labels are produced by deterministic comparison with generated ground truth rather than human annotation or an LLM judge. Results characterize behavior in this specific multi-agent escape-room environment and should not be treated as a general ranking of model quality or real-world tool-use reliability.

The task content and model outputs may inherit limitations from the underlying clue sources and evaluated models. No personal or sensitive human-subject data is included.