TRIAGE_Bench

TRIAGE-Bench: Testing Resolution of Inter-Authority Guideline Evidence

TRIAGE-Bench is a benchmark for evaluating how LLMs resolve conflicts between authoritative clinical knowledge sources. It covers 2,000 items across four conflict types, each with an explicit governing policy that defines policy-consistent correctness.

Key Features

• 2,000 benchmark items (500 per conflict type) grounded in real guideline and drug-label discrepancies
• Four conflict types: Jurisdictional, Temporal, Regulatory, Polypharmacy
• 2x2 factorial evaluation: Hint x Source design (NS, NS+Hint, SP, SP+Hint)
• 56,000 evaluation records across 7 frontier LLMs
• Two-stage error decomposition: Source Selection x Action Mapping bias classification
• 400 expert-validated subset (TRIAGE-Bench-Verified)

Dataset Structure

triage_bench/                  # Full benchmark (2,000 items)
  jurisdictional.json          # 500 items
  temporal.json                # 500 items
  polypharmacy.json            # 500 items
  regulatory.json              # 500 items
  all_items.json               # All 2,000 items combined
  bias_classification.json     # Source Selection x Action Mapping results
  model_outputs/               # 56,000 evaluation records (7 models x 4 modes)
  DATASHEET.md                 # Dataset documentation

triage_bench_verified/         # Expert-validated subset
  verified_400.json            # 400 items (100 per type)

triage_bench_code/             # Evaluation code
  src/                         # Pipeline, evaluator, prompts
  prompts/                     # Judge, eval, and hint prompt templates
  reproduction/                # GLMM R scripts for statistical analysis
  README.md                    # Code documentation

Item Format

Each benchmark item contains:

{
  "item_id": "CKPA-V3-CORE-001",
  "layer": "Jurisdictional",
  "disease": "Type 2 Diabetes",
  "clinical_decision_point": "Blood glucose control target for elderly patients",
  "visible_prompt": {
    "patient_scenario": "A 72-year-old male with T2DM and mild cognitive impairment...",
    "question": "What HbA1c target should be set for this patient?"
  },
  "hidden_labels": {
    "governing_source_id": "A",
    "clinical_action_canonical": "HbA1c target < 8.0%",
    "key_values_gold": {"HbA1c_target": "<8.0%"},
    "arbitration_basis": ["jurisdiction_match", "population_scope_match"]
  }
}

Conflict Types

Evaluation Protocol

The 2x2 factorial design crosses two factors:

Models Evaluated

GPT-5.6, Claude-Opus-4.8, GLM-5.2, Qwen-3.7, MiniMax-M3, DeepSeek-V4-Pro, Kimi-K2.6

Key Findings

1. Source dominates: Providing authoritative evidence increases policy-consistent response odds 6.5x (p<0.0001)
2. Hint has no net effect: Applicability reminders yield no overall improvement and interact negatively with Source
3. Two distinct bias mechanisms: Knowledge priority bias (wrong source) and reasoning strategy bias (wrong action from correct source) require different mitigations

Data Sources

• DXY Clinical Guidelines (26,719 guidelines)
• DXY Clinical Practices (3,434 decision modules)
• Chinese NMPA Drug Labels (153,332 labels)
• AMBOSS International Guidelines (69 PDFs)

Usage

import json

# Load full benchmark
with open("triage_bench/all_items.json") as f:
    items = json.load(f)

# Load verified subset
with open("triage_bench_verified/verified_400.json") as f:
    verified = json.load(f)

# Load model outputs
with open("triage_bench/model_outputs/gpt56_4modes.json") as f:
    gpt_results = json.load(f)

License

This dataset is released under the CC BY 4.0 license.

Ethical Considerations

• All patient scenarios are synthetic — no real patient data is used
• The benchmark evaluates policy consistency, not clinical optimality
• Governing policies are benchmark conventions for reproducible scoring
• Results should not be used as the sole basis for clinical deployment decisions