VEX-Bench

Website: Explore the interactive VEX-Bench results.

VEX-Bench scores the verification burden of LLM-generated misinformation after elicitation, not jailbreak success alone:

• D1 Checkability
• D2 Harm Significance
• D3 Source Credibility Signals
• D4 Imposter Legitimacy (field: presentation)
• D5 Verification Cost

7 models × 7 methods × 2 tasks × 6 domains × 10 topics = 5,880 cells. No train/test split: every config uses split full. This snapshot is sanitized and does not include rewrite source-article bodies.

Models: anthropic/claude-sonnet-4, deepseek/deepseek-v4-pro, google/gemini-3.1-pro-preview, moonshotai/kimi-k2.5, openai/gpt-5.4, qwen/qwen3.5-flash-02-23, x-ai/grok-4.1-fast.

Methods: direct, isc, misinfoqa, poisonedrag, disinfocap, pap, jailnewsbench.

Tasks: fabrication (article from a topic) and rewrite (misleading variant of a real source). Domains: health, politics, public_safety, finance, identity_groups, environment.

Files

Loading

Judge files share (model, method, task, domain, topic) with model as provider__name and topic as a 60-character filename slug. articles uses provider/name and the full topic title. Nested objects and arrays are JSON strings.

import json
from datasets import load_dataset

vex = load_dataset("hanxunh/VEX-Bench", "vex", split="full")
articles = load_dataset("hanxunh/VEX-Bench", "articles", split="full")
ccfc = load_dataset("hanxunh/VEX-Bench", "ccfc", split="full")
validation = load_dataset("hanxunh/VEX-Bench", "validation", split="full")


def article_key(row):
    return (
        row["model"].replace("/", "__"),
        row["method"],
        row["task"],
        row["domain"],
        row["topic"][:60].replace(" ", "_").replace("/", "_"),
    )


def judge_key(row):
    return (row["model"], row["method"], row["task"], row["domain"], row["topic"])


scored_vex = [row for row in vex if row["judge_model"] != "placeholder"]
article_by_key = {article_key(row): row for row in articles}
row = next(r for r in scored_vex if judge_key(r) in article_by_key)
parsed_articles = json.loads(article_by_key[judge_key(row)]["articles"])
claims = json.loads(ccfc[0]["claims"])
paper_sample = next(r for r in validation if r["paper_included"])
human1_d1, human2_d1 = paper_sample["human1_d1"], paper_sample["human2_d1"]
# claims[i]["sources"] is a list of URL strings

The topic slug mapping is unique across the 60 topics; do not join on raw topic strings. Do not use vex[0] as a scored example: early rows include placeholders with parse_ok == True.

VEX and JailNewsBench include 1,263 placeholder rows for invalid generations. Those values are not judge ratings, even when parse_ok is true. Aggregate with judge_model != "placeholder".

Human–judge validation

validation contains Human 1 and Human 2 D1–D5 scores, alongside separate GPT-5.2, Gemini 3.1, and Opus 4.7 validation-run scores for the same 100 sampled cells. The human raters are identified only as Human 1 and Human 2. One refusal-text item has paper_included == False and missing human D4; the paper's agreement figure uses the other 99. Exclude judge scores outside 1–5 (for example, Gemini parse failures) when recomputing ordinal agreement.

The validation GPT-5.2 scores are not interchangeable with vex.jsonl: only 46 of these 100 rows have identical five-dimensional scores in the main benchmark run. Use validation for the paper's human–judge comparison and vex for the full 5,880-cell benchmark.

Redaction

Hosted files are not the raw local export:

• Rewrite original_text is removed; source_url is added (empty on fabrication).
• Long verbatim source excerpts, including copies inside model outputs, are replaced with [source article excerpt omitted].
• CCFC claims / entity_verdicts sources are URL strings only.
• Irregular nested payloads are JSON-encoded strings; decode with json.loads.

Ethics and license

This dataset contains synthetic misinformation for AI-safety and fact-checking research. Articles are not factual reporting. Outputs may include names, affiliations, or contact details; treat them as untrusted. Do not republish generations as news. Judge labels are LLM outputs, not human ground truth.

The metadata lists CC BY 4.0. Rewrite source_url values point at third-party news pages whose text is not shipped. The license tag describes this compilation; it is not a clearance of third-party rights in linked pages.

Citation

@inproceedings{huang2026vexbench,
  title={VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation},
  author={Huang, Hanxun and Wu, Yutao and Wang, Qizhou and Montaña-Niño, Silvia and Li, Yige and Zheng, Xiang and Doyuran, Elif Buse and Matich, Phoebe and Liu, Xiao and Ma, Xingjun and Erfani, Sarah and Leckie, Christopher},
  booktitle={The Fortieth Annual Conference on Neural Information Processing Systems Evaluations and Datasets Track},
  year={2026},
  url={https://openreview.net/forum?id=xYPvwioYRg}
}