AgentsSci — Agents for Science Research Group

EMNLP 2026 Accepted Paper LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks This is not a language model. It contains no foundation-model weights. The paper evaluates two third-party solver families — openai/gpt-oss-120b and google/gemma-4-31B-it — which we did not train, do not own, and do not redistribute. Their licenses and terms are their own. This is a research-artifact card, not a checkpoint release. It documents the paper's lightweight routers: small scikit-learn classifiers that pick which collaboration protocol to run. It ships their training code, configurations, selected…

The short version. We ran the same 6,803 reasoning problems through four different LLM collaboration setups — from a single direct answer up to a four-agent deliberation — and recorded, for every problem, which ones got it right. Then we asked whether a model can look at a problem beforehand and predict which setup is worth paying for. It can predict whether it will fail. It cannot predict which collaboration protocol will fix the failure. That gap is what this dataset is for. This is the data release for the EMNLP 2026 paper LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off. Every released problem was run under all four, with the same solver…