EMNLP_Cost-Aware-Protocol-Routing

Cost-Aware Protocol Routing — research artifact card

EMNLP 2026 Accepted Paper LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks

Paper (arXiv) · PDF · Project page · Code · Dataset

Read this first: what this repository is, and is not

This is not a language model. It contains no foundation-model weights. The paper evaluates two third-party solver families — openai/gpt-oss-120b and google/gemma-4-31B-it — which we did not train, do not own, and do not redistribute. Their licenses and terms are their own.

This is a research-artifact card, not a checkpoint release. It documents the paper's lightweight routers: small scikit-learn classifiers that pick which collaboration protocol to run. It ships their training code, configurations, selected hyperparameters, and per-problem held-out predictions — but no serialized model file. The reason is stated honestly below.

Why no checkpoint is published

If you see a joblib.load(...) snippet on this page, ignore it. Hugging Face auto-generates a "Use this model" widget for scikit-learn repositories, and it points at a sklearn_model.joblib that does not exist here. That is not a broken upload: no checkpoint is published, deliberately, for the reasons below.

A trained metadata-only router exists in our internal results. We chose not to publish it:

1. Its serialized estimator stores integer classes [0, 1, 2, 3, 4] with no embedded label mapping. The mapping from index to protocol is recoverable only by assuming sorted label-name order. That is a convention, not a stored fact, and we could not verify it from the checkpoint alone. Publishing it would ask users to guess which action index 3 means — and a wrong guess produces confident, silently incorrect routing.
2. It was pickled under a different scikit-learn version and already emits InconsistentVersionWarning on load here. Pickled estimators are version-fragile and are not a durable research artifact.
3. Nothing is lost. Every number in the paper can be reproduced without it: the per-problem held-out predictions are released with string labels, and the training code is released and tested. A user can re-fit the router in minutes on CPU, or score our exact predictions directly.

If you need a checkpoint for a downstream system, train one with the released code and record your own label mapping.

What the router predicts

For each problem, before any protocol is executed, the router picks one of five actions:

none is a router action, not a fifth protocol. It is the correct choice when every protocol would have failed, so spending on escalation is wasted.

The target label is the fixed-order oracle: the first protocol that actually succeeded, in the order above. That label is retrospective — it is what the router is trained to predict, not something available at inference.

Feature inputs, and the leakage boundary

Routers see only problem-level information available before execution:

• problem text (text+metadata variant only), as TF-IDF features
• difficulty tier, source, and domain metadata
• benchmark and condition identifiers

They never see: gold answers, Baseline correctness, any protocol outcome, the oracle label, or any downstream collaboration result. This is enforced in code, not merely by convention — protocol_routing.features maintains an explicit forbidden-column set and raises LeakageError on contact, including for case variants. The repository's test suite proves the guard refuses deliberately constructed violations.

Router variants

All are multinomial logistic regression over the five actions.

Training and selection procedure

Six-setting evaluation (the main router results): stratified 70/15/15 split by oracle label, seed 20260712. Hyperparameters selected on dev only, then refit on train+dev, then evaluated once on untouched test problem identifiers. All routers are scored on identical held-out ids, which we verified.

Primary split (Table 1): stratified 80/10/10, seed 42, test n=423.

Confidence intervals throughout are 2,000-resample problem-level percentile bootstraps. They quantify uncertainty over which problems are in the benchmark — not run-to-run variability, since each problem-protocol pair has exactly one realized outcome.

Supported settings

Router evaluations cover six settings only: both solvers on OmniMath, LAB-Bench strict, and LAB-Bench text-no-tool. The matched four-protocol outcome data covers ten settings, but the router analysis does not. Please do not read these results as holding across all ten.

Results

Held-out router performance (from results/aggregate/heldout_router_evaluation.csv):

The router beats Baseline everywhere. It also leaves 18.5–28.9 points on the table against the retrospective oracle, and oracle-label macro-F1 stays in the 0.27–0.50 range. That gap is the paper's point, not a footnote.

Why this does not solve collaboration-value prediction

The paper's central negative result: a model's own confidence ranks Baseline failures well (0.8847 AUROC) but is much weaker at identifying which protocol pays off (0.1674 AUPRC for PER-specific value, 0.1041 for Broadcast-specific). Learned routers improve on fixed policies but do not close the oracle gap.

Cost-aware selection among Single, PER, Broadcast, and None remains unresolved. These artifacts are published so that others can attack that problem on matched data, not because the problem is solved.

Intended use and limitations

Intended: reproducing the paper; research on cost-aware routing and escalation policies; a baseline to beat.

Not intended: production routing without re-validation on your own workload. These routers are fit to specific benchmarks and two specific solver families. Protocol value varies substantially by task — the paper shows Broadcast's advantage over PER ranging from about 10 to 45 points across settings — so a router fit here should not be assumed to transfer.

Further limitations

• One realized execution per problem-protocol pair; no run-to-run variance.
• The fixed-order oracle is retrospective and is not a deployable policy.
• Cost is measured in logged tokens only — not latency, price, energy, or parallelism.
• No model snapshot was version-pinned at run time, so exact historical execution cannot be recreated. We claim functional, not bit-for-bit, reproducibility.
• Routing errors are asymmetric in a way these metrics do not capture: under-escalating loses a solvable problem, over-escalating only wastes tokens.

Reproducing

git clone https://github.com/ChihHsuan-Yang/EMNLP_Cost-Aware-Protocol-Routing.git
cd EMNLP_Cost-Aware-Protocol-Routing
make setup
make test
python scripts/train_router.py --help
python scripts/evaluate_router.py --help

See docs/REPRODUCE.md in the repository for the full guide.

Citation

@article{yang2026protocolrouting,
  title  = {{LLMs Can Predict Failure Risk, But Struggle to Predict Which
            Collaboration Protocol Pays Off: Cost-Aware Protocol Routing
            Across Reasoning Tasks}},
  author = {Yang, Chih-Hsuan and Jiang, Jingyan and Yang, Cheng-Hau and
            Vasudevan, Vikram and Zheng, Huihuo and Vishwanath, Venkatram and
            Thakur, Rajeev},
  journal = {arXiv preprint arXiv:2608.14927},
  year    = {2026},
  note    = {To appear at EMNLP 2026},
  url     = {https://arxiv.org/abs/2608.14927}
}

Acknowledgments

This research used resources of the Argonne Leadership Computing Facility, a U.S. Department of Energy (DOE) Office of Science user facility at Argonne National Laboratory (ANL) operated under Contract No. DE-AC02-06CH11357.

Contact

Chih-Hsuan (Bella) Yang — [email protected]