belief-engine-argument-quality-deberta-v3-large

Belief Engine argument-quality scorer

This repository releases the DeBERTa-v3-large argument-quality regressor used in Belief Engine's judgment layer. Fine-tuned on the IBM Argument Quality Ranking dataset's WA human ratings, it takes an English argument and returns a quality score in [0, 1].

This is a reproducibility release of a fine-tuned checkpoint using established argument-quality regression methods. It provides the exact scorer used in the framework, verified inference, evaluation tools, and a documented recipe for training a new checkpoint. It has 435,062,785 parameters and a 1.74 GB weight file; the original December 2025 weights are preserved unchanged.

Download the published model, scorer, and training/evaluation tools from joshcyang/belief-engine-argument-quality-deberta-v3-large. The release is MIT licensed and can be downloaded without an account.

Authors and contributions

• Belief Engine project authors: Joshua Yang, Damian Dailisan, and Maurice Flechtner, in the order listed in the workshop paper.
• Original model fine-tuning: Maurice Flechtner. Training provenance is recorded in release_manifest.json and NOTICE.md.
• Shared and maintained by: Joshua Yang (joshcyang).
• Upstream contributions: Microsoft's DeBERTa-v3-large backbone and IBM Research's Argument Quality Ranking dataset; see NOTICE.md for attribution.

Download and verify

Use Python 3.11. The versioned release includes the original weights and all standalone tools needed for inference, evaluation, and new training:

python3.11 -m venv .venv
source .venv/bin/activate
pip install huggingface-hub==1.4.1
hf download joshcyang/belief-engine-argument-quality-deberta-v3-large \
  --revision original-2025-12-v1 --local-dir argument-evaluator
cd argument-evaluator
pip install -r requirements-inference.txt
python scripts/verify_release.py --output verification.json
python usage_example.py 'Universal Basic Income reduces poverty by providing a guaranteed income floor.'

For experiments, replace the version tag with its full commit SHA from the model's Files and versions history or the code release's publication.json. Verification checks the checkpoint and tokenizer hashes and five reference scores with an absolute tolerance of 0.0001. The tested reference environment is Python 3.11, PyTorch 2.10.0 and Transformers 5.1.0 on CPU. Other devices may have different numerical tolerances. CPU inference is supported; CUDA is optional. The requirements files pin core dependencies, not every platform-specific transitive wheel.

The original weights' SHA-256 is: f9a073e14eb1e695afda5fd4e104b826b29d7660d35a335b9d332b970baa8f7d. SHA256SUMS covers the complete built release; release_manifest.json identifies original model files, provenance, and exact dataset splits.

Scoring contract

from argument_quality_model import ArgumentQualityModel
model = ArgumentQualityModel('.', device='cpu', local_files_only=True)
scores = model.predict_batch(['I think UBI is good.', 'UBI is bad because it costs money.'])
# Approximately [0.433586, 0.609802]

Input is the argument text alone; the model does not receive the topic, polarity, or supporting documents. Tokenization truncates and pads to 128 tokens. Output is the raw regression value clipped to [0, 1], without a sigmoid or calibration. Do not substitute a generic text-classification pipeline's softmax/sigmoid scores. This score estimates human-rated argument quality; it does not establish factual truth, evidence reliability, or a calibrated probability that a claim is correct.

Evaluate

python scripts/download_data.py --output-dir data/aqr
python scripts/evaluate.py --data data/aqr/test.csv --limit 1000 --output-dir results/historical-slice
# Omit --limit to evaluate all 6,315 test rows.

The downloader uses upstream dataset commit 590726b3765b1b90c5e53a17e3b1f77d92d3aa8a and verifies every CSV checksum. The splits contain 20,974 training, 3,208 development, and 6,315 test arguments. Evaluation saves metrics and row-indexed predictions.

The historical notebook evaluated the first 1,000 test rows, reporting:

These are recorded historical results, not a claim of evaluation on the entire test set. The supplied original training script uses test.csv for early stopping and best-checkpoint selection. Consequently these metrics are not an untouched holdout estimate. Unknown original details include the initialization seed, base-model revision, training hardware, and full environment lock. The model configuration records Transformers 4.57.3; the inference environment above has been checked against the saved example outputs.

Train a new checkpoint

pip install -r requirements-training.txt
python scripts/download_data.py --output-dir data/aqr
python scripts/finetune_argument_quality.py \
  --data-dir data/aqr --output-dir trained/dev-selection-v2 --seed 42

The corrected recipe seeds before model initialization, pins the base-model revision, trains on train.csv, selects using dev.csv, and evaluates test.csv once after selection. Defaults are 3 epochs, batch size 8, learning rate 1e-5, 500 warmup steps, weight decay 0.01, and 128 tokens. It saves model/tokenizer files, training provenance, dataset hashes, trainer state, and final test predictions. A CUDA GPU is recommended for full training; no GPU-memory or runtime estimate has been benchmarked for this release. A full corrected training run has not been performed as part of packaging this original checkpoint.

The output must be a new or empty directory. This recipe creates a new model; it does not regenerate the original weights byte for byte. The original script is retained in scripts/legacy/ for provenance and should not be used for a new held-out benchmark. --allow-custom-data explicitly permits different CSV splits.

Use with Belief Engine

Install pip install -e '.[classifier]' in the framework checkout. Download this release to a local directory and configure:

memory:
  strength_estimation_method: classifier
  classifier_model_path: /absolute/path/to/argument-evaluator
  classifier_kind: argument_quality
  classifier_allow_llm_fallback: false

Alternatively use the Hub repository ID and set classifier_revision to the full release commit SHA. A model-loading failure stops classifier-based runs. LLM-only runs can still proceed when a supplementary classifier is unavailable.

Citation

If you use this checkpoint in research, cite the related Belief Engine workshop paper and identify this model repository and the exact commit SHA used. Please also cite the base-model and dataset papers listed in NOTICE.md.

Joshua Yang, Damian Dailisan, and Maurice Flechtner. 2026. Belief Engine: Configurable Opinion Dynamics for Reliable Social Simulation. PoliSim@CHI 2026 workshop paper. PDF.

@misc{yang2026beliefengine,
  title = {Belief Engine: Configurable Opinion Dynamics for Reliable Social Simulation},
  author = {Yang, Joshua and Dailisan, Damian and Flechtner, Maurice},
  year = {2026},
  howpublished = {PoliSim@CHI 2026 workshop paper},
  url = {https://polisim.net/assets/papers/accepted_papers/Belief_Engine_Configurable_Opinion_Dynamics_for_Reliable_Social_Simulation.pdf}
}

Attribution and licensing

See NOTICE.md for the base model, dataset, and citations. The contributed code and fine-tuned checkpoint are released under MIT (see LICENSE). Microsoft's upstream MIT notice is retained in LICENSE.deberta. The publisher verifies that license metadata and the license file are included.