DeltaRoutingAI

HorneSci's open-weight continued-pretraining release for Qwen3.8-27B. DeltaRoutingAI trains language adapters on Python technical documentation and delivers a 4.61% reduction in held-out text NLL in its paired evaluation.

The downloadable checkpoint is a PEFT LoRA adapter in safetensors format (233.6 MB). Load it with Qwen/Qwen3.8-27B, pinned to revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. Qwen authors the base model; HorneSci authors the continued-pretraining recipe and adapter weights. The adapter weights are released under Apache 2.0 with upstream attribution and data notices included.

Load

Use a CUDA environment with the versions in requirements-inference.txt. The evaluated environment used PyTorch 2.9.1 with CUDA 12.8.

python -m pip install -r requirements-inference.txt
hf download HorneSci/DeltaRoutingAI --local-dir DeltaRoutingAI

from load_adapter import load

model, tokenizer = load()
prompt = "Explain how Python context managers handle exceptions."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Run the example from the downloaded repository directory. load_adapter.py pins the base revision and reproduces the evaluated NF4/BF16 loading path. Pass revision="<release-commit>" to pin the adapter too.

Training

Continued pretraining used 256 optimizer updates and 261,632 predicted tokens, seed 20261006, rank 8, alpha 16, dropout 0, sequence length 512, microbatch 1, gradient accumulation 2, and AdamW at 2e-5. NF4 double quantization uses BF16 compute. Adapters cover language DeltaNet, attention, and MLP projections; base, vision, and MTP parameters stay frozen. The objective is causal next-token likelihood on raw text.

The corpus contains 30 CPython library-documentation files at revision ebf955df7a89ed0c7968f79faec1de49f61ed7cb. Eight separate documents form the held-out split. A shared-25-word-span filter was applied before model evaluation. DATA_PROVENANCE.json records the source URLs, split, licenses, transformations, and hashes.

Benchmarks

Evaluation compares the same NF4 base with the release adapter disabled and enabled. Text NLL is token-weighted across 104 blocks from eight held-out documents (53,144 predicted tokens). Each BIG-bench task uses a frozen 64-example subset; summed choice log-likelihood is the primary metric, with length-normalized scores reported alongside it. The additional run covers 56 frozen, previously unused cases from MMLU-Pro, GSM8K, and IFEval and retains all 112 paired responses.

The additional benchmarks use adapted frozen subsets: MMLU-Pro scores next-token option-letter likelihood, GSM8K checks the exact numeric final answer after ####, and IFEval scores eight strict single-constraint cases. Generation uses the pinned chat template with thinking disabled, greedy decoding, a 768-new-token limit, and a 60-second bound. Truncated responses remain scored and in the denominator: GSM8K base 2/16 and adapter 6/16; IFEval base 2/8 and adapter 5/8. The run completed with zero missing responses, execution errors, or optimizer updates. These scores describe the declared subsets and protocols rather than full official leaderboard evaluations.

The paired document-bootstrap 95% interval for the text NLL change is [-0.072400, -0.053326], using 10,000 resamples. REPORT.md gives every paired confidence interval, source revision, and replay command. The baseline pack, additional results pack, and selection-source pack retain inputs, per-item scores and responses, scripts, hashes, environments, and notices. TRAINING_AND_VALIDATION.json records the fresh-process reload and adapter-disabled control.

Files and licenses

• adapter_model.safetensors and adapter_config.json: released adapter.
• load_adapter.py: pinned-base loading helper.
• DATA_PROVENANCE.json, TRAINING_AND_VALIDATION.json, and benchmarks/: reproducibility evidence.
• LICENSE: Apache 2.0, retaining the Qwen upstream notice.
• CPYTHON-LICENSE.txt, CPYTHON-DOC-LICENSE.rst, and BIGBENCH-LICENSE: data and benchmark notices.
• benchmarks/additional/licenses/ and benchmarks/selection/licenses/: MMLU-Pro, GSM8K, IFEval, and source-checker notices.
• ATTRIBUTION.md: upstream authorship and modification record.

Adapter SHA256: f1eabb12cffad143e788139f85c32ce9032e03b76cc562b7b6f72b6cd5b3ba1b.