overthinking-logit-bias

Overthinking penalty logit bias for llama.cpp

Logit bias lists and a script to apply the overthinking penalty of Lotfi et al., "Quantized Reasoning Models Think They Need to Think Longer, but They Do Not" (arXiv:2606.00206) with llama.cpp's llama-server.

The method subtracts a fixed λ from the logits of 50 overthinking markers (Appendix C, Table 8: " Wait", " But", " Alternatively", " perhaps", ...) at every decoding step. It needs no extra forward pass, so llama.cpp's logit bias is all it takes.

For Ornith-1.5 and models built from it (e.g. Tiel-Coder), use the paragraph-initial list (--initial, bias/ornith-1.5-35b-a3b/initial-lambda-*) rather than the paper's list. These models open a new line of reasoning at the start of a paragraph ("\n\nWait,", "\n\nActually,"), where the word has no leading space and is a different token from the paper's " Wait". The paper's list barely shortened Ornith-1.5's reasoning (−6% tokens); the paragraph-initial list at λ = 4 cut it by 34% on MATH-500 level 3–5 with all answers still correct, and on hard coding-agent tasks it let the model get past the reasoning cut-off (see Verification and REPORT.md). This variant is not in the paper.

Files

Token IDs depend on the vocabulary, not on the model weights or the quantization. The Qwen3.8-27B family, the Qwen3.5-35B-A3B family and the Ornith family derived from them share one tokenizer (248,320 tokens, sha256:f307bf31f9c6130c as printed by the script), so the lists here are identical and fit any of these models.

For any other model, build the list yourself (below); it takes a second. 49 of the 50 markers are single tokens in this vocabulary; " recheck" is two tokens and is skipped.

1. Build the list

# Local GGUF (the first shard of a split model)
python3 make_logit_bias.py model.gguf -p 2.0 -o bias.json

# GGUF on the Hub: only the header is downloaded (a few MB)
python3 make_logit_bias.py hf://OWNER/REPO/model-Q4_K_M.gguf -p 2.0 -o bias.json

# llama-server options instead of JSON
python3 make_logit_bias.py model.gguf -p 2.0 -f cli

# Paragraph-initial markers instead of the paper's list
python3 make_logit_bias.py model.gguf --initial -p 4.0 -o bias.json

-p is λ (> 0). The paper sweeps 0.5–4.0 and reports the best λ per configuration; there is no single recommended value. The script prints the token ID of each marker and the vocabulary hash to stderr. Set HF_TOKEN for gated repos. Supported tokenizers: byte-level BPE (gpt2, e.g. Qwen, Llama 3) and SentencePiece (llama).

Paragraph-initial markers (--initial)

--initial replaces the paper's 50 leading-space markers with 10 capitalized markers spelled without a leading space, the form a word takes at the start of a paragraph:

Wait, Hmm, Actually, But, However, Alternatively, Instead, Or, Maybe, Perhaps

These are the words Ornith-1.5 opens its reasoning branches with, plus the capitalized hedges of the paper's list. "Actually" is not in the paper (only " actually" is), but it is the most frequent opener in Ornith-1.5 (100 times over 15 MATH problems without bias); in a sample of 10, 7 started a revision and 3 a neutral rephrasing. Lower-case forms without a space are left out: they are mostly word pieces ("thres|hold", "Iterat|or"). An --also-no-space option, which adds the no-space form of all 50 markers (83 tokens), shortens the reasoning about as much but penalizes such word pieces as well.

2. Use it with llama-server

Per request

llama-server accepts logit_bias as an array of [token_id, bias] pairs on /completion and the OpenAI-compatible /v1/chat/completions:

jq -n --slurpfile lb bias.json '{
  messages: [{role: "user", content: "How many primes are below 100?"}],
  logit_bias: $lb[0]
}' | curl -s http://127.0.0.1:8080/v1/chat/completions \
       -H 'Content-Type: application/json' -d @-

With the OpenAI Python client:

import json
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="none")
bias = json.load(open("bias.json"))
resp = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "How many primes are below 100?"}],
    extra_body={"logit_bias": bias},  # a list of pairs, so pass it in extra_body
)

Many agents and clients have an "extra body" setting where the same {"logit_bias": [...]} can go.

Server-wide

llama-server -m model.gguf $(cat bias/qwen3.8-27b/lambda-2.0.args)

Caveats

• A request's logit_bias replaces the server's -l list rather than adding to it.
• --models-preset (INI) cannot hold the list: a preset keeps one value per key and --logit-bias takes one token, so only the last entry would apply. Use the per-request form with router mode.
• The penalty applies to the whole output, not only to the reasoning: the answer, code and comments also see fewer " or", " but", " error", " could", and so on. Only the leading-space forms are penalized, so e.g. error at the start of a line or after ( is not affected. With --initial it is the other way round: only the no-space forms (start of a line, after (, ", **, ...) are penalized, so a sentence or a bullet that starts with "But" or "However" becomes less likely, while the same words mid-sentence are unaffected. Identifiers such as Or or Maybe in code are affected too.
• The reasoning gets shorter on average, but single runs vary a lot, and a run does not always finish sooner (see REPORT.md).
• Penalizing one form moves the hesitation to the others: with --initial alone, Tiel-Coder hesitated mid-sentence instead; without "Actually", Ornith opened its paragraphs with "Actually" instead.
• The IDs are only valid for the vocabulary they were built from. A list for another vocabulary silently penalizes unrelated tokens.

Verification

See REPORT.md for the setup and all results. In short, on Ornith-1.5-35B-A3B and Tiel-Coder-35B-A3B:

• The token IDs from the GGUF match llama-server's /tokenize, and the bias lowers the marker probabilities as expected.
• MATH-500 (15 problems, levels 3–5, Ornith): the paper's list at λ = 2 cut the tokens by 6%, --initial at λ = 4 by 34%, all answers still correct.
• Coding agent (pi, three hard tasks): without the penalty, the first long reply often reasoned until the 16384-token cut-off and nothing was written. With --initial, the agent got to write and test code that passed the hidden tests: Ornith-1.5 Q3_K_L needed λ = 4, Tiel-Coder UD-IQ4_XS managed with λ = 2. The paper's list, alone or together with --initial, did not get either model past the cut-off on the hardest task.
• Removing "Actually" from the list moved the hesitation to "Actually".
• The penalty does not always save time, and single runs vary a lot.
• In progress: the same tasks on lower quantizations of Ornith (Q2_K_L, IQ2_XXS), to see whether the λ needed depends on the quantization.

Evaluate on your own tasks before relying on the penalty.

Citation

@misc{lotfi2026quantized,
  title         = {Quantized Reasoning Models Think They Need to Think Longer, but They Do Not},
  author        = {Lotfi, Sanae and Kirichenko, Polina and Li, Steven and Liu, Zechun},
  year          = {2026},
  eprint        = {2606.00206},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2606.00206}
}

The marker list is taken from Table 8 of the paper. This repository is not affiliated with the authors.

License

The script and the lists are released under the Apache License 2.0 (see LICENSE).