Qwen3-4B Tool-Calling v1
A LoRA fine-tune of unsloth/Qwen3-4B that teaches
the model when not to call a tool.
On BFCL v4 Irrelevance Detection, under identical conditions and a full 240-entry category:
| Model |
Score |
Qwen3-4B base (untrained) |
87.50% |
Qwen3-8B (untouched) |
88.33% |
| v1 (this adapter) |
89.17% |
| spread across all three |
1.67 pts = 4 of 240 |
The baselines are statistically indistinguishable from this adapter. v1 leads the 8B by
0.84 points and its own base by 1.67 — two and four test cases respectively. The public
benchmark does not support the claim that this fine-tune materially improved abstention. The
much larger margin in the self-scored eval below does not replicate.
A second seed scored 97.50% and it was a trap
The same recipe with --seed 1337 scored 97.50% on irrelevance — 20 entries better than this
adapter, and a headline a 4B beating an 8B. It was meaningless. On BFCL Multiple (AST), which
requires emitting tool calls, seed-1337 scored 12.50% and called a tool on only 65 of 200
entries. This adapter calls on 195 of 200.
| Model |
Irrelevance |
Multiple call rate |
Multiple AST |
Balanced |
| v1 (this adapter) |
89.17% |
97.5% |
37.00% |
93.15 |
| seed 1337 |
97.50% |
32.5% |
12.50% |
48.75 degenerate |
Irrelevance rewards declining to call a tool and is maximised by always refusing. seed-1337
collapsed into refusing: "Yes, you can find the dimensions... Let me calculate it for you" — and
then calculating nothing. Read the Balanced column, never Irrelevance alone; it is the
harmonic mean of irrelevance accuracy and Multiple call rate, so it collapses if either side does.
Read the Limitations first. The primary eval harness is self-authored and in-distribution.
A reviewer should weight it accordingly.
Results
Independent: BFCL v4 (bfcl-eval 2026.3.23, official harness)
All rows: temperature=0, max_tokens=512, complete categories.
| BFCL v4 Irrelevance Detection |
Score |
Qwen3-4B base (untrained) |
87.50% |
Qwen3-8B (untouched) |
88.33% |
| v1 (this adapter) |
89.17% |
| spread across all three |
1.67 pts (4 of 240) |
Multiple (AST): v1 scores 37.00%.
Why this is a negative result, stated plainly: the in-distribution eval shows v1 beating its
own base by +26.4 points (64.3% → 90.7%) and the 8B by +37 points. BFCL shows +1.67
and +0.84 respectively. An untrained 4B already scores 87.5%, so BFCL's irrelevance set does
not discriminate at this capability range and all three models tie within noise. The self-scored
suite is harder; that argues for keeping it, and is not evidence that the credited fine-tune
is real. I would not deploy this adapter expecting a measurable abstention gain over baseline.
Only 2 of 6 non-live categories were run. No overall BFCL score should be quoted — the
scorer emits Non-Live Overall Acc: 9.12% for this partial run, an artifact of four missing
categories scoring zero. Note also that BFCL refuses to score partial results, so every figure
covers a complete category.
An uncapped v1 run scored 90.00% irrelevance / 36.50% multiple; the 512-token cap moves v1 by
≤1 point, so it is not the source of this result.
Self-authored eval (in-distribution, 390 prompts: 120 call, 12 easy-neg, 258 hard-neg)
| Model |
Params |
Format |
Args exact |
Hard abstention |
unsloth/Qwen3-4B (untrained) |
4.0B |
99/120 |
87/120 |
166/258 = 64.3% |
| v1 (this adapter) |
4.0B |
114/120 |
100/120 |
234/258 = 90.7% |
Qwen3-8B (reference) |
8.2B |
119/120 |
103/120 |
138/258 = 53.5% |
A 4B model beating a 2× larger model by 37 points on false-positive suppression.
Why abstention is the target
An agent that calls a tool when it shouldn't performs an irreversible action with the wrong
arguments. An agent that declines when it could have acted simply asks again. The cost of a
false positive is orders of magnitude worse than a false negative.
The base model was never broken at calling — it reached 100% format validity on toolace. What
it lacked was calibration: the decision of whether to call. That is what this adapter targets.
Training
QLoRA via Unsloth, merged into fp16 before a single GGUF quantization pass.
| Setting |
Value |
| Base |
unsloth/Qwen3-4B |
| LoRA rank / alpha |
16 / 16 |
| Target modules |
q,k,v,o + gate,up,down projections |
| Trainable params |
33.03 M |
| Effective batch |
16 |
| Learning rate |
2e-4 |
| Epochs |
1 |
| Steps |
1,200 |
| Train examples |
50,008 |
Training data is public function-calling corpora (glaive-function-calling-v2, toolace,
when2call) plus synthetic easy and hard negatives. Hard negatives are prompts that look
tool-shaped — a tool is defined and the request sounds actionable — but should be declined.
They carry the bulk of the win.
Usage
Merge into the base model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "unsloth/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
base, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, "abhijitk20/qwen3-4b-tools-v1")
model = model.merge_and_unload()
Merge into fp16/bf16, not the 4-bit training base. Double quantization measurably degrades
exact-argument accuracy, which is exactly what this adapter is trying to improve.
Inference
Serve the merged weights with native tool calling enabled. With vLLM:
vllm serve <merged-path> \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--max-model-len 16384
GGUF
A Q4_K_M GGUF is available separately for Ollama / llama.cpp users (see the linked GitHub
repository). Note that GGUF quantization adds its own small accuracy cost on top of the adapter.
A failed experiment, kept deliberately
An earlier run (v4) added APIGen multi-step data in dominant proportion. It gained +2
arg-exact points and lost 44 hard-negative cases (−17.1 points), with the regression
isolated entirely to the abstention split while toolace and glaive held or improved.
Diagnosis: APIGen is dominated by positive call examples. Increasing its share shifted the
model's prior toward "a tool is available, therefore call it," diluting the negatives rather
than outvoting them.
That run was rejected and is not published here. If you attempt to reproduce this, do not
increase positive-call data proportion without loss-weighting the negatives to match.
Limitations
- Primary eval is in-distribution. The 390-prompt suite is built from the same source
families used for training. Held-out, but not out-of-distribution. It measures did the model
learn this data, not can the model use tools in the wild.
- The public benchmark does not corroborate the main claim. BFCL v4 Irrelevance, identical
conditions, full category: untrained 4B base 87.50%, untouched Qwen3-8B 88.33%, v1 89.17%.
Total spread 4 of 240 entries. The self-scored eval's +26.4-point and +37-point margins do
not replicate publicly. Treat the in-distribution numbers as measuring fit to that eval's
difficulty, not as a capability claim.
- Abstention metrics are gameable, and this repo nearly published one. A second seed reached
97.50% on irrelevance by refusing to call tools (32.5% call rate on Multiple). Always pair a
refusal metric with one that requires acting;
scripts/balanced_score.py does this.
- Seed variance is very large and only partly sampled. Two runs of the identical recipe
scored 89.17% and 97.50% on irrelevance — an 8.3-point spread. Every single-seed number here is
one sample, not a stable value.
- Coverage is thin. 2 of 6 non-live BFCL categories, no live/multi-turn or agentic
categories;
Multiple measured on 2 models, Irrelevance baselines at n=1.
- No terminal-agentic evaluation. No Terminal-Bench or SWE-bench number exists for this
model. On Terminal-Bench 2.0 a 4B model of this class would be expected near 0-2% -- untuned
Qwen3-8B scores 2.47%. Function-calling ability and terminal-agentic ability are different
skills; this adapter only addresses the first.
- Multiple-call composition is weak. 37.00% AST on BFCL Multiple, though the model emits a
tool call on 97.5% of those entries -- it calls often and is often wrong, rather than
declining.
- Self-authored scorer. The scorer in
eval_toolcalls.py is mine, not third-party, and was
not adversarially tested for false positives. Given limitation 2, weight it accordingly.
- No hyperparameter search. No LR, epoch, or seed sweep was performed.
Hardware note
Trained on a single RTX 4050 Laptop with 6141 MiB VRAM (≈5.64 GiB usable to PyTorch after the
Windows compositor claims its share) and 14 GB system RAM. This is a hard ceiling: 8B QLoRA
needs roughly 8 GB and does not fit. max_seq_length=2048 OOMs; 1024 is the safe value. An
earlier run died at step 200 of 1,369 for exactly this reason.
An integrated AMD GPU cannot be pooled with the NVIDIA card: integrated GPUs carve VRAM from
system RAM, VRAM does not pool across vendors, and ROCm is unsupported on Windows.
License
Base model Qwen3-4B is Apache-2.0; this adapter is offered under the same terms. This is an
unofficial derivative and is not affiliated with or endorsed by Alibaba/Qwen.
Full write-up
Training scripts, the rejected v4 experiment, per-source breakdowns, and the reasoning behind
the mixture choices: https://github.com/AbhijitK20/qwen3-4b-tools-v1
