Ministral-3-14B-lil-logbook

Ministral-3-14B lil — hyperparameter runs

Seven LoRA adapters trained on the same dataset, one variable changed at a time: the three original runs below, plus a four-point learning-rate ladder under lr-sweep/. F16/Q8_0 GGUFs for the original three.

The original three runs were evaluated on loss curves and spot generations only; the ladder adds an out-of-domain loss axis and a generation-level composition probe (sections below).

Dataset

Multi-turn dialogue packed to 512-token samples — 13,667 samples, 6.16M tokens total. Turns are delimited by # <code> headers — short per-character abbreviations, not names. Unattributed narration is folded into the speaker's turn:

# m
Okay.
This is as far as I come.
# s
We come to a stop in front of a tree positioned in the middle of three spotlights.
It's not a particularly interesting tree or anything like that, so I'm a little confused as to why it's been so carefully illuminated.
What is this?
# m
A tree, idiot.
# s
No, like...what is this?

The pack was built in two phase-shifted passes — the second pass offsets every chunk boundary by 256 tokens, so the corpus appears twice with different windowing and exact-duplicate chunks removed. Experimental idea for cheap augmentation on small corpora; its actual effect is not isolated or evaluated in this repo.

Runs

Shared config: rsLoRA r512 / α512, dropout 0, targets q,k,v,o,gate,up,down, batch 16 (no accumulation), 838 steps (one pass), constant LR after warmup, BF16 base + FP32 AdamW, seed 42, eval on 256 held-out samples every 25 steps.

Comparison

Init barely matters — according to metrics. base-lil matched v3-lil's eval curve point-for-point for the first 250 steps and finished 0.05 behind at 33% lower LR — the 186M-token RP adapter it continues from bought approximately nothing on this corpus by loss. Loss can't tell the whole story here: when the two models miss, they might miss differently.

Gradient clipping at 1.0 changed nothing. base-lil-clip1 clipped on 100% of steps (raw grad norms mean 14.9, max 36) and still landed within 0.001 of the unclipped control. AdamW divides the first moment by the root second moment, so a rescale applied to every gradient approximately cancels out of the update — clipping only matters when it fires intermittently.

LR ladder (lr-sweep/)

The question: how low can LR go while the base model still learns this corpus — smaller weight modification for relatively the same result, because every unit of fit is paid for in general-capability damage. Fresh adapters on base, same recipe as base-lil-clip1 (max_grad_norm 1.0, justified above), plus an eval before step 0 (the untrained baseline all deltas are measured from) and a second eval set: 256 held-out forum-RP samples from the v3 corpus, line-truncated to 512 ctx — out-of-domain prose the training never shows, scored by the same eval machinery. In-domain fit and out-of-domain damage, side by side:

No cliff down to 1e-7, and the exchange rate improves monotonically as LR drops. A 50× LR cut keeps 48% of the learning for 25% of the damage. Forgetting here is distribution shift, not training per se: nothing in a 100%-narrow stream defends the out-of-domain directions, so damage scales with total movement — and lower LR simply moves less.

Composition probe

Loss can't answer the question that matters for an RP adapter: do the learned personas compose with content the corpus never contained? Probe: 32 scenes placing the corpus's recurring speakers (by their # code headers) into mundane/technical situations foreign to the corpus — broken spreadsheets, router channels, tax forms, bed-leveling a 3D printer — each ending on a character header, each embedding a concrete technical fact the continuation can get right or wrong. 5 rerolls × 6 checkpoints (base, the four ladder runs, base-lil-clip1), scored 0–2 on voice (unmistakably that speaker), coherence (engages the scenario's content correctly), format (clean header structure, no degeneration) by an LLM-judge panel.

Sampler first, because it dominated round one. The sampler recommended for v3 (temp 0.5, min_p 0.1, rep 1.03) is a loop incubator on these base-derived checkpoints: 30–65% of completions collapsed into verbatim loops or ... header volleys, at every LR. At temp 0.8, min_p 0.05, rep 1.1 the collapse rate drops to ≤1.2% everywhere. Use the latter with these adapters.

Scoring was only trusted blinded. Per-checkpoint judging (one judge instance per model) swung good-rates by ±8pp from judge strictness alone, so the contested rungs were re-scored as a pooled, shuffled, label-stripped set with each judge seeing a mix of all three. Blinded good-rate (voice+coherence+format ≥ 5/6), t0.8 sampler, n=160 per rung:

Differences are ≤2σ at this n. Read: 5e-7 is the best point estimate; 1e-7 matches it within noise at half the prose damage. Bare base scores 7% — it cannot hold the header format at all, leaking pretraining documents mid-scene, so some adapter is load-bearing for the format itself. The universal bottleneck at every LR is coherence (~1.0/2): completions open on-task, then drift back into the corpus's relationship register within a few turns. That is the same distribution-shift mechanism as the prose-loss column, observed at generation level — and the motivation for the replay-mixing item below.

Apostrophe pathology

Above 5e-7 a contraction-corruption artifact appears, LR-dependent: 0 corrupted contractions at ≤5e-7, 14 at 2e-6, 97 at 5e-6, counted across 1,920 probe completions. Shape: the model emits a contraction stem then neither apostrophe form — don + corpus-boosted junk (speaker-name fragments, the corpus's stutter idiom) where 't should be.

Mechanism: the source corpus's own typography runs 10:1 curly (’) to ASCII (') apostrophes, and the tokenizer makes the two forms competing single tokens at the same slot ([' don'] + ['’t'|'\'t']). High-LR training suppresses the ASCII suffix (the curly share of generated apostrophes climbs 25% → 67% up the ladder), while an ASCII-styled prompt pushes the copy direction the other way. Usually one side wins; occasionally the conflict flattens the slot and whatever else the adapter boosted leaks through. The cleanest microscope in this repo on what the prose-loss column physically is. Fix for a dataset v2: normalize apostrophes to one form at build time — token counts are unchanged and the conflict never starts.

GGUFs

/gguf/ — F16 and Q8_0 for each of the original three runs, merged into the base. No GGUFs for the ladder adapters yet.

Tests — to do

• ~~LR-range sweep, 5e-7 to 1e-5~~ — done, see the ladder.
• LR tail: 5e-8 / 2e-8 — does the exchange rate keep improving once almost nothing is learned?
• Rank axis: the ladder held r512 fixed; rank is the second dimension of the grid.
• Replay mixing: 5–10% forum RP in the training stream, aimed at the coherence drift — attacks the distribution shift directly instead of hiding it with LR.
• Dataset v2: apostrophe normalization at build time.