kev-4b-actions

LoRA for Kev (Qwen3.5-4B-Base + rank-16 LoRA + pointer head, the /v1/systemone contract) that picks the next browser action in the custom_browser harness: from a store's homepage, find a named product and open its page. It replaces only the action question. The done check is a separate LoRA, Flolep2607/kev-4b-guard@v2.

Results

Test sets (the number that matters)

14 tasks x 2 runs on obscura, starting at the store's homepage, sites never trained on. Corrected 2026-10-14: every check is exact now (the final address must end with the target product's path). Two timber products are not sold on their site (Bunnings, NZ Timber): those tasks pass only when the agent gives up on its own. The first version of this card used looser checks that counted a search page and a 138x32 deck as found (r1 22/28, prod 10/28).

r1 alone beats every cascade with holo. Its 8 failed runs: footcom (product page is a 404), eel and scott (results pages show no product links in obscura), and Bunnings timber, where it called a 138x32 deck the product instead of giving up.

Held-out sites (83 tasks, 59 sites)

Round-1 tasks on sites held out by host hash, never trained on: far more varied stores than the test sets (office, electronics, auto, garden, pets, farm...). The gain holds, smaller than on the test sets.

Caveat: these r1/r1s runs were served through a relay to molab; relay failures show up as kev_error escalations in the cascade, so the cascade rows understate r1.

Offline: held-out sites, action accuracy

Steps from winning runs on sites held out by host hash. Caution: the harness orders options best-first, so the right answer is the first option ~70% of the time; "always pick the first option" scores 66.7% here. The shuffled rows (options reordered, keys renamed) remove that shortcut.

r0b partly learned "pick the first option" (-16 points when shuffled); order-free, both rounds gain ~10 points over prod. Gives up where a winning run acted: prod 13%, r0b/r1 0%. On 377 logged production requests r0b gives up 9% vs prod 42%. Guard answers move on ~1% (devtools replay keeps them).

Training

Speed notes: flash-linear-attention = 2.5x (5.2 -> 2.1 s/record on long states); batch 1 x accum 8 is the fastest setting; batch 8 + length sorting was 7x slower (padding). Collection, not training, is the bottleneck: ~2.5 h for 542 tasks x (Kev + cascade).

Serving: three adapters on one int8 base (zev-fast)

zev-rs branch multi-lora: Kev adapters on the same base share one copy of it. The first one listed is merged (fused int8 path, full speed); the others attach as exact deltas ((W + dA) + (dX - dA) = W + dX). Measured on molab (RTX PRO 6000), int8, 377 requests at concurrency 16:

int8 vs bf16, merged vs unmerged: no measurable change on the held-out sets (actions 0.839-0.849 / 0.983-0.989, every guard F1 0.987-0.988, within one or two examples). Full tables: llm-lab models/kev/README.md.

KEV_W8=int8 zev serve --model Flolep2607/kev-4b-actions@r1 --model Flolep2607/kev-4b-guard@v2 --model jxxxll/kev-4b-substitutes-judge
# harness: HARNESS_KEV_URL=<server>  HARNESS_KEV_GUARD_URL=<server>  HARNESS_KEV_GUARD_MODEL=Flolep2607/kev-4b-guard@v2

Data: expert iteration (ReST-style RL)

Reward is reaching the target URL, never a model's opinion.

1. Product URLs from the index (327 broad seed queries, marketplaces and test-set sites excluded), opened in obscura through the harness; Claude Haiku labels product / not product / error and reads the product name.
2. Tasks: start on the site's home, find the named product (round 0: 248 tasks / 167 sites; round 1: 542 / 373 new sites).
3. Rollouts with every Kev request recorded (HARNESS_KEV_RECORD=1): round 1 collected on-policy with r0b (alone 173/542, cascade 198/542).
4. Labels: steps of winning runs that changed the page, plus any step (from any run) where an option links to the target page. Only the action question is kept.

Data: Flolep2607/kev-rl-guard (actions/). Code: flolep2607/llm-lab models/kev/finetune/harness-rl.

One adapter for actions + guards (combo-r1)

Trained on the action steps plus the guard pages (incl. reworded queries), 2x devtools replay. One request answers the action and the three guards, so a done-check step costs one pass instead of two (zev int8: 36 ms vs 68 ms per step).

Cheaper, slightly weaker: the separate adapters stay the default; the combo needs more reworded-query guard data or a separate guard head on its backbone.

Next

• r1-clean: r1 retrained on corrected labels (wins from the exact check, not the agent's claim), then benched.
• Round 2: difficulty filter (several sampled runs per task, keep the ones that sometimes pass) and a contrastive term on Kev's choice probabilities.
• Retune the cascade's hand-off thresholds once r1 is served directly (no relay).

Use

python -m kev.serve --run <local snapshot of this repo> --port 8009
# harness: HARNESS_KEV_URL=http://host:8009  HARNESS_KEV_GUARD_URL=<guard-v2 server>