tb21-dsv41-flash-dsh

Terminal-Bench 2.1 trajectories: DeepSeek-V4.1-Flash, and three lines re-scored on a level environment

The point of this release is not the score. It is that the four lines being compared were not running in the same environment, and we only noticed after publishing the first comparison. Everything here is the repair and the re-measurement.

Raw, unedited agent trajectories for the DeepSeek-V4.1-Flash run over all 89 Terminal-Bench 2.1 tasks, plus every remediation re-run for it and for the two older lines, plus the scoring and analysis scripts. Every number in the write-up can be traced to a file here.

Only the re-runs of the two older lines are here, not their main runs. For DeepSeek-V4-Flash-0731 + dsh, the 89-trial main run and both re-run passes are collected in one place in openguardrails/tb21-dsv4-flash-0731-dsh — start there if that line is what you are after. The older terminus-2 and Qwen main runs are in openguardrails/tb21-qwen3.8-27b-terminus2.

Results

Exact McNemar, paired over the same 89 tasks:

dsv4.1+dsh  vs  dsv4+dsh      net +8    p = 0.15    not significant
dsv4.1+dsh  vs  qwen3.8+t2    net +8    p = 0.13    not significant
dsv4.1+dsh  vs  dsv4+t2       net +13   p = 0.019   significant
dsv4+dsh    vs  qwen3.8+t2    net  0    p = 1.00    dead even

V4.1 is the only line above 70%, and it is 9 points above V4 on the same scaffold — but 89 tasks cannot carry that to significance. The direction is clear; "proven stronger" is not a claim this sample supports.

The three scoring tiers

Applied identically to every line, by one script (scripts/score4.py):

1. as-shipped — what the run actually scored. This is what you get if you deploy the same stack today.
2. + infra re-run — a trial whose verifier never ran a single test (broken dpkg, unreachable mirror, container start failure) was really re-run under the repaired environment, and the re-run's verdict replaces the original. Nothing is estimated or replayed.
3. + work-product standard — a trial whose work product satisfied the task's own acceptance criterion, evidenced in the agent's own log while its service was still up, counts as solved even though dsh killed the process tree before verification. Each such trial is listed with the log line that justifies it (scripts/evidence_v41.py).

What we had to repair, and what we got wrong

Checking which hosts each run's logs actually talked to showed the four lines had not been run under equal network conditions:

Host egress here is 8–70 KB/s. A cold apt-get install tesseract-ocr-all … measured 755 s; dsh kills any single command at 300 s, which cuts the dpkg transaction in half and leaves the package database broken, after which the verifier cannot install its own tools and runs zero tests. Through the local cache the same install is 10–25 s.

Re-running the two older lines under the repaired environment raised their scores too (dsv4+dsh 65.2% → 68.5%, dsv4+terminus-2 61.8% → 62.9%). The repair is not a thumb on the scale for the new model.

A fault we could not isolate

Four DSv4.1 trials still ended with a broken dpkg after the network was verifiably fast (assertion: a representative install under 60 s, scripts/verify_repair.sh). We ruled out, by measurement: upstream bandwidth, cache warmth, container CPU/memory limits, the container→proxy path, DNS, and a PTY stdin stalling apt (reproduced with pty.fork(); 25.8 s with a PTY vs 27.3 s without — see scripts/pty_apt_test.py).

The decisive observation is that the same tasks behaved differently for the other model: dsv4 passed write-compressor, polyglot-c-py and sparql-university under the same repaired environment, and broke on two different tasks instead. The fault is stochastic, roughly 20–40% per attempt, and not task- or model-specific. We did not find the mechanism and are not guessing at one.

It left 4 tasks charged to V4.1 and only 2 to V4, which is luck, not capability — so 77.5% is, if anything, an under-estimate.

Agents repairing their own environment

45 of the 87 trials with complete session logs had at least one command killed at dsh's 300 s ceiling, 7.0 h of wall clock in total. Counting only commands the model itself issued (scripts/net_adapt.py):

The clearest single example, sam-cell-segmentation seq 70 — three correct moves in one command: clean up the killed process, re-launch under nohup where the ceiling cannot reach it, and switch to a domestic mirror (it later downgrades further to CPU-only wheels, having worked out the task needs no CUDA):

pkill -f 'pip install torch' || true; sleep 1; \
nohup pip install torch torchvision \
  --index-url https://mirrors.tuna.tsinghua.edu.cn/pytorch/whl/cpu \
  > /tmp/pip_torch_cpu.log 2>&1 & echo $!

Qwen's line does the same kind of thing (26 × --index-url .../whl/cpu, 350 × nohup). The two counts are not comparable — terminus-2 logs screen captures, dsh logs structured tool calls. Qualitatively, both models self-repair.

None of this is scored. It buys back the starting line, not points.

Layout

trajectories/
  dsv41-dsh/main/<task>/           89 trials: dsh-session.jsonl, dsh-result.json,
  dsv41-dsh/remediation-1/<task>/   8         instruction.txt, result.json, config.json,
  dsv41-dsh/remediation-2/<task>/   4         verifier/test-stdout.txt
  dsv4-dsh/remediation/<task>/     11        (main run: tb21-dsv4-flash-0731-dsh)
  dsv4-terminus2/remediation/<task>/ 1 trial: trajectory.json, terminus_2.pane, …
                                            (main run: tb21-qwen3.8-27b-terminus2)
scripts/
  score4.py              the only scorer; produces all three tiers + McNemar
  classify_v41.py        infra vs budget vs model failure, per trial
  net_cost.py            wall clock lost to commands killed at the ceiling
  net_adapt.py           model-side environment repair, from its own commands
  evidence_v41.py        work-product standard, with the proving log line
  verify_repair.sh       pre-flight assertion that the environment is actually fast
  bench_cfg.py           the TP=8 vs DP=2×TP=4 concurrency sweep
  pty_apt_test.py        the PTY hypothesis that turned out to be wrong
  start_*.sh             exact serving configs for both models
  *.json / *.log         run configs and run logs

Serving configuration

8×A800 80GB PCIe, vLLM with the Ampere port of DeepSeek-V4.1-Flash (openguardrails/vllm-dsv41-flash-sm80). TP=8 + EP was chosen by measurement, not assumption — at the 4-concurrency the harness actually uses it is 7.8% faster than DP=2×TP=4, which only overtakes at concurrency ≥ 8 (scripts/bench_tp8_p40000.json, scripts/bench_dp2tp4_p40000.json).

V4.1's thinking switch is binary: on by default (/tokenize returns 31 for "hi"), off with reasoning_effort: "none" (returns 5). Asserted before and after every run.

Limits

• One sample per task. No averaging. The stochastic dpkg fault alone moves the score by 2–4 tasks.
• Not comparable to vendor-published numbers, which are typically multi-run averages.
• The environment repair is partial by design: only trials that produced no measurement at all were re-run. Trials that produced a verdict keep it, including the two that ran out of wall clock — and those two had zero commands killed at the ceiling, so their timeouts were their own (scripts/net_cost.py).