tb21-qwen3.8-27b-terminus2

Terminal-Bench 2.1 trajectories: Qwen3.8-27B vs DeepSeek-V4-Flash-0731

All scores on this page are reported after network and timeout faults were repaired. Nothing here is scored against a model because a package mirror was slow, a client library gave up early, or a container failed to start. Every trial lost to infrastructure was re-run under the repaired environment -- not estimated, not replayed -- and the re-run's verdict is what counts. What remains is model + harness capability.

Raw, unedited agent trajectories for three full runs over the same 89 Terminal-Bench 2.1 tasks, plus every remediation re-run, so each number below can be checked against what the agents actually did.

Superseded for the dsv4-flash-0731-dsh line. That line was later re-run under a network-levelled environment — the runs on this page were not all served by the same apt/pip caches, which this release did not account for — and re-scored at 61/89 = 68.5%, not the 65.2% in the table below. Its complete data, both re-run passes and the scorer now live in a dedicated release: openguardrails/tb21-dsv4-flash-0731-dsh. The four-line re-scoring that produced it is in openguardrails/tb21-dsv41-flash-dsh. Nothing on this page has been altered or removed; the trajectories here are the originals.

Results

The gaps are not statistically significant. Paired over the same tasks: Qwen vs DeepSeek+terminus-2 is net +6 (exact McNemar p=0.29); Qwen vs DeepSeek+dsh is net +3 (p=0.68). At 89 tasks and one attempt each, a lead this size is what run-to-run noise produces. Read the rows as "comparable", not as a ranking.

The scoring standard, applied identically to all three runs

1. Start from what each run actually scored.
2. Trials lost to infrastructure were re-run under the repaired environment; the re-run's verdict replaces the original.
3. A trial whose work product satisfied the task's own acceptance criterion -- evidenced in the agent's own session log while its service was still up -- counts as solved even if the process did not survive to verification, since a user would simply start it.

Rule 3 is where a scaffold that kills its own background services stops being charged for it. It was checked against all three runs; only dsh had trials that qualified, because only dsh kills them. The two terminus-2 runs had zero dead-service failures -- their live-service losses were ordinary wrong answers (wrong VM result, wrong chess move, wrong jump frame), so rule 3 moved them by nothing.

The three trials rule 3 credited to dsh, each with the log line that justifies it:

Two service tasks were refused under the same rule: mailman (service alive, but the leave flow never removed the member) and install-windows (VM alive and passing test_qemu_running_with_correct_params, but Windows never reached a GUI). Qwen failed install-windows in exactly the same way.

What changed from the earlier replay-based estimate

An earlier analysis of the dsh run estimated its infrastructure-corrected score at 61/89 by replaying agent commands in a clean shell. Re-running those trials for real recovered only one of them. The replay was too generous; these numbers supersede it.

Layout

trajectories/<run>/<task>/     the three full runs
remediation/<run>/<task>/      re-runs of infra-damaged trials (these set the final scores)
  agent/trajectory.json        terminus-2 runs: every turn, with reasoning_content
  agent/dsh-session.jsonl      dsh run: the harness's own session log
  result.json                  harness verdict
  verifier/test-stdout.txt     the task's own tests, verbatim
  verifier/reward.txt          0 or 1
analysis/                      scoring scripts, infra scan, per-failure analysis

The dsv4-flash-0731-dsh splits (89 + 17 trials) are also carried, together with the 11 later re-runs that are not on this page, in openguardrails/tb21-dsv4-flash-0731-dsh.

Per-split counts: {"qwen3.8-27b-terminus2": 89, "dsv4-flash-0731-terminus2": 89, "dsv4-flash-0731-dsh": 89, "remediation/qwen3.8-27b-terminus2": 4, "remediation/dsv4-flash-0731-terminus2": 4, "remediation/dsv4-flash-0731-dsh": 17}

How these were produced

All runs: agent_timeout_multiplier=6.0, verifier_timeout_multiplier=8.0, temperature 1.0, 4 concurrent trials, each model at its maximum thinking setting (Qwen reasoning_effort=xhigh; DeepSeek max thinking, asserted before each run via /tokenize).

Qwen3.8-27B: vLLM on 8xA800-80GB, tp=4 x data-parallel=2, bf16, 262k context, MTP speculative decoding (76 tok/s per request at 4-way concurrency). DeepSeek-V4-Flash-0731: fp8, tp=8, same 8 GPUs.

Infrastructure repairs applied equally: a local disk cache for apt (host egress was 8-70 KB/s, making apt-get install tmux asciinema take ~293s against the harness's hardcoded 120s ceiling; the cache took it to ~17s), a local PyPI mirror, a 3600s LiteLLM client timeout (the 600s default was discarding generations that took 1801s), and mirror fallback for deb.debian.org, whose index advertised a .deb its pool returned 404 for -- that one broke task verifiers, not agents.

Caveats

• Single attempt per task (k=1), and the gaps are inside the noise. Do not rank on these.
• The harnesses differ by design: terminus-2 drives a 160x40 tmux pane from the host; dsh sdk-minimal runs inside the container with persistent bash and a file editor. That is why the harness-matched row is included.
• filter-js-from-html failed in all three runs with a verifier-side Connection refused that we could not repair. It costs every run one task equally.
• Token-shaped strings were redacted; no real credential was present.