tb21-dsv4-flash-0731-dsh

Terminal-Bench 2.1 trajectories: DeepSeek-V4-Flash-0731 + dsh sdk-minimal

Every trial of this one line, in one place: the 89-task main run, both re-run passes, and the scoring scripts. The trajectories are raw and unedited — each step's reasoning, each tool call, and the verifier's own stdout.

This is a re-packaging, not a new measurement. The same files were published before, split across two releases, which made the line look incomplete in both: the first release carried the main run without the later re-runs, the second carried 11 re-runs without the main run. Nothing here is new data and no number changes. Both source releases remain up and unmodified.

Results

Three tiers, applied by one script (scripts/score4.py) identically to this and the three other lines it was compared against:

Reproduce all three from this repo alone:

python3 scripts/score_local.py     # no dependencies, reads trajectories/ directly

The three tiers:

1. as shipped — the verdicts the run produced. What you get deploying this stack today.
2. + infra trials re-run — a trial whose verifier never ran a single test (broken dpkg, unreachable mirror, container start failure) was really re-run, and the re-run's verdict replaces the original. Nothing is estimated or replayed. Eight verdicts changed this way; score_local.py prints each one.
3. + work-product standard — three trials whose work product satisfied the task's own acceptance criterion, evidenced in the agent's own log while its service was still up, count as solved even though dsh SIGKILLs the process tree before verification. Each is listed with the log line that justifies it, and the same rule was applied to every other line (only dsh had qualifying trials, because only dsh kills them).

Layout

trajectories/
  main/<task>/            89 trials   the run as shipped
  remediation-1/<task>/   17 trials   first re-run pass
  remediation-2/<task>/   11 trials   re-run under the levelled network environment
scripts/                              the scorers and analysis tools
  score_local.py                      reproduces the table above from this repo
  score4.py                           the original four-line scorer, verbatim
  from-first-release/                 the first release's analysis scripts

Per trial:

config.json                 task + harness configuration
result.json                 the verdict
agent/instruction.txt       the task as the model received it
agent/dsh-session.jsonl     dsh's own session log: every step, reasoning and tool call
agent/dsh-result.json       token counts, wall clock, stop reason
verifier/test-stdout.txt    raw output of the task's test script
verifier/reward.txt         the verifier's reward

Re-runs are applied in directory order — remediation-1, then remediation-2 — with later verdicts overriding earlier ones. The 11 tasks in remediation-2 are a subset of the 17 in remediation-1; all 28 are among the 89 in main.

Provenance

Byte-for-byte copies. scripts/score_local.py is the only file written for this release; it exists because score4.py has absolute paths into the benchmark host and scores four lines at once, so it cannot run against a clone.

The network caveat

The main run was not network-levelled. Its apt traffic went through a v2raya proxy and pip through tuna, while the Qwen line it was first compared against had a local disk cache for both. Host egress here is 8–70 KB/s; a cold apt-get install tesseract-ocr-all … measured 755 s, and dsh kills any single command at 300 s, which cuts the dpkg transaction in half and leaves the package database broken — after which the verifier cannot install its own tools and runs zero tests. Through the local cache the same install is 10–25 s.

remediation-2 is the re-run of the affected trials under the levelled environment, which is why it exists at all. The repair raised this line from 65.2% to 68.5% — it was not a thumb on the scale for the newer model that prompted it; it helped every line it was applied to.

dsh's 300 s ceiling was left alone on purpose. It is an ordinary config item (timeoutMs: 300000 in packages/bundle/sdk-minimal/cordis.patch.yml, overridable per-id from $DSH_HOME/cordis.patch.yml), but 300 s is how dsh ships, and dsh is the thing under test.

Serving configuration

8×A800-80GB PCIe, vLLM, image openguardrails/vllm-dsv4-0731-sm80. fp8 KV cache, TP=8 + expert parallel, 1M context, dspark speculative decoding, max thinking (reasoning_effort: "max", asserted before each run with /tokenize). Exact invocation: scripts/start_dsv4.sh, restored byte-for-byte from the container that produced the run.

Harness: agent_timeout_multiplier=6.0, verifier_timeout_multiplier=8.0, temperature 1.0, 4 concurrent trials.

Cost of the main run: 388.6 M input tokens, 4.1 M output, 12.9 h wall clock. Input dwarfs output because every step re-feeds the whole terminal history; dsh sends complete command output back into context, where terminus-2 sends a 160×40 pane capture.

Limits

• One attempt per task. No averaging, and not comparable to vendor-published numbers, which are typically multi-run averages.
• A stochastic dpkg fault survived the network repair and was never isolated — roughly 20–40% per attempt, not task- or model-specific. We ruled out upstream bandwidth, cache warmth, container CPU/memory limits, the container→proxy path, DNS, and a PTY stalling apt's stdin (reproduced with pty.fork(), scripts/pty_apt_test.py). It cost this line 2 tasks. We did not find the mechanism and are not guessing at one.
• The repair is partial by design: only trials that produced no measurement at all were re-run. A trial that produced a verdict keeps it.
• filter-js-from-html failed with a verifier-side Connection refused we could not repair. It cost every line one task equally.
• Token-shaped strings were redacted; no real credential was present.

Related releases

• openguardrails/tb21-qwen3.8-27b-terminus2 — Qwen3.8-27B + terminus-2, and this model on terminus-2
• openguardrails/tb21-dsv41-flash-dsh — DeepSeek-V4.1-Flash + dsh, and all four lines re-scored together