Raw, unedited agent trajectories for three full runs over the same 89 Terminal-Bench 2.1 tasks, plus every remediation re-run, so each number below can be checked against what the agents actually did. The gaps are not statistically significant. Paired over the same tasks: Qwen vs DeepSeek+terminus-2 is net +6 (exact McNemar p=0.29); Qwen vs DeepSeek+dsh is net +3 (p=0.68). At 89 tasks and one attempt each, a lead this size is what run-to-run noise produces. Read the rows as "comparable", not as a ranking. 1. Start from what each run actually scored. 2. Trials lost to infrastructure were re-run under the repaired environment; the re-run's verdict replaces the original. 3. A trial whose work…
Publicly accessible
mit
n<1K
Raw, unedited agent trajectories for the DeepSeek-V4.1-Flash run over all 89 Terminal-Bench 2.1 tasks, plus every remediation re-run for it and for the two older lines, plus the scoring and analysis scripts. Every number in the write-up can be traced to a file here. Exact McNemar, paired over the same 89 tasks: V4.1 is the only line above 70%, and it is 9 points above V4 on the same scaffold — but 89 tasks cannot carry that to significance. The direction is clear; "proven stronger" is not a claim this sample supports. Applied identically to every line, by one script (scripts/score4.py): 1. as-shipped — what the run actually scored. This is what you get if you deploy the same stack today. 2.…
Publicly accessible
mit
n<1K
Every trial of this one line, in one place: the 89-task main run, both re-run passes, and the scoring scripts. The trajectories are raw and unedited — each step's reasoning, each tool call, and the verifier's own stdout. This is a re-packaging, not a new measurement. The same files were published before, split across two releases, which made the line look incomplete in both: the first release carried the main run without the later re-runs, the second carried 11 re-runs without the main run. Nothing here is new data and no number changes. Both source releases remain up and unmodified. Three tiers, applied by one script (scripts/score4.py) identically to this and the three other lines it was…
Publicly accessible
mit
n<1K
