redline-bench-jubarte-trajectories

RedlineBench Agent Trajectories: jubarte vs. Python tools

785 agent runs of Claude Code redlining real Word contracts on all 140 tasks of the Crosby–micro1 RedlineBench. Every row is one attempt at one task, passed or failed. Each row carries everything needed to study it:

• the full parsed trajectory, with reasoning, messages, tool calls and tool results;
• the same trajectory rendered as Markdown;
• every shell command the agent ran;
• the task prompt, the skill instructions and the grounding documents;
• the delivered contract converted to Markdown, with its tracked changes and comments;
• the rubric grade with per-item verdicts and judge justifications;
• tokens (cached and non-cached), timings, estimated cost and the failure reason.

A second table, tasks, has one row per task and setup: the prompt, skill files, grounding text, rubrics, and the source contract and the attorney's reference redline converted to Markdown, in the same notation as the delivered contracts. A third table, incomplete, holds 69 further attempts that Harbor never finished (no result.json), with whatever they left behind.

The dataset was built to answer one practical question: does an agent redline contracts better with jubarte, a purpose-built Rust CLI for Word tracked changes, than with the benchmark's original Python scripts? It also asks a second question: how much do the tool's instructions matter? It compares three setups across two models:

The runs cover the benchmark's 140 tasks in two blocks (task_set):

• first_69: every setup and both models. Haiku has a valid score on all 69 tasks in all three setups; GLM on 62 (instruction two) and 43 (baseline).
• second_71: the remaining 71 tasks, Claude Haiku 5.5 only, baseline and instruction two. Coverage is partial: runs stopped when the API gateway's budget ran out (48 of 71 tasks scored with instruction two, 36 with the baseline, 33 with both). See the second 71 tasks.

The contracts, playbooks and rubrics are fictional SaaS and services deals (AgentCo, LargeCo, GiantCo) written by attorneys for RedlineBench. No real client data is involved.

---

Contents

• Headline results
• Differences and conclusions: the first 69 tasks
• The second 71 tasks (Haiku only, partial)
• Quick start
• What a task looks like
• The three setups
• How the runs were produced
• Composition
• Schema of the trajectories table (68 columns)
• The tasks table
• The incomplete table
• Validity, failures and how to filter
• Where agents struggle with jubarte
• Cost and token accounting
• Known issues and caveats
• Secrets and privacy
• Intended uses
• License and attribution
• Citation

---

Headline results

Each comparison below is paired: it uses only the tasks where both setups have a valid score, taking the best valid score per task. Scores are RedlineBench's weighted rubric score (0–1). Win/loss uses a ±0.005 margin.

What the data shows:

• The tool alone did not help; the instructions did. With its first instructions, jubarte scored the same as the Python scripts (0.393 vs. 0.394 on 69 tasks). With the revised instructions and the same binary, Haiku gained about 0.03 mean and 0.09 median over both.
• The gain is concentrated in the first reply to a counterparty markup. On scenario 1, turn 2, Haiku's mean went from 0.272 (instruction one) and 0.311 (baseline) to 0.470 (instruction two): 14 wins and no losses against instruction one. On later turns (3–4) the revised instructions are even or behind. In the second block, which is mostly later turns, instruction two trails the baseline (0.412 vs. 0.430 mean, 0.333 vs. 0.476 median on 33 shared tasks). The losses are legal choices, not tool failures; see grade_per_rubric_json.
• The revised instructions are cheaper. Against instruction one, Haiku read about 22% fewer cached input tokens and made 17% fewer tool calls per run (median 35 vs. 42). Against the baseline on the second block, it cost about $0.49 vs. $0.64 per run at list price.
• GLM-5.3-flash is time-bound. More than half its attempts hit the agent timeout; it finishes many tasks only after time extensions (see timeout_extension). Its jubarte gain over its baseline is small (+0.019 on 42 tasks). Per run it costs roughly 2–17× less than Haiku, depending on which OpenRouter provider served it, but takes 2–4× longer.
• The effects are small. Single-judge noise at this sample size is about ±0.04 (see the single-judge check). Only the scenario-1, turn-2 gain is large and consistent across models and comparisons.

Compare scoreswithin one model only. Haiku runs were graded by a Claude Haiku 5.5 judge and GLM runs by a GLM-5.3-flash judge (see judging). A GLM re-judge of many Haiku runs is included in glm_rejudge_json for cross-model work.

The previous version of this card (521 runs, first 69 tasks only) reported larger gains: +0.041 for Haiku instruction two over one on 61 tasks, and +0.050 for GLM on 29. Adding the eight missing Haiku scenario-2, turn-2 tasks and 13 more GLM pairs shrank both. The numbers above replace them.

---

Differences and conclusions: the first 69 tasks

Every number in this section is computed from the first_69 rows, using each setup's best valid run per task (the same selection as the Quick start query). Unless noted, scores come from the judge of the run's own model (see Judging).

The setups at a glance

Where the differences come from: scenario × turn

Paired best-valid scores per scenario (s1, s2) and turn (t1 = first draft review, t2 = first reply to a counterparty markup, t3–t4 = later rounds). Δ = A − B; W/L/T with a ±0.005 margin.

What the judge rewarded: pass rate by rubric category

Share of rubric items passed (unweighted), over each setup's best valid run per task. Penalty items (8 runs carry any, none triggered) are omitted. Item counts in parentheses.

How the agents worked: per-run medians (best valid run per task)

Reliability: how often an attempt produced a valid score

Attempts that reached the agent (excludes RuntimeError sandbox failures and operator cancellations).

Haiku's four other failures are UnknownApiError rows from a gateway credit cut-off, not model failures.

GLM by time limit (each tier only reruns tasks still lacking a valid score):

Single-judge check (GLM-5.3-flash re-grade of Haiku runs)

Re-graded by GLM, Haiku instruction one vs baseline on 68 shared tasks: mean 0.463 vs 0.424, median 0.441 vs 0.402, W/L/T 32/27/9. Under the original Haiku judge the same pairs score 0.391 vs 0.392.

Conclusions

1. Instructions, not the tool, moved the score. Swapping the Python scripts for jubarte with the first instructions changed nothing on average (Haiku Δ −0.001 over 69 tasks, 26/27/16). The same binary with revised instructions gained +0.035 over instruction one (32/21/16) and +0.033 over the baseline (35/24/10), and +0.019 for GLM over its baseline (42 tasks, 16/11/15). These averages are near the single-judge noise level (point 8).
2. The gain lives in the first reply to a counterparty markup (turn 2). Haiku instruction two vs one on s1-t2: +0.198, 14 wins / 0 losses; vs baseline +0.159, 13/2. GLM's s1-t2 points the same way (+0.114). On s2-t2, instruction two beats the baseline (+0.051, 8/4) but not instruction one (+0.008, 3/7). Turn 1 (first review of a clean draft) is flat: there are no counterparty changes for the new rule to act on.
3. Late turns are where jubarte loses. Haiku on s1-t4: −0.060 vs baseline with instruction two and −0.064 with instruction one (4/9/2 both); GLM −0.064 on s1-t3 and −0.060 on s1-t4. The second block, mostly turns 3–4, shows the same pattern (s2-t3 −0.109, s2-t4 −0.043; see below). The per-rubric justifications show the losses are negotiation choices (re-opening settled points, not conceding), not tool errors.
4. jubarte makes agents edit far more, and the judge partly punishes it. Median tracked revisions per final document: 36 (Haiku baseline) vs 104–111 (Haiku jubarte); 11 vs 64 for GLM. Pass rate on Deal-closing Orientation drops from 79% to 56–61% for Haiku with jubarte. With instruction two, Negotiation Quality (34% → 44%), Legal correctness (34% → 41%) and Counterparty Acceptance Prediction (44% → 53%) rise. A cheap plan-based editor lowers the cost of every edit; the instructions have to say when not to edit.
5. Instruction two is leaner at equal or better quality. Versus instruction one, Haiku used 17% fewer tool calls (35 vs 42 median), 20% fewer jubarte calls (20 vs 25) and 22% fewer cached input tokens (7.7 M vs 9.9 M) per run, for the same ≈ $0.80 list-price cost. FOREIGN_AUTHOR validation errors fell from 52 to 14 (see Where agents struggle).
6. For GLM, jubarte mostly buys time. GLM with instruction two finished in a median 47 min vs 60 min on the baseline, made fewer tool calls (58 vs 78), had half the tool-error rate (1.9% vs 3.9%), and produced valid results on 44% of attempts vs 33%. Its jubarte errors are interface mistakes (invented or repeated flags, unknown plan fields), not semantic refusals.
7. GLM is the cheaper model per point; Haiku is the reliable one. GLM yields 5.3–6.9 score points per USD (low price estimate) vs ≈0.5 for Haiku, but only a third to half of its attempts finish within the limit, even with 2× and 4× time extensions. Every Haiku attempt that reached the model finished in time.
8. The judge matters as much as the tool. Re-graded by GLM instead of Haiku, Haiku's instruction-one runs beat the baseline by +0.039 (32/27/9) where the Haiku judge saw a tie (0.391 vs 0.392). Single-judge results are noisy at this sample size; treat differences under ≈0.04 as unresolved.

Bottom line. A purpose-built document tool does not improve a legal redlining agent by itself. Paired with instructions that encode the negotiation protocol (leave the counterparty's record intact, answer with your own tracked changes), it gives a large gain exactly where that protocol applies (the first reply to a markup), costs less per run, and cuts time for a weaker model. In later rounds it trails the original tools on both task blocks: the agent edits more than the judge wants. Net over all tasks, the effect is small.

Sample sizes per segment are small (3–15 tasks). The segment tables mark |Δ| ≥ 0.05 in bold to guide the eye, not as a significance test.

---

The second 71 tasks (Haiku only, partial)

The other 71 RedlineBench tasks (scenario 2, turns 3–4, and all of scenario 3) were run afterwards with Claude Haiku 5.5 in two setups: baseline_task and jubarte_instruction_two (the same revised instructions, applied to these tasks in .tasks-skillfix-71). Instruction one and GLM were not run on this block; GLM's API credit had run out. All runs went through the ZnapAI gateway, and they stopped when its spending cap was reached, so coverage is partial:

Paired results (33 tasks with both setups)

Pass rate by rubric category and per-run medians, over the same 33 tasks (best valid run per task):

Reliability over all attempts that reached the agent: baseline 42 attempts, 36 valid; instruction two 53 attempts, 48 valid. None timed out. The 11 failures are 10 UnknownApiError rows (the gateway refusing requests at its spending cap) and one RewardFileNotFoundError.

What this block shows:

• Instruction two wins more tasks but loses by more. On the 33 shared tasks it wins 12 and loses 8, yet its mean is 0.018 lower and its median 0.14 lower.
• The later-round weakness repeats. s2-t3 (−0.109) and s2-t4 (−0.043) match the s1-t4 losses in the first block. Only s3-t1 and s3-t2, the earlier turns of scenario 3, favour instruction two, and both setups score very low on s3-t1 (0.166 vs 0.038).
• s3-t3 has no baseline score. Instruction two scored 10 of its 15 tasks (mean 0.404). Its grades show legal-judgment losses, not tool errors. It struck concessions that favoured its own client, such as a liability cap raised from 4× to 5× fees (g04a, g06a). It narrowed clauses the rubric wanted deleted, deleted the insurance article instead of adding coverage, and left counterparty insertions standing.
• It is cheaper and faster. Fewer tool calls (29 vs 42), 39% less cached input, and about $0.49 vs $0.64 per run.

Unpaired means per segment, over every task each setup scored: instruction two s2-t3 0.363 (9), s2-t4 0.422 (12), s3-t1 0.166 (5), s3-t2 0.538 (11), s3-t3 0.404 (10), s3-t4 1.0 (1); baseline s2-t3 0.472 (9), s2-t4 0.466 (12), s3-t1 0.038 (5), s3-t2 0.540 (9), s3-t4 1.0 (1).

---

Quick start

from datasets import load_dataset

ds = load_dataset("arthrod/redline-bench-jubarte-trajectories", split="train")           # trajectories
tasks = load_dataset("arthrod/redline-bench-jubarte-trajectories", "tasks", split="train")  # one row per (setup, task)
row = ds[0]
print(row["setup"], row["model_family"], row["task_set"], row["task"], row["reward"], row["valid"])
print(row["commands_executed"][:5])
print(row["trajectory_md"][:2000])

The trajectories file is about 289 MB, and most of it is trajectory text. Load only the columns you need:

import polars as pl

url = "hf://datasets/arthrod/redline-bench-jubarte-trajectories/data/redline_bench_trajectories.parquet"
meta = pl.read_parquet(url, columns=[
    "setup", "model_family", "task_set", "task", "reward", "valid", "exception_type",
    "agent_seconds", "cached_input_tokens", "non_cached_input_tokens", "output_tokens",
    "est_cost_low_usd", "est_cost_high_usd", "n_tool_calls", "n_jubarte_calls",
])

# best valid score per task, per task block, setup and model
best = (meta.filter(pl.col("valid"))
            .group_by(["task_set", "model_family", "setup", "task"]).agg(pl.col("reward").max()))
print(best.group_by(["task_set", "model_family", "setup"]).agg(
    pl.len().alias("tasks"), pl.col("reward").mean().alias("mean"), pl.col("reward").median().alias("median")))

Put a delivered contract next to the source contract and the attorney's reference redline (all three use the same Markdown notation):

turl = "hf://datasets/arthrod/redline-bench-jubarte-trajectories/data/redline_bench_tasks.parquet"
tasks = pl.read_parquet(turl, columns=["setup", "task", "source_contract_md", "attorney_redlines_md", "rubrics_json"])
runs = pl.read_parquet(url, columns=["setup", "task", "trial_name", "reward", "valid", "final_contract_annotated_md"])
j = runs.filter(pl.col("valid")).join(tasks, on=["setup", "task"], how="left")
r = j.row(0, named=True)
print(r["task"], r["reward"])
print(r["final_contract_annotated_md"][:1500])
print(r["attorney_redlines_md"][:1500])

You can also query the files with DuckDB SQL:

SELECT task_set, setup, model_family, count(*) AS runs, avg(reward) FILTER (WHERE valid) AS mean_score,
       median(agent_seconds)/60 AS median_minutes, sum(est_cost_low_usd) AS est_cost_usd
FROM 'hf://datasets/arthrod/redline-bench-jubarte-trajectories/data/redline_bench_trajectories.parquet'
GROUP BY ALL ORDER BY task_set, model_family, setup;

Every jubarte command that failed, with what the tool said:

import json
for r in ds.filter(lambda r: r["n_jubarte_calls"] > 0).select(range(20)):
    for c in json.loads(r["commands_json"]):
        if c["uses_jubarte"] and c["is_error"]:
            print(r["task"], "|", c["command"][:120], "\n   ->", c["result_head"][:200])

---

What a task looks like

RedlineBench tasks drop the agent into a live, multi-turn contract negotiation. The agent is in-house counsel for one side, at a specific turn. Its inputs are the contract as it stands (a .docx, at turn 2+ already carrying the counterparty's tracked changes and comment threads), its side's playbook and commercial context (grounding_md), and the task prompt (task_instruction_md). It must deliver the same .docx with native Word tracked changes and margin comments at /app/contract.docx.

A judge model grades the delivered file against attorney-written rubrics: 8–21 weighted items per task, such as "Inserts in Section 3.1 that the Agreement does not pre-approve future services…" or "Accepts the preamble in its entirety." A gate first checks that the document actually contains tracked changes or comments by the agent's author name; a failed gate scores 0.

Task ids encode the negotiation tree: redline-s{scenario}-t{turn}-g{group}{variant}. Variants of one group (g01a, g01b, …) see identical inputs and differ only in which attorney's rubric grades them.

All 140 tasks, by block:

in_jubarte_69_tasks is true exactly for the first_69 rows.

---

The three setups

baseline_task: the benchmark's original tools

This is the unmodified RedlineBench contract-redliner skill. The agent edits the .docx through bundled Python scripts: read_document.py for Markdown with paragraph ids, plus scripts for tracked replacements, insertions, deletions, comments and replies. Its SKILL.md is about 4.5 KB. n_jubarte_calls is 0 for these rows.

jubarte_instruction_one and jubarte_instruction_two: jubarte

jubarte is a single static Rust binary (Linux arm64, version 0.12.0) shipped inside the task at /skills/contract-redliner/bin/jubarte. Its subcommands are:

• read: Markdown or JSON with paragraph, revision and comment ids, and CriticMarkup for tracked changes.
• edit: single edits or a JSON plan of operations such as replace, insert, delete, insert_paragraph, delete_paragraph, comment and reply, applied as tracked changes.
• changes and comments: list the tracked changes and comments.
• validate: Word validity, plus --original/--author to check that every change is the agent's own and is tracked.
• Also compare, convert and scrub.

The prompt starts with "USE THE JUBARTE TOOL FOR MODIFYING THE DOCUMENT…", and the full --help text of every subcommand is provided in /app/jubarte-help-all.md.

Both jubarte setups use the byte-identical binary (sha256 87fc217e540c952e…), in both task blocks. They differ only in the instructions.

What changed between instruction one and two

You can diff the exact texts per row with the skill_instructions_md and task_instruction_md columns, or per task in the tasks table.

---

How the runs were produced

• Harness: Harbor running the claude-code agent (Claude Code 2.1.306–2.1.310) in a Docker sandbox per trial. Up to 10 retries per trial on infrastructure errors. Concurrency was 6–24 trials.
• Claude Haiku 5.5 (claude-haiku-5-5), reasoning_effort=max, summarized thinking. It was served through the Anthropic API directly for jubarte_instruction_one and most of the first-block baseline_task, and through API gateways (Kilo, LLM Gateway, ZnapAI; see model_provider_base_url) for jubarte_instruction_two, the rest of the first-block baseline, and all of the second block (ZnapAI). The model is the same in every case; the gateway only changes the endpoint.
• GLM-5.3-flash (z-ai/glm-5.3-flash:floor, OpenRouter's cheapest-provider routing), reasoning_effort=high. CLAUDE_CODE_MAX_OUTPUT_TOKENS=128000. A Stop hook sends the agent back to work, at most 5 times, when it tries to end the run without changing the contract; GLM otherwise often stopped to ask "shall I proceed?".
• Time limits (agent_timeout_multiplier / timeout_extension). The base limit is 60 min. GLM tasks that timed out were rerun with one extension (2×, 120 min) and then two extensions (4×, 240 min). Each extension tier covers only the tasks still lacking a valid score, so tiers are not the same task set. Haiku never timed out and has only the base tier.
• Relaunches. When runs were stopped (gateway budget caps, a disk-space guard, a host reboot), new Harbor jobs were started for the tasks still lacking a valid score. Each job is a separate job value; nothing was deleted, so stopped attempts stay in the data as CancelledError rows.

Judging

• The judge is the RedlineBench LLM judge, run as a "panel" of one model. Haiku runs were graded by claude-haiku-5-5. GLM runs were graded by glm-5.3-flash with a 3× verifier timeout. The judge used is recorded in grade_json.judge_model.
• Each rubric item gets PASS/FAIL plus a written justification (grade_per_rubric_json, judges_json). The reward is the weighted share of passed items, after the authorship and tracked-change gate.
• glm_rejudge_json: a separate GLM-5.3-flash re-grade of the same delivered document. It is present for 218 first-block rows: 68 Haiku baseline, 68 Haiku instruction-one, 5 Haiku instruction-two, and the 77 GLM runs that were valid in the first snapshot. Use it for cross-model comparisons with a single judge.

Composition

First 69 tasks:

Second 71 tasks:

Means in these two tables are over every valid run, unpaired; the second-block means cover different task sets. Median minutes are over runs where the agent started; tool-call medians are over runs with at least one tool call.

Rows by failure reason:

Runs span 2026-10-21 09:45Z to 2026-10-24 02:29Z.

---

Schema of the trajectories table (68 columns)

Identity and setup

Timing

Outcome and grade

Tokens and cost

Agent behaviour

Trajectory and documents

Where both exist, the verifier's file and the artifact rendering were spot-checked to be byte-identical.

---

The tasks table

data/redline_bench_tasks.parquet (config tasks, 420 rows, 27 columns, 1.5 MB): one row per (setup, task), 140 tasks × 3 setups. It holds everything the agent was given and everything the grader used, as text. Join it to the trajectories on (setup, task). The second_71 rows exist for all three setups, although instruction one was never run on them.

Document notation

Word documents are converted with the verifier's own renderer (tests/annotated_view.py, identical in every task). It is the same renderer that produced final_contract_annotated_md, so the source contract, the attorney's reference redline and every delivered contract share one notation:

• [p-NNN] (Style) starts each paragraph: its id and Word style. {REVS,CMTS} flags paragraphs with tracked changes or comments.
• ++inserted text++ and ~~deleted text~~ are tracked changes; {cmt-N} marks a comment anchor.
• ## Comment appendix lists every comment: id, author, the comment it replies to, paragraph and text.
• ## Tracked-change appendix lists every revision: id, kind, author, paragraph and text.

Columns

The counts are rendered runs: one insertion split across runs or authors counts more than once. Use the tracked-change appendix for exact revisions.

---

The incomplete table

data/redline_bench_incomplete_trials.parquet (config incomplete, 69 rows, 7 MB): attempts that never got a Harbor result.json, because the job was stopped, the host restarted, or Harbor died mid-trial. They are kept out of the main table so its counts and scores are unchanged. The columns are the trajectories table's 68 plus incomplete_reason.

Most (40) come from two Haiku instruction-two jobs on 2026-10-21 that were stopped when their gateways (LLM Gateway, then Kilo) started failing with API errors; the run then moved to another gateway. Another 15 are CancelledError attempts from the first Oct 21 baseline and instruction-one jobs, cancelled before the agent started. The 5 instruction-one rows among them are the only instruction-one attempts on second-block tasks.

How the columns are filled without result.json:

• Task, model, base URL, reasoning effort, time limit: from the trial's config.json.
• exception_type, exception_message: the last Type: message line of Harbor's exception.txt (30 rows); null when there is none.
• Trajectory and commands (48 rows): trajectory.json where Harbor wrote one (13), else rebuilt from the Claude Code stream log (35).
• Tokens (48 rows): summed from Claude Code's session log, one usage per message id, which is how Harbor computes them. This was checked to reproduce Harbor's totals exactly on 12 completed trials. Haiku costs are priced at the long-prompt rate, because the stream log of a stopped run has no per-request usage (see est_cost_basis).
• agent_started_at, agent_finished_at: first and last message timestamps in the stream log. started_at, finished_at: modification times of the trial's files, so treat them as approximate. task_checksum and the per-phase durations are null.
• Delivered contract (22 rows): the verifier's file (2) or the delivered .docx converted with the verifier's renderer (20).
• valid is always false, even where a verifier reward exists.

---

Validity, failures and how to filter

• Score only valid rows. That is RedlineBench's rule: a reward with no exception. Timed-out runs often have a reward (the verifier graded the partial document), but they are not valid results.
• One task can have several rows per setup: retries, the timeout tiers, relaunches, and runs repeated after a gateway switch. For per-task comparisons, take the best valid row per (model_family, setup, task), as the tables above do, or the most recent.
• RuntimeError rows (86) never reached the agent. Docker failed to start the sandbox, so n_steps = 0 and the trajectory and command columns are null. Drop them for behaviour analysis, but keep them if you study infrastructure reliability.
• CancelledError rows (122) are runs stopped from outside: a gateway credit cap, a disk-space guard, a host reboot, or a restart at a different concurrency. Most (104) are in the second block, which was relaunched several times. Their trajectories are partial and some never reached the model.
• UnknownApiError rows (16: 14 Haiku, 2 GLM) are runs whose API gateway refused requests, mostly at its credit cap.
• NonZeroAgentExitCodeError rows (7) are GLM baseline runs at the 4× tier where Claude Code exited with an error. Two of them had a truncated trajectory.json, so their trajectory was rebuilt from claude-code.txt.
• Retries overwrite. Harbor reruns a failed trial in the same directory, so a row holds only the last attempt of that trial.

---

Where agents struggle with jubarte

Counts of jubarte errors in the trajectories of the jubarte setups, by class. One count per jubarte tool result whose output contains an error line, classed by the first matching error pattern. "GLM" is GLM with instruction two.

Patterns worth knowing:

• GLM mostly makes interface mistakes. It invents flags, repeats flags, writes the output over the input, and adds unknown plan fields. Haiku mostly hits semantic refusals: editing a paragraph the counterparty already changed, or overlapping edits.
• Revision-attribution errors (FOREIGN_AUTHOR) fell from 52 to 14 between Haiku's instruction one and two on the first 69 tasks, over more runs (79 vs 69). This is the effect of the "never accept or reject the counterparty's changes" rule.
• A real jubarte bug shows up in these logs. The 0.12.0 build splits a run that carries another author's formatting revision (w:rPrChange), or edits text inside another author's tracked insertion, and writes the same w:id on both halves. validate then reports DUPLICATE_REVISION_ID together with FOREIGN_AUTHOR.
• Search commands_json for DUPLICATE_REVISION_ID to find affected runs.
• validate --repair renumbers the duplicate ids, but the renumbered copies then fail validate --original as new counterparty revisions.
• In most affected runs Haiku diagnosed the problem itself, restored the original and re-planned around it. The binary was kept fixed for every run so all setups stay comparable.
• For Haiku instruction two these errors rose from 14 to 37 once the eight scenario-2, turn-2 reruns were added. Those documents carry many counterparty revisions, the condition under which the bug appears.
• The second block has far fewer jubarte errors per run (77 over 85 runs, against 202 over 79 in the first block). Many second-block runs were cancelled part-way, so they had less time to make errors.
• Haiku output-limit loops. At reasoning_effort=max, Haiku sometimes spends a whole reply thinking and is cut at the output-token limit (output_limit_cuts). After three cuts in a row, Claude Code ends the run and Harbor retries it.

---

Cost and token accounting

• Token counts are sums over every model request in the run. Each request re-reads the conversation, so cached input runs into the millions.
• Claude Haiku 5.5 list price (per million tokens): $0.10 input / $0.50 output when the prompt is ≤100K tokens, and $0.50 / $2.50 above that. Cache reads cost 0.1× the input rate and 5-min cache writes 1.25×.
• Cost is computed per request from Claude Code's logged usage, choosing the tier by each request's prompt size.
• For gateway runs whose logs lack per-request usage, the totals are priced at the long-prompt rate (noted in est_cost_basis).
• Actual gateway bills can be lower. One gateway billed about 37% of the list-price estimate.
• GLM-5.3-flash has no single price. est_cost_low_usd uses the cheapest OpenRouter endpoint, where :floor routes first ($0.0052 input and cache / $0.1985 output per MTok). est_cost_high_usd uses Z.AI's own endpoint ($0.15 input / $0.03 cache / $0.50 output).
• Judge cost is not included anywhere.

---

Known issues and caveats

1. Different judges per model. Haiku runs are graded by Haiku and GLM runs by GLM, so absolute scores are not comparable across models. glm_rejudge_json provides a single-judge view for part of the first block.
2. The extension tiers select for harder tasks. The 2× and 4× GLM rows contain only tasks that failed at the shorter limit, so do not read tier-to-tier score differences as the effect of time alone.
3. The second block is partial and Haiku-only. 48 of 71 tasks have a valid instruction-two score and 36 a valid baseline score; 33 have both and 20 have neither. s3-t3 has no baseline score, and s3-t4 has one task per setup. Runs stopped at the gateway's spending cap. Instruction one and GLM were not run on this block.
4. GLM coverage is partial. On the first block, 62 of 69 tasks (instruction two) and 43 of 69 (baseline) have a valid score, and 42 have both.
5. Many stopped runs. 122 CancelledError rows, mostly in the second block, come from relaunches after budget caps, a disk guard and a host reboot. They are kept for completeness.
6. Gateways. Some Haiku runs went through third-party gateways (model_provider_base_url). The model is the same, but latency and rate limits differ.
7. One 1-judge panel. RedlineBench supports multi-judge panels; these runs used a single judge for cost reasons.
8. Heuristic columns. n_tool_errors, n_jubarte_calls and the error classes above are text heuristics over tool calls and results, not ground truth.
9. Grounding PDFs are not embedded. Use their .extracted.md text in grounding_md, or fetch the PDFs from the source dataset.
10. Two tasks have no reference redline. redline-s2-t4-g03a and redline-s3-t4-g01a ship no attorney_redlines.docx upstream; their rubrics are intact.
11. Numbers changed from the previous version. The first release (521 runs) reported larger gains on fewer tasks. See the note under Headline results.

---

Secrets and privacy

• All API keys used to produce the runs were kept out of the data. Every string column of all three tables is passed through a redactor that replaces sk-… keys, Bearer … tokens, JWTs, Hugging Face, GitHub and AWS key patterns with [REDACTED].
• Both files were scanned with gitleaks 8.30 over a text export of every row. The tasks table had no matches. The trajectories table has seven, all reviewed as false positives: each is ordinary contract wording quoted in a trajectory (several English words separated by spaces). Six were matched by the generic-key rule and one 8,400-character clause passage by the Square-token rule. The incomplete table had no matches.
• As a second check, every cell of the three tables was searched for the literal values of the API keys and tokens in the author's environment files. None appears.
• The contracts, parties and playbooks are fictional, written for RedlineBench. No personal data is included.

---

Intended uses

• Agent and tool-design research: how instruction changes to a CLI tool change agent behaviour, with the same model and the same binary.
• Failure analysis: expectation-vs-reality errors (invented flags), silent tool failures, and recovery strategies, all with the exact commands and outputs.
• Document-level evaluation: delivered contracts, source contracts and attorney reference redlines in one Markdown notation, ready to diff.
• Training signal: filtered successful trajectories for imitation learning, or paired (better/worse) runs of the same task for preference data. Mind the license attribution, and the fact that these are benchmark tasks: do not train on this data and then report RedlineBench scores.
• Judge research: per-item rubric verdicts with justifications from two judge models on the same documents.
• Cost and latency modelling of long-horizon agent runs.

Do not use the legal content as legal advice. These are benchmark negotiations, and many runs are graded as failing.

---

License and attribution

This dataset is released under CC-BY-4.0, the same license as its source.

It is derived from Crosby–micro1 RedlineBench: - Dataset: crosbylegal/RedlineBench, CC-BY-4.0, © 2026 Crosby Legal. The tasks, contracts, grounding documents, rubrics, attorney reference redlines and grader belong to it. The tasks table redistributes them, with the Word documents converted to Markdown. - Report: intelligence.crosby.ai/benchmark.

The trajectories, tool outputs, grades and derived columns were produced by the dataset author running the benchmark. Claude is a model by Anthropic. GLM-5.3-flash is a model by Z.AI. jubarte is a separate tool and is not part of RedlineBench.

Citation

If you use this dataset, cite RedlineBench and this dataset:

@misc{redlinebench2026,
  title        = {Crosby--micro1 RedlineBench: Measuring Contract Negotiation as a Sequence of Judgment Calls},
  author       = {{Crosby Legal} and {micro1}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/crosbylegal/RedlineBench}}
}

@misc{redlinebench_jubarte_trajectories2026,
  title        = {RedlineBench Agent Trajectories: jubarte vs. Python tools},
  author       = {Rodrigues, Arthur Souza},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/arthrod/redline-bench-jubarte-trajectories}}
}