Your Data Is Worth More Trained Than Handed Over

When you train a model on your own data, your documents stay inside the company and the improvement becomes yours. When you hand those documents to an outside frontier model, you can still get a good answer back. However, the data leaves your systems, and nothing about that model becomes your own. We think that difference decides who ends up holding the valuable asset.

So we tested it on our own work. On August 20, 2026, we downloaded Ai2’s fully open OLMo 3. It did not go smoothly: after 24 training rounds on a rented GPU, our scores stalled, and we stopped to fix the problem one piece at a time. Twenty-two days after the download, on September 11, a confirmation run on 36 real SAVRN documents finished. Every final document passed every check, and 92% of first drafts passed before any repair.

Frontier models do great work, and this post is not a head-to-head comparison, because no frontier model wrote these 36 documents. Instead, it is the record of what it took to get client-ready results from our own data, including the mistakes we made along the way.

Why your data is the asset

A company’s own documents carry what no public model has seen: its standards, its figures, its corrections and the way it explains its work. That is why we treat our data as our most valuable AI asset. Every night, our settled work becomes training data, and every correction a person makes to a draft becomes a lesson for the next round. As a result, the system gets better at our work specifically, and that improvement stays with us.

Sending proprietary documents to an outside model works differently. You get an answer, but the documents leave your systems, and the model that produced the answer still belongs to someone else. Meanwhile, open models have closed much of the gap. The 2025 AI Index from Stanford HAI reports that the performance difference between open-weight and closed models fell from 8% to 1.7% on some benchmarks in a single year. In other words, a strong open starting point is now widely available.

How we set it up

We made three choices before any training started. Each one kept the data, the model and the improvement under our control.

A fully open base model

The SAVRN platform runs on a post-trained variant of Ai2’s OLMo 3 32B Instruct model. SAVRN uses OLMo 3 as its base and applies the OLMo 3 post-training flow (supervised fine-tuning, direct preference optimization, and reinforcement learning with verifiable rewards) against SAVRN’s own domain datasets, evaluated on a SAVRN-authored held-out set with a paired assertion-level instrument.

Olmo 3 from Ai2 is a fully open release, so Ai2 publishes the weights, the training data, the training code and the intermediate checkpoints. Open weights alone, however, only let you run a model. By comparison, a fully open release also lets you see where its behavior came from and apply the same published recipe to your own data. That is why we made OLMo 3.1 32B Instruct our writing model on August 21, the day after we first downloaded it.

Our own local GPU

The model runs on our own local GPU, not behind another company’s API. Because production runs locally, an extra model call costs only power, so we could design for quality instead of rationing calls. Training, however, needs far more GPU memory than serving. For that reason, we rented a single cloud GPU for training sessions only and destroyed it when the sessions ended.

A closed network

Every model call from the SAVRN platform passes through one gateway. Its policy allows local models only and denies outbound traffic by default. Therefore, documents and prompts are not sent to an outside model.

What we measured first

On August 20, we built our first quality test: 24 cases modeled on real SAVRN work, including corrections we had made to earlier drafts. Deterministic checks graded every document, including a check that every number in the output appears in the request or the source. OLMo 3 passed 50% the day we downloaded it, close to the 54% of the model then in production.

A model judge scored each document too, but it gave high marks to documents that had failed their checks. So we steered by the deterministic checks instead. That same day, we also built a dataset factory that turns settled work into training data every night. Importantly, it keeps test topics out, so the model never trains on its own exam.

Training on our own data

From August 21 to August 23, we rented a single high-end cloud GPU and ran an automatic loop that post-trained OLMo 3 on SAVRN data. Each round trained a candidate, scored it on our test and promoted it only if it beat the model in production. In about 38 GPU hours and about $170, the loop ran 24 training rounds, and in those first four days we saved 60 evaluation runs.

The first round made the model worse. It passed 25%, against 50% for the stock model, because the recipe was too aggressive for 301 training examples. As a result, the model invented numbers and wrote too little. We responded with more data, a gentler recipe and training only on the answers. The next rounds then used about 9,800 examples generated from our own knowledge base, and each one was checked so that every number in an answer appears in its source passage.

Round 11 was promoted on August 22, passing 58% against 52% for the model in production. That same day, we added a length rule and a memo contract around the model, and the same round 11 model rose from 58% to 65%. In other words, the harness moved the score as far as training had.

Then we hit a wall. None of the 13 rounds after round 11 could beat it. Earlier, four rounds had scored exactly 52%, no better than the untouched model, and the later rounds landed between 48% and 62% while the bar in production had risen to 65%. We kept changing the recipe, the data and the rewards, and still the score would not move past it.

Later rounds tried reinforcement learning with verifiable rewards, where the test’s own checks become the reward. The mechanism worked, although it also taught us a lesson about incentives. Our length reward punished short answers harder than long ones, so the model learned to write too much. Then a round that penalized only short answers made it worse, pushing average answers from 264 to 527 words.

Fixing the test before trusting it

On the evening of August 22, an audit found the test itself miscounting. Of 125 flags for invented numbers across six runs, 90 were list positions, row numbers and ordinals. Correcting only that moved production from 65% to 69% and one candidate from 62% to 71%, which reversed a promotion decision. In addition, the test had only 48 cases, so one case moved a score by about two points, as much as the same model moved between runs. Consequently, on August 23 we destroyed the rented GPU and paused training, because we could not yet tell what was holding the results back.

Next, we rebuilt the test from real documents we had already shipped, and we set aside a held-out group before making any fixes. That held-out group earned its cost right away. It showed that a new layer of document contracts did not help, and it exposed a product bug: when a client named their own sections, they got our standard skeleton instead. We fixed it the same day.

What followed was slower work, one problem at a time. On August 24 alone, we saved 67 evaluation runs while tracing each stubborn failure to its cause. That search found another production bug: the quality gate recognized only Markdown headings, so correct documents with plain-text headings were regenerated again and again. We fixed it that day, and on August 25 we shipped a claim ledger that checks each required figure against the source it came from.

On August 31, a matched evaluation settled the training question. Round 11 passed 25 of 32 cases, or 78.1%, against 22 of 32, or 68.8%, for the stock model, and its unsupported figures fell from 14 to 10. Still, the report reads the gain as consistency and control of unsupported figures, not a lift on every metric. A newer candidate scored 81.3%, but it stays out of production until its privacy and tenant-isolation check has run.

The breakthrough on September 10 and 11

On September 10, we tested round 11 on 36 real documents we had already written and accepted. A document passed only if every required heading and figure was present, it fell inside its word range, every number traced to its source and no internal notes leaked in. The first run passed 19 of 36, or 53%, and most failures looked like invented numbers.

Reading every failure line by line showed three different problems. First, the test was weaker than production, because its repair step skipped the invented-figure check and allowed one attempt where production allows two. Second, the number checker was wrong more often than the model, since it flagged correct figures written another way, such as a hyphenated count or a spelled-out unit. Third, there were real product defects, including drafts cut off by a small output budget and repair notes leaking into documents.

We fixed all three, and three clean runs then passed 36, 35 and 36 of 36. By comparison, only 22% of first drafts passed across those runs, and half of all first drafts were missing a required heading.

The model was leaving whole sections out when asked to follow an outline, so we moved the outline into code. Now the harness writes every required heading, while the model writes one section at a time with the figures that section needs and the source lines they come from. Length is judged across the whole document, and a missing figure gets a narrow call that writes only that sentence.

First drafts rose to 34 of 36. Then one run added a rule that figures be written in digits, with a sample rack count as the example. First drafts fell to 24 of 36, because the model copied the example into documents that never mentioned it.

Consequently, we removed the example from the rule. The confirmation run then passed 33 of 36 first drafts, or 92%, and 36 of 36 final documents, with a median of 4 model calls and 95 seconds per document. Meanwhile, the model weights never changed during those two days.

The mistakes we made

We are publishing the mistakes because they are most of the lesson. Each one cost time, and several of them looked like model problems at first.

• Our first training recipe was too aggressive for 301 examples, so it made the model worse.
• We kept training for 24 rounds before we had verified the test that scored them.
• A length reward that punished short answers harder than long ones taught the model to write too much.
• The test counted list numbers as invented figures, and a promotion decision landed on the wrong side of that error.
• A 48-case test could not tell a real three-point gain from run-to-run noise.
• A document contract replaced the sections a client asked for with a generic skeleton.
• Twice, our evaluation harness measured a weaker system than production.
• The number checker rejected correct figures, in production as well as in the test.
• The output budget was too small, so drafts stopped mid-sentence.
• An example inside an instruction became content.
• We reported some scores before the checker was fully corrected, and then we re-checked every count against the saved results.

One shortcut we did not take: a rule in our test tooling would have removed 14 of the 91 required figures. It would have raised the score by making the test easier, so we left those figures in.

What made the difference

Post-training on our own data made the model itself more consistent, and the matched evaluation shows round 11 beating the stock model. The largest gains, however, came from the harness engineered around that same model: from 58% to 65% on August 22, and then on September 10 and 11, final documents from 53% to 100% and first drafts from 22% to 92%.

That pattern matches Anthropic’s guidance on building effective agents, which describes splitting a task into steps and adding programmatic checks, trading some latency for accuracy. Similarly, Anthropic’s guide to evaluating agents notes that evaluating an agent means “evaluating the harness and the model working together.”

Neither result would have been visible without a test built from our own documents. Because the model runs on our own local GPU inside a closed network, none of this work sent our documents to an outside model. After 22 days, what we have is ours: a post-trained model, a harness and a test, all built from our own data.

What this does not prove yet

These results were tuned on the same 36 documents, so the real confirmation is a fresh held-out set, tested after the system has run stable for a sustained period. That has not happened yet. In addition, section-by-section drafting runs in our test harness today, while production still drafts in a single pass. Finally, no frontier model wrote these documents, so this is not a head-to-head comparison.

What comes next

First, we will move section-by-section drafting into production. Second, we will test on a fresh held-out set once the system has been stable for a sustained period. Third, we will use pairs of first drafts and corrected drafts as training data, so the model learns the discipline itself and needs fewer repair calls. Finally, SAVRN is preparing to fork the OLMo 3 lineage into its own model line, with SAVRN-defined versioning, evaluation and release.

This is part three of our series on training our own model, after Three Levers, One Loop and The Model Was Fine. The Instrument Was Not.. For the economics of running models locally, read The Inference Paradox.