A LoRA adapter and trained answer heads for Qwen/Qwen3.8-27B that turn a document plus a set of
structured questions into one probability per decision, each with a retrieval pointer to where the
support for it plausibly sits in the source. It does not generate text.
Read this before you trust a number on this page
Every document accuracy and calibration figure here was measured on documents this project
generated synthetically. The only evaluation against real documents scored 0.8131
whole-question agreement, not the ≈99% the synthetic figures show, and the loss is concentrated in
two of the five answer types: entity 0.4359 and
multi-label 0.7000.
The probability calibration does not transfer out of domain. On general-knowledge benchmarks this
model is more overconfident than the base model it is built on.
Both are set out in full under Limitations, below. Nothing on this page is a certified error rate
and nothing is guaranteed.
What it is, and what it is for
Give it a document once and ask structured questions against it. Each answer comes back as a
probability, and with source spans beside it. The answer is read from the whole document; the spans
are a separate word-overlap search over the source text. They are a pointer to where the support
probably is, not the passage the answer came from — Evidence spans, below, measures the difference.
There is no chat, no reasoning trace and no sampling: every answer is read from letter logits at a fixed
position through trained linear heads, so the same document and the same question return the same
numbers every time.
Five answer types:
| Type |
Question shape |
What comes back |
| Yes / no |
does the document establish X? |
one probability |
| Single choice |
which of these does it state? |
one probability over the listed options |
| Ordered choice |
which threshold does it state? |
one probability over the ordered options |
| Entity |
which of these parties is the X? |
one probability per candidate |
| Multi-label |
which of these apply? |
one probability per candidate |
It is for turning documents into structured, machine-readable answers where you need a number
attached to each one, and where determinism and a refusal to drift matter more than fluency.
It is not for general knowledge question-answering, chat, generation, or summarisation. It is not
for any setting where the cost of a confident wrong "No" on something the document never mentions is
high — that is this model's worst measured failure and it is documented below.
How it works
|
|
| Base model |
Qwen/Qwen3.8-27B, Apache-2.0, pinned revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. Not redistributed here |
| Adaptation |
LoRA, rank 64, question-side placement, float32 |
| Answer projection |
trained linear heads (adapter/heads.npz), not the language-model head |
| Readout |
structured letter logits; yes/no-shaped units collapse to a binary log-odds before temperature |
| Calibration |
one positive scalar per answer type, applied to the logits before the softmax |
| Runtime identity |
a 15-key binding that refuses to load if the engine is not the one measured |
The document is prefilled once; the questions branch off it. The document goes through the model a
single time and becomes a reusable state (POST /states). Every question is then answered as an
isolated branch off that prefix. Two consequences are worth stating because they are tested on real
hardware and not merely intended: asking the same questions in a different order returns the same
answers, and answering from the cached document state matches a full forward pass to within 1e-3.
The adapter is off while the document is read, and on from the question onward. That is what
question-side placement means. Applying it across the whole sequence gives a different model to
the one that was measured, and the identity binding exists partly to stop that happening by accident.
The readout is structured, not generated. Rather than sampling an answer and parsing it, the model
is asked to commit at a fixed position and the letter logits at that position are read through trained
heads (readout mode four_collapsed). Yes/no-shaped units — a yes/no question, and each individual candidate
inside an entity or multi-label answer — are collapsed to a single binary log-odds,
p = sigmoid(z / T) with z = log P(yes)/P(no), before the temperature is applied. Choice questions
apply the temperature to the listed slice, softmax(logits[:n] / T). The ruled temperatures are
boolean 1.9476 · single 1.0 · ordered 1.5063 · multilabel 1.6895 · entity 2.1834. Single choice ships at 1.0 deliberately: at its accuracy there is essentially nothing to
calibrate and a fitted value there would be noise.
A branch whose head_key is not in the calibration artifact's map is refused, not served at an
assumed 1.0.
The temperatures live inside the runtime binding, covered by its checksum. A calibration fitted on
one model may not be served on another: the loader refuses by name rather than serving scalars that
mean nothing. Serving at temperature 1.0 everywhere is always permitted, on any model.
What the returned score means
ordering_score means two different things depending on the question, and the difference matters.
- Single-unit — a yes/no question, a single or ordered choice, and every per-candidate value
inside an entity or multi-label answer. Here the score is the tempered readout probability, and the
held-out synthetic panel below validates it as a calibrated magnitude.
- Multi-unit — the rolled-up question-level score for an entity or multi-label question with more
than one candidate. It is the product of the per-candidate probabilities, which assumes those
candidates are independent. That assumption has never been validated as a joint probability. It
orders such questions well; it is a heuristic ordering, not a calibrated joint. If you need a
magnitude for one of these, read the per-candidate numbers.
The field is not called probability because that would be accurate for the first case and an
overclaim for the second.
There is no abstention. Every question is answered. The service will not emit a field named
abstain, confidence, threshold or certified_error_rate; it raises rather than return one. If
you want to decline low-confidence answers, that is your policy, made on your population, and this
release makes no claim about where to put the line.
How to run it
The package is a library, not a daemon: you build the engine, wrap it in the serving layer and start the
HTTP surface in four lines. It needs one CUDA device with enough memory for the base model in float32.
pip install -r requirements.lock
huggingface-cli download DoccyHealth/Solomon --local-dir ./solomon
huggingface-cli download Qwen/Qwen3.8-27B --revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 --local-dir ./solomon/base
cd solomon
export PYTHONPATH=$PWD/src
Run from inside that directory: the engine looks for the base model in base/, and the serving layer
reads serving/selection.json and the binding it names, both relative to the package root.
from solomon import api, service
from solomon.serving import ServiceEngine
engine = ServiceEngine('adapter/adapter.safetensors', 'adapter/heads.npz')
layer = service.service('./store', engine) # './store' holds cached document states
server = api.serve(layer) # GET /health, POST /states, POST /v1/decide
print('http://127.0.0.1:%d' % server.server_port)
server.serve_forever() # or skip the server and call layer.decide(...) directly
ServiceEngine also takes expected_adapter= and expected_heads=, the two sha256 values printed
below; pass them and the engine refuses to start on a file that is not the one this card describes.
GET /health reports the contract, the readout mode and the runtime identity. If the engine you built
differs from the one the numbers were measured on — a different torch build, a different adapter, a
different arithmetic mode — the binding refuses to load rather than quietly serving different
numbers. That is intended behaviour; do not work around it.
What is in this repository
README.md this file
LICENSE Apache-2.0
NOTICE attribution and the Apache-2.0 4(b)/4(c) notices
MODIFICATIONS.md statement of changes, third-party text scan, maintainer notes
MANIFEST.json every file, its size and its sha256
requirements.lock the pins of the image this was qualified on, and dependency licences
licenses/Qwen-Apache-2.0.txt the upstream licence, verbatim
adapter/adapter.safetensors the LoRA adapter
adapter/heads.npz the trained answer heads
adapter/config.json base repo, pinned revision, checksums, placement
serving/serving-binding.json runtime identity binding and the frozen temperatures
serving/readout-temperature-v2.json the calibration artifact, standalone
serving/selection.json readout mode and model identity
serving/service-export.json qualification envelope hashes
src/solomon/__init__.py solomon: a document plus structured questions in, one probability per decision out
src/solomon/api.py HTTP surface for the Solomon layer (solomon/service.py)
src/solomon/binding.py serving identity: which readout is served, and proof that it is the one that was measured
src/solomon/calibration.py the readout calibration: one positive scalar per answer type, and nothing else
src/solomon/engine.py question-only CUDA runtime for immutable semantic-head checkpoints
src/solomon/engine_contract.py the answer contract engine: contract v3 on the reference engine's float32 cached path
src/solomon/engine_cuda.py CUDA contract-v3 engine
src/solomon/engine_numerics.py versioned CUDA repair: bounded FP32 attention, FP64 recurrent accumulation
src/solomon/engine_reasoning.py the confidence layer text reasoning on immutable answer-contract base-prefix states
src/solomon/engine_reference.py the reference engine: task-agnostic document prefix, float32 arithmetic, chunked prefill
src/solomon/evidence.py deterministic source references and explicit evidence interventions
src/solomon/evidence_packages.py evidence packages (v3): selected spans plus source-derived governing context, per question unit
src/solomon/heads.py final normalized feature extraction, preserving the qualified CUDA engine
src/solomon/prompts_two_letter.py two-letter (Noul) prompts and block conversions
src/solomon/readout.py contract v3 readouts: branch jobs for every answer type, predictions from letter logits, and metrics
src/solomon/reliability.py what the readout says about its own answer, with nothing fitted behind it
src/solomon/retrieval.py inference-only source candidates and explicitly labelled retrieval baselines
src/solomon/routing.py real callback-driven escalation
src/solomon/semantics.py answer semantics (design note, not distributed)
src/solomon/service.py decision layer: the Solomon serving contract over the pinned readout chain
src/solomon/service_answers.py confidence-aware five-task service, reusing immutable answer-contract input persistence
src/solomon/service_checked.py backend-neutral contract-v3 service; restart replays immutable inputs, not tensors
src/solomon/service_evidence.py optional source-grounded evidence around the existing confidence service
src/solomon/service_heads.py trained-head fast/views service; unchanged decoder and separate stage confidence
src/solomon/service_packages.py evidence packages (v3) in the answer service: per-unit, source-grounded, page-referenced
src/solomon/service_states.py local contract-v3 prototype: task-neutral text/image states and five answer types
src/solomon/serving.py preserve five-task trained-head confidence/routing with optional evidence
mlx/ the optional Apple-silicon package: its own library, tests and notices
What is deliberately not here: no training data, no evaluation panels, no document corpora, no
datasets, no cached states, no rendered images, no score archives, no test fixtures, no logs and no
base model weights. The package is default-deny: a manifest names every permitted file and the build
fails if anything else is present.
The files under serving/ carry identity only: hashes, the readout mode, the served design and the
five temperatures. The loader verifies the binding's checksum and every runtime key it carries. The
measured provenance behind those hashes (which panels, which fit, which qualification run) is held in
the maintainer's records and is not distributed.
Evidence spans
Every answer carries source spans, and evidence is on by default (evidence='support'). This section
is what that is worth, measured rather than described. The short version: the spans are exact, cheap and
not the model's reasoning. The answer is read from the whole document. The spans are a word-overlap
search over the source text, run on the CPU, that the model never sees at the default level. They are
worth having as a place to start reading. They are not a justification, and this page does not present
them as one.
A span is an exact slice of your document. start, end and text, counted in Python code points,
re-validated against the source before they are returned — a span that no longer matches the document is
raised as an error, never returned as a result. Nothing is paraphrased, summarised or generated. Entity
and multi-label answers additionally carry candidate_evidence, one span list per candidate — but with
the shipped selector those lists are identical for every candidate, which is measured below and is a
trap worth knowing about before you build on the field.
At the default level the model never reads the spans. The shipped selector
(solomon.retrieval.lexical_select) ranks the document's sentences by plain word overlap with the question
and keeps the top four per answer unit. No model call is involved, and the cost proves it: in the run below,
evidence='support' and evidence='none' produced identical usage — same branches, same input tokens,
zero evidence calls — and finished 4.08 s against 4.04 s. Whatever the spans are, they had no part in
producing the answer.
The service says this about itself rather than leaving it to the documentation: ask with
evidence_detail=True and the response carries method: "lexical_overlap",
verification: "retrieval_only" and faithfulness_established: false. The model is engaged only at
'sufficiency' and above, where the selected text is re-prefilled as a fresh document and the answer
re-derived from it. Pass selector= to service.service(...) to supply a better one.
The four levels, and what each costs
An answer unit is whatever gets one probability: one for a yes/no or a choice question, one per
candidate for entity and multi-label.
evidence= |
Extra forward passes |
What comes back |
'none' |
none |
evidence_status: "not_requested" and no spans |
'support' (the default) |
none — retrieval only, the model is not called |
the spans |
'sufficiency' |
one fresh document prefill per answer unit |
the spans, plus the answer re-read from that evidence alone |
'removal' |
one per answer unit plus one per question |
the above, plus the answer re-read from the document with every selected span cut out |
sufficiency rebuilds each unit's spans, with automatically attached governing context, into a small
standalone document capped at 30% of the source, and answers that unit again from it and nothing else.
removal adds one further pass per question over the document with the spans deleted. Both re-encode
from scratch: a cached state cannot demonstrate removal.
Two runs measured this on hardware. The live acceptance run, on 6 questions carrying
10 answer units, cost 0, 10 and 16 fresh document
prefills for the three levels. The real-document run below, on 6 questions carrying
14 units, is where the wall-clock difference shows:
|
'none' |
'support' |
'sufficiency' |
| Fresh document prefills |
0 |
0 |
14 |
usage.evidence_calls |
0 |
0 |
14 |
| Answer branches / input tokens |
14 / 2,085 |
14 / 2,085 |
14 / 2,085 |
| Extra tokens spent on evidence |
0 |
0 |
14,918 |
| Wall clock |
4.04 s |
4.08 s |
21.03 s |
'support' is free. 'sufficiency' cost about 5x
the wall clock on that document. 'removal' has never been timed — the acceptance run counted its
prefills, nothing has measured its tokens or its latency, and this page will not guess.
evidence_max_calls (default 64) is checked before any call is made. Over budget, the level is
skipped: evidence_status comes back "budget_exhausted" and the answer is unaffected.
evidence_detail=True adds the package record — every span with its role and the reason it was attached,
the fraction of the source covered, the selector's identity — and costs nothing extra. The response's
usage.evidence_calls reports what was actually spent.
Evidence is text only. A state containing images comes back with
evidence_status: "unsupported_page_selector" unless you supply a page_selector= of your own, and no
image claim is made anywhere in this release.
What comes back
layer.decide(state_id=key, evidence='removal',
questions={'certified': 'Does Rookwood Ltd hold a current certification?'})
{
"answers": {
"certified": {
"type": "noul",
"noul": 0.9913, // the probability
"ordering_score": 0.9913,
"temperature": 1.9476,
"evidence_status": "found", // or no_support_found / budget_exhausted / not_requested
"evidence": [
{"start": 1840, "end": 1906,
"text": "Rookwood Ltd holds certification RW-4471, current to 30 June 2027."}
],
"evidence_verification": {
"evidence_only": {"noul": 0.9902, "agrees_with_full": true},
"evidence_removed": {"noul": 0.2140, "agrees_with_full": false}
}
}
},
"usage": {"branches": 1, "input_tokens": 4096, "document_prefill_tokens": 0, "evidence_calls": 2}
}
agrees_with_full compares decisions, not probabilities: the yes/no side of 0.5 for each unit, or
which option is on top for a choice. The probabilities beside it are the re-read ones and are shown so
you can see how far they moved.
What the live acceptance run establishes
20 of the 57 live checks on this exact binding are evidence checks, and all
of them passed on the hardware in the release record.
- The spans are real source ranges. 12 checks re-sliced every returned span out of
the document it came from and required an exact match. All returned
found, with
4–5 spans per question.
- Asking for evidence cannot change an answer. At each of the three levels the whole answer object,
with every evidence field stripped out, was required to be identical to the answer returned without
it — same probabilities, same ordering score, same temperature. The real-document run below reproduces
this to the last bit. This is the one property here worth relying on: evidence is an addition to the
response and never an input to it. The serving code carries the same statement as a field,
answer_qualification_unchanged: true.
- Running out of budget is not an answer. With the budget set below what the level needs, every
question came back
budget_exhausted with its answer unchanged rather than degraded.
What that run is, in its own words: live mechanics on padded fixture documents; not a population sample and not an accuracy, calibration or evidence-faithfulness claim.
It is a mechanics test on 2 padded fixture documents, and it says nothing about whether
the spans it returned are the right spans.
What one real document showed
The evidence path has been run against a real document exactly once: 6 questions,
14 answer units, on a public privacy policy, on a B200, through the shipped
adapter, the shipped binding and the shipped default selector. It is the only evidence run of any kind
on the model in this repository. In the report's own words: DEMONSTRATION ONLY on one real out-of-support document. Not a population sample, not an accuracy, calibration or evidence-faithfulness claim, and no qualification artifact is written.
Read it off-support. That document is 4,440 prefix tokens against a
binding whose measured prefix support is 1,242–2,327 tokens. The report
records within_measured_support: false. Nothing below is a quality number, and none of it should be
mixed with the synthetic figures in the next section.
What held up.
- The spans are exact. All 29 returned spans are consistent slices of the source, and
the service validates every one against the document before returning it — a span that no longer
matches raises rather than returns.
- Evidence never touched the answer. Across
'none', 'support' and 'sufficiency' every
probability for all 14 units was bit-identical, not merely close. The same property
the acceptance run checks, on a real document.
- Every evidence-only re-ask agreed with the full-document decision, on all 14 units.
What did not.
- Span precision was poor. The selector returns its top four overlapping sentences per unit whether
or not they are relevant. The clearest case: for a question about whether the policy commits to a
complaint-response deadline, the returned evidence included the site's footer navigation block
(
Services / Medical Certificates / Prescriptions …). Reading all
29 spans by hand, roughly 8 were on point. That is a hand count on one document, with
no scoring artifact behind it, and it is the only precision figure that exists for the shipped
selector.
candidate_evidence gave no per-candidate discrimination at all. On both multi-candidate
questions — 10 candidates in total — every candidate received the identical span
set. The shipped selector reports no per-unit attribution, so the service hands each unit the same
union. The field's shape invites you to read it as "the evidence for this candidate"; with the
default selector it is not that, and a caller who trusts it will be misled.
- Agreement is near-vacuous on a negative answer. One question asked for a commitment the policy
does not make. The model answered correctly — P(yes) = 0.00288 — but the selector
never surfaced the relevant part of the document, and the sufficiency check still reported
agrees_with_full: true on evidence that was about something else. Agreeing that something is
absent is not the same as showing it is absent, and nothing in the response distinguishes the two.
Treat agrees_with_full on a negative answer as carrying almost no information.
- Decisions were stable; magnitudes were not. Every unit agreed, but the probabilities moved when
re-read from the evidence alone — the largest by 0.062, an entity candidate going
from 0.914 to 0.975.
agrees_with_full compares decisions, so it
reports agreement across moves of that size without flagging them.
What has been measured on synthetic documents
One registered run measures evidence quality. It is not a run on the model in this repository.
The metric is evidence-only agreement: the fraction of questions whose answer, read from that
question's evidence package alone, equals the answer read from the whole document.
|
|
| Evidence-only agreement |
0.960 (288/300), 95% interval 0.937–0.980 |
| Control: same documents with the spans deleted |
0.153 (46/300) |
| Span precision / recall |
0.950 / 0.982 over 964 answer units |
| Correctly finding no support where there is none |
0.998 over 403 units |
| Population |
60 documents, 300 questions, text only |
| Answer type |
Agreement |
Questions |
| Yes / no |
0.967 |
58/60 |
| Single choice |
0.967 |
58/60 |
| Ordered choice |
1.000 |
60/60 |
| Multi-label |
0.967 |
58/60 |
| Entity |
0.900 |
54/60 |
The control line is the one that carries information: deleting the selected spans drops agreement from
0.960 to 0.153, so the spans are not incidental. Entity is the weak cell at
0.900, missing the gate this project set for it — the same answer type that
collapses on real documents under Limitations.
Three qualifiers, and they are large.
- The documents are synthetic, generated by this project's own generators, exactly like every other
in-domain panel on this page.
- The model measured was not this one. That run used an earlier adapter from a previous training
round (
ea869eace19a…), not the adapter shipped here (2addaf841ecc…).
The packager compares those two hashes and refuses to build if they are ever equal, so this
paragraph cannot quietly become false.
- The selector measured is not the one shipped. That run used a fitted relevance selector with its
own trained weights. Those weights are not in this repository and are not released. What you get
here is the lexical-overlap default described above, which is a weaker selector.
What is not established
- No scored evidence measurement exists for the adapter in this repository. Every figure in the
agreement table belongs to an earlier model and a selector that is not shipped. The only run on this
adapter is the single real document above, which has no reference labels and scores nothing.
- No registered measurement exists for the shipped default selector. The only run that used the
shipped
lexical top4/unit; question/source only policy is a development diagnostic over 30 development
sources and 150 questions. It reports no aggregate, and its own record states:
"Development oracle diagnostic; exact gold anchors may omit provenance context. Agreement is not faithfulness. Lexical retrieval is a baseline, not a trained relevance head."
Nothing is rolled up from it here, and no number from it appears above.
- Faithfulness is not established, and is not claimed. Agreement means the answer survives on the
evidence alone; it does not mean the passage caused the answer, and it does not mean no other passage
would have done. Every response says so in a field.
- No scored evidence figure on real documents exists at all. One document, read by hand, off the
binding's measured support, is what there is. Given what the real-document panel does to entity and
multi-label accuracy under Limitations, assume evidence quality degrades there too, and do not
assume the size of the drop is known.
- Agreement is not accuracy. A wrong answer that the evidence reproduces faithfully agrees with
itself. And on a negative answer, agreement barely discriminates at all — see above.
candidate_evidence is not per-candidate with the shipped selector. It is the same union of spans
repeated once per candidate.
'removal' has never been timed or costed, only counted.
- Span precision is unmeasured on real documents and, for the shipped selector, unmeasured anywhere:
the only figure is a hand count on one document.
- The per-answer-type gates were not all met even on the synthetic panel that was run.
If a decision matters, treat these spans as a place to start reading, verify them yourself, and do
not present them to anyone as the model's reasons. They are not.
What it scores
On documents, held out, and synthetic
1,500 generated documents, none of them seen by the temperature fit. The calibration
this repository ships is applied.
| Answer type |
Units scored |
Accuracy |
ECE |
Errors stated ≥0.99 |
Share of mass ≥0.99 |
| Yes / no |
1,500 questions |
98.600% |
0.0079 |
14.3% |
73.6% |
| Single choice |
1,500 questions |
99.867% |
0.0004 |
0.0% |
99.1% |
| Ordered choice |
1,500 questions |
98.600% |
0.0012 |
4.8% |
85.2% |
| Multi-label (per candidate) |
7,634 candidates |
99.528% |
0.0026 |
13.9% |
95.8% |
| Entity (per candidate) |
12,000 candidates |
99.217% |
0.0073 |
3.2% |
67.9% |
Ordered choice is the best-calibrated cell here. Yes/no is the weakest and stays weak: it still
states 14.3% of its errors
above 0.99, so a yes/no answer above 0.99 is worth less than an entity candidate above 0.99 and the two
should not be treated as interchangeable.
These documents are synthetic. They were produced by this project's generators. That is the single
most important qualifier on this table and the next section is about what happens without it.
On public benchmarks, with no document at all
These are out-of-domain by construction: Solomon reads documents, and every item here is a bare
multiple-choice question. They are included because they are the only comparison a reader can check
against a model they already know — the unmodified base model, item for item, on the same prompts.
- Solomon — the shipped adapter and heads, structured letter readout over the answer options (
head_key single/choiceS), production cached path, fp32.
- Base model, standard prompt — Qwen3.8-27B with no adapter, a standard zero-shot multiple-choice chat prompt, letter logits from the unmodified
lm_head.
- Base model, same readout — Qwen3.8-27B with no adapter, read through the same structured prompt shape as Solomon, letter logits from the unmodified
lm_head. This is the like-for-like control.
Accuracy, with a 95% bootstrap interval
| Panel |
Solomon |
Base model, standard prompt |
Base model, same readout |
| ARC-Challenge (n = 600) |
97.2% [95.8%–98.3%] |
97.0% [95.5%–98.3%] |
97.2% [95.7%–98.5%] |
| MMLU (n = 2,400) |
82.4% [80.9%–83.9%] |
82.5% [81.0%–84.0%] |
81.7% [80.3%–83.3%] |
| MMLU-Pro (n = 800) |
60.5% [57.1%–63.7%] |
62.3% [59.0%–65.5%] |
61.8% [58.4%–65.0%] |
| Fresh math (n = 190) |
55.8% [48.9%–63.2%] |
52.1% [45.3%–58.9%] |
43.7% [36.3%–50.5%] |
| Fresh math, multi-step only (n = 170) |
51.2% [43.5%–58.2%] |
46.5% [38.8%–54.1%] |
37.1% [30.0%–44.1%] |
Expected calibration error — lower is better
| Panel |
Solomon |
Base model, standard prompt |
Base model, same readout |
| ARC-Challenge (n = 600) |
0.022 |
0.007 |
0.028 |
| MMLU (n = 2,400) |
0.117 |
0.057 |
0.027 |
| MMLU-Pro (n = 800) |
0.190 |
0.063 |
0.059 |
| Fresh math (n = 190) |
0.063 |
0.063 |
0.050 |
| Fresh math, multi-step only (n = 170) |
0.076 |
0.070 |
0.056 |
Errors stated at 0.99 or above — count, and share of that arm's errors
| Panel |
Solomon |
Base model, standard prompt |
Base model, same readout |
| ARC-Challenge (n = 600) |
4 (23.5% of 17) |
1 (5.6% of 18) |
1 (5.9% of 17) |
| MMLU (n = 2,400) |
76 (18.0% of 423) |
30 (7.1% of 421) |
10 (2.3% of 439) |
| MMLU-Pro (n = 800) |
21 (6.6% of 316) |
9 (3.0% of 302) |
4 (1.3% of 306) |
| Fresh math (n = 190) |
0 (0.0% of 84) |
0 (0.0% of 91) |
0 (0.0% of 107) |
| Fresh math, multi-step only (n = 170) |
0 (0.0% of 83) |
0 (0.0% of 91) |
0 (0.0% of 107) |
AUROC — how well the stated probability separates right answers from wrong ones
| Panel |
Solomon |
Base model, standard prompt |
Base model, same readout |
| ARC-Challenge (n = 600) |
0.946 |
0.966 |
0.887 |
| MMLU (n = 2,400) |
0.854 |
0.872 |
0.853 |
| MMLU-Pro (n = 800) |
0.847 |
0.850 |
0.844 |
| Fresh math (n = 190) |
0.845 |
0.764 |
0.782 |
| Fresh math, multi-step only (n = 170) |
0.818 |
0.704 |
0.716 |
On ARC-Challenge and MMLU, accuracy is essentially the base model's: every interval above overlaps. On the two fresh-math panels — items generated for this comparison and present in no public corpus — Solomon is clearly ahead of the base model read the same way (55.8% against 43.7%, and 51.2% against 37.1%), and ahead of it under a standard prompt as well.
The calibration does not survive the trip. On MMLU, Solomon's expected calibration error is 0.117 against 0.027 for the same base model read the same way, and 0.057 for the base model under a standard prompt. Solomon states 76 of its 423 errors at 0.99 or above; the base model on the same readout states 10 of 439. MMLU-Pro is worse again — 0.190 against 0.059, on lower accuracy (60.5% against 61.8%).
Read plainly: this adapter makes the model more confident, and out of its own domain that confidence is not earned. Solomon is MORE overconfident on general-knowledge questions than the model it is built on. The per-answer-type temperatures were fitted on structured document question-answering and they do not transfer; nothing here entitles a reader to carry the in-domain calibration results over to a general-knowledge population.
The ranking signal degrades more gently than the magnitude does: AUROC stays close to the base model's on every panel, and beats it on both fresh-math panels. An out-of-domain score is worth more as an ordering than as a probability — and in domain the same caution applies to the multi-candidate roll-up, below.
How to read these numbers
- NO TEMPERATURE IS APPLIED in these numbers: the scoring harness takes a plain softmax over the raw letter logits. The shipped serving path would not serve these items calibrated either -- the readout used here is
head_key single/choiceS, which is not in the calibration artifact's map, and the loader REFUSES an unmapped head_key rather than assuming 1.0.
- These panels have no document. Solomon is a document question-answering model and every one of these items is a bare multiple-choice question, so this is out-of-domain by construction.
- The fresh-math panels use 10-option items, an option count outside the range the readout was developed on.
- The fresh-math items were generated for this comparison and are in no public corpus. Only top probability and correctness were retained for some arms there, so no Brier score is computed on those panels.
- 95% CIs are 2,000-resample item bootstraps.
Limitations
The synthetic-to-real gap
Everything above about documents is measured on generated documents. One evaluation has ever been run
against real documents, on a panel of 40 real documents and
198 questions.
|
Whole-question agreement |
Yes/no |
Single |
Ordered |
Multi-label |
Entity |
| This model |
0.8131 |
0.9750 |
1.0000 |
0.9500 |
0.7000 |
0.4359 |
| A previous training round, same panel |
0.8232 |
0.9750 |
1.0000 |
0.9750 |
0.7000 |
0.4615 |
| Questions |
198 |
40 |
39 |
40 |
40 |
39 |
Set that against ≈99% on the synthetic panel. The gap is real, it is large, and it is concentrated in
entity and multi-label. Slot accuracy on the earlier round was 0.8529; the
earlier round's whole-question agreement carries a 95% interval of
0.7789–0.8629.
The shipped model was measured on this panel and did not improve on it. The difference against the
previous round is -1.0 points, 95% interval
[-3.0, +1.0], McNemar
p = 0.62 on 4
discordant questions. The two rounds are indistinguishable here. A round of training aimed specifically
at this failure did not move it.
The dominant error. The model answers "No" where the adjudicated reference is "not
established", for candidates the document never mentions. Of the
110 entity slots whose reference is "not
established", this model answers "No" on 57; the earlier
round answered "No" on 56 of the same
110. The earlier round's analysis recorded it this way —
B is the "No" letter and C the "not established" letter:
answers B (No) where adjudicated gold is C (not established) for entities/labels absent from the document: 60 of 94 wrong entity/multilabel slots; 25 of 33 wrong entity/multilabel questions are only this confusion
These are not marginal errors. The project's analysis of this panel records that the model stays
confident on exactly these slots, and that a round of training aimed at the failure left both the count
and the confidence essentially unchanged. If your questions ask about things that may simply be absent from the
document, this is the behaviour that will hurt you, and no released version of this model handles it
well.
Analysis of why the synthetic population does not transfer, recorded before the shipped model was
trained: synthetic documents state their own reading conventions and real ones do not; the synthetic
entity questions are about permissions while the real ones are about exclusive roles; and the
conventions for "mentioned in another role" differ between the two label sets.
What the real-document panel is, and is not
- The reference labels are adjudicated agent labels — two blind passes plus an adjudicator — not
human ground truth. The artifact records
reviewer: "agent" and human_verified: false.
No human has verified this panel.
- The two blind passes did not agree with each other on the convention either. The artifact records:
the two blind labelling passes disagreed on the same convention (entity agreement 0.786, multilabel 0.833 question-level)
So part of the measured gap may be a labelling-convention disagreement rather than a model error:
24 of the 60 slots are in document-referential questions ("named in the letter as ...") where No is defensible; 36 are substantive questions where the model contradicts its own prompt rule "A missing fact is not a negative fact"
- The shipped model's scores on this panel were produced by the project's paired evaluation harness on
the frozen design, not through the serving API in this repository. Temperature is a monotone
rescaling and cannot change which answer is on top, so accuracy carries; calibration on real
documents has not been measured at all.
- 40 documents and 198 questions is a small panel. Treat the per-type numbers as
indicative, not precise.
The rest
- Out of domain, the probability magnitude is not trustworthy. See the benchmark section: on
general knowledge this model is more overconfident than the base model it is built on.
- The multi-candidate roll-up is an ordering, not a joint probability. Read per-candidate values if
you need a magnitude.
- Nothing here is a certified error rate. No certification was run for this release and no
threshold is enforced anywhere on the serving path.
- A confirmation run on the held-out synthetic panel did not happen — its compute went to training.
The synthetic claim rests on the held-out panel above plus repeated paired panels.
- No image modality claim. Text only.
- Yes/no above 0.99 is the least trustworthy high-confidence signal in the model.
Training-data provenance
No training document, panel, corpus or source text is distributed in this repository.
The adapter was trained on synthetic documents, and some of those documents were produced by editing
third-party natural texts. Those texts are listed here so that their provenance is on the record, and
so that a reviewer doing lawful-sourcing diligence can see what was used without having to ask.
The model card discloses the provenance of the third-party texts the training panel was edited from, and attributes them. This is provenance disclosure and attribution as good practice, and it supports documented-lawful-sourcing procurement review. It is NOT a concession that the trained weights are a derivative work or an adaptation of those texts; that question is open and nothing in this package answers it.
13 of the 42 reviewed sources were used in the training panel:
|
Source |
URL |
Licence, as recorded |
| Django's security policies |
https://docs.djangoproject.com/en/dev/internals/security/ |
BSD 3-Clause (Django project LICENSE, which covers the documentation in the django/django repository) |
| CC BY-SA |
Wikipedia:Arbitration/Policy (English Wikipedia arbitration policy) |
https://en.wikipedia.org/wiki/Wikipedia:Arbitration/Policy |
CC BY-SA 4.0 (Wikipedia text; attribution: English Wikipedia contributors, 'Wikipedia:Arbitration/Policy') |
| Common Paper Mutual Non-Disclosure Agreement, Version 1.0 – Standard Terms |
https://commonpaper.com/standards/mutual-nda/1.0/ |
CC BY 4.0 (stated in the agreement footer and in the CommonPaper/Mutual-NDA repository README: 'free to use and modify under CC BY 4.0') |
| 36 CFR Part 2 (National Park Service) - Resource Protection, Public Use and Recreation: sections 2.10, 2.13, 2.14, 2.15, 2.16, 2.21 and 2.22 |
https://www.ecfr.gov/current/title-36/chapter-I/part-2 |
US Government work (public domain): Code of Federal Regulations text, not subject to copyright (17 U.S.C. 105) |
| NSF Proposal & Award Policies & Procedures Guide (PAPPG, NSF 24-1), Chapter IV: Non-Award Decisions and Transactions |
https://www.nsf.gov/policies/pappg/24-1/ch-4-non-award-decisions-transactions |
US Government work (public domain): U.S. National Science Foundation policy guide, not subject to copyright (17 U.S.C. 105) |
| GOV.UK: Make a court claim for money |
https://www.gov.uk/api/content/make-court-claim-for-money |
Open Government Licence v3.0 |
| 42 CFR 68: NIH Loan Repayment Programs (2025 edition) |
https://www.govinfo.gov/content/pkg/CFR-2025-title42-vol1/xml/CFR-2025-title42-vol1-part68.xml |
US Government work (public domain) |
| CC BY-SA |
WordPress.com (Automattic) Terms of Service, last updated April 10, 2026 |
https://github.com/Automattic/legalmattic/blob/master/Terms%20of%20Service/WordPress.com/EN-Terms-of-Service.md |
CC BY-SA 4.0 (Automattic/legalmattic LICENSE.txt and README; the Terms themselves state they are available under a Creative Commons Sharealike license) |
| CC BY-SA |
GitLab Handbook: Global Travel and Expense Policy (sections 1 to 3) |
https://handbook.gitlab.com/handbook/finance/expenses/ |
CC BY-SA 4.0 (GitLab handbook content; licence badge in the footer of handbook.gitlab.com pages) |
| Sourcegraph Handbook: Spending company money |
https://github.com/sourcegraph/handbook/blob/main/content/benefits-pay-perks/benefits-perks/spending-company-money.md |
Apache License 2.0 (LICENSE of the public sourcegraph/handbook repository) |
| Project Jupyter Governance: Executive Council |
https://raw.githubusercontent.com/jupyter/governance/main/docs/executive_council.md |
CC0 1.0 Universal (jupyter/governance repository LICENSE.md) |
| 29 CFR 1904.30–1904.34: establishment records and annual summaries (2025 edition) |
https://www.govinfo.gov/content/pkg/CFR-2025-title29-vol5/xml/CFR-2025-title29-vol5-part1904.xml |
US Government work (public domain) |
| 31 CFR 1.2, 1.4 and 1.6: Treasury FOIA requests and appeals (2025 edition) |
https://www.govinfo.gov/content/pkg/CFR-2025-title31-vol1/xml/CFR-2025-title31-vol1-part1.xml |
US Government work (public domain) |
3 of these are recorded as CC BY-SA 4.0 — Wikipedia:Arbitration/Policy (English Wikipedia arbitration policy), WordPress.com (Automattic) Terms of Service, last updated April 10, 2026, GitLab Handbook: Global Travel and Expense Policy (sections 1 to 3). Share-alike is the one term
attribution cannot cure. Whether a share-alike obligation can propagate through training into model
weights is legally unsettled; there is no authority settling it in either direction, and the
project's own licence review explicitly declines to infer one. The owner of this release
accepted that residual risk on 2026-09-21 rather than resolving
it, and kept this package under Apache-2.0. A reader should treat the question as open, not answered.
On the strength of this evidence. The licences above are as recorded by the person who collected
each source, from the source's own stated terms at the time of collection. The review records
evidence_level: "authoring metadata assertion, not archived governing licence text" and
upstream_terms_independently_verified: false for every row. No governing licence text was archived
alongside most of these sources, and this listing should not be read as a licence audit.
Listing these sources is provenance disclosure and attribution as good practice. It is not a
statement that the trained weights are a derivative work, an adaptation, or a copy of any of these
texts.
Third-party dependency licences
requirements.lock names the packages the serving layer needs. None of them is redistributed in
this repository — you install them yourself from their own publishers — so Apache-2.0 section 4(a)
imposes no bundled-notice obligation here and no dependency licence text is packaged. This summary
exists because a reviewer will ask for one.
Each licence in the table is the one declared in that distribution's OWN package metadata -- the METADATA file of an installed wheel -- read from a copy on the maintainer's machine. Where no copy existed, the row says NOT VERIFIED instead of guessing.
| Package |
Pinned as |
Licence, as declared by the distribution itself |
torch |
2.13.0 |
BSD-3-Clause — read from version 2.8.0 |
torchvision |
0.28.0 |
NOT VERIFIED |
transformers |
5.17.0 |
Apache 2.0 License |
flash-linear-attention |
0.5.2 |
NOT VERIFIED |
safetensors |
unpinned in the qualified image |
Apache Software License — read from version 0.8.0 |
accelerate |
unpinned in the qualified image |
Apache (Apache Software License) — read from version 1.15.0 |
numpy |
unpinned in the qualified image |
BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0 — read from version 2.5.3 |
scipy |
unpinned in the qualified image |
BSD License (classifier); the METADATA License field carries the Enthought / SciPy Developers copyright line rather than an SPDX identifier — read from version 1.18.1 |
pillow |
unpinned in the qualified image |
MIT-CMU — read from version 12.3.0 |
Two rows say NOT VERIFIED and mean it. A licence read from one installed version is evidence about that version only. Where the table names a version different from the pin, that is the version whose metadata was actually read, and the pinned version's own metadata could differ. Treat this table as a starting point for
your own review, not as a legal opinion, and re-check the distributions you actually install.
Full evidence paths for each row are in the release descriptor
(release/solomon-release.json → dependency_licences), which is not distributed; the same
information is repeated in the comments of requirements.lock.
Licence and attribution
Copyright 2026 Doccy Pty Ltd.
This repository is licensed Apache-2.0 — the adapter and head weights, the calibration artifact,
the serving code and the documentation alike. See LICENSE and NOTICE.
It is a derivative work of Qwen/Qwen3.8-27B, Copyright 2026 Alibaba Cloud, licensed under Apache-2.0.
The upstream licence text is reproduced verbatim at licenses/Qwen-Apache-2.0.txt (sha256
bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a). NOTICE carries the attribution required by section 4(c) and
MODIFICATIONS.md the statement of changes required by section 4(b).
No NOTICE file exists in the upstream repository at the pinned revision (HTTP 404, checked
2026-09-18), so section 4(d) carries nothing forward and no upstream attribution text has been
invented.
"Qwen" and "Alibaba Cloud" are used nominatively to identify the base model. Apache-2.0 section 6
grants no trademark rights and none are claimed. No endorsement or affiliation is implied.
Verify what you downloaded
shasum -a 256 adapter/adapter.safetensors # 2addaf841ecc58829ad73081603b8d3e52743c53c6d558a17a1dd30e9bd2dbca
shasum -a 256 adapter/heads.npz # 126a9b5487dca937a768a4f228f2d2e7d513900d8ad0e99c4137fbaa42a1aa6f
shasum -a 256 serving/readout-temperature-v2.json
# -> d6925a0168b2577d0e4d517cd25dd5ceb0b7cc8eee91b70aefd3a3aa9dcb64ce
MANIFEST.json carries the size and sha256 of every file in this repository.
The calibration artifact has two legitimate and different hashes, and confusing them makes a sound
provenance chain look tampered with. d6925a01… is the file hash, what shasum
returns. 6506e239… is the artifact's own internal sha256 field, computed over its
contents with that field removed — a self-referential field cannot hash the file containing it. The
loader verifies the payload hash; use the file hash to check the file you were given. Both are recorded
in the serving binding's provenance, under those names.
Release record
Machine-readable identity for citation and pinning. The model identity, the calibration and the runtime
binding move together; pin by revision.
|
|
| Repository |
DoccyHealth/Solomon |
| Release |
1.1.0, 2026-09-21 |
| Serving contract |
solomon-v1 |
| Adapter sha256 |
2addaf841ecc58829ad73081603b8d3e52743c53c6d558a17a1dd30e9bd2dbca |
| Heads sha256 |
126a9b5487dca937a768a4f228f2d2e7d513900d8ad0e99c4137fbaa42a1aa6f |
| Base model |
Qwen/Qwen3.8-27B at 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 |
| Runtime binding sha256 (payload) |
49904e27814e0ccfc483b0503237d84f8ca60a79713f393b23b0dbe57d87a6bf |
| Calibration sha256 (file / payload) |
d6925a0168b2577d0e4d517cd25dd5ceb0b7cc8eee91b70aefd3a3aa9dcb64ce / 6506e2394bda5f8e7c7b9742f70a5b8782efad53d5451d5d836ba3463cece24b |
| Runtime fingerprint |
c3d933e068e19879b45fb6dc6ce4af83a4d22f615c263b209e15f9c4fb2ae952 |
| Readout |
four_collapsed |
| Ruled temperatures |
boolean 1.9476 · single 1.0 · ordered 1.5063 · multilabel 1.6895 · entity 2.1834 |