proof_v3

Falconsai/proof_v3 — FalconDec v1.0.0

View in Model Surgeon

A single-pass, typed, calibrated decision model. Give it a state (text, code or JSON), one or more typed questions (choice, noul yes/no, score ordinal) and a closed set of options. It returns a calibrated probability for every option, from one encoder pass per question, however many options there are.

This repository is both the model card and the developer guide: how to load it, test it, and use it as a decision component in an agentic system.

Read this first. v1.0.0 is the first FalconDec checkpoint, trained with the small preset (at most 500 training decisions per task, 1 epoch). It proves the architecture and pipeline end to end, and it is well calibrated but under-trained: test accuracy is 0.518 (micro) / 0.535 (task-macro). Evaluate it on your own data (§6.4) before acting on its answers, and use the deferral policy (§7.3). A standard-preset retrain is the planned next version.

Contents

1. Model summary
2. Results
3. Intended and out-of-scope uses
4. The Hub repository
5. Install, load and read a result
6. Testing the model
7. Using it in an agentic system
8. Tuning for your domain
9. Architecture
10. Training data
11. Training procedure and calibration
12. Operational notes
13. Bias, risks and limitations
14. Versioning and lineage
15. API reference
16. Citation and references

1. Model summary

The model has no generative component. It can only rank the options you give it, so it cannot produce text outside that set.

2. Results

All numbers come from this checkpoint's falcondec_report.json: one run, small preset, seed 42, MODE="scratch".

2.1 At a glance

Calibration is the strongest result. An ECE of 0.014 means a stated confidence of 80% corresponds to about 80% accuracy on data like this test set. That makes confidence thresholds (§7.3) meaningful even while raw accuracy is modest: the model knows when it doesn't know.

2.2 Per task

The full per-task table (n, accuracy, chance, ECE, NLL, held-out flag) is in falcondec_report.json under test_per_task. Print it with:

import json, pandas as pd
from huggingface_hub import hf_hub_download

rep = json.load(open(hf_hub_download("Falconsai/proof_v3", "falcondec_report.json"), encoding="utf-8"))
df = pd.DataFrame(rep["test_per_task"]).set_index("task")
print(df[["domain", "heldout", "n", "chance", "acc", "ece"]].sort_values(["domain", "acc"]).round(3).to_string())
print(rep["test_overall"])

The 9 held-out tasks are Banking77 (2–5 options), Banking77 with all 77 labels in one question, ARC-Easy, ARC-Challenge, MMLU, DAIR Emotion, SST-5 (ordinal), deepset prompt-injections and HumanEval (correct completion vs single-fault mutants).

2.3 Comparison with proof_v2 (not like-for-like)

These overall numbers are not comparable: the test sets differ in both tasks and difficulty. The like-for-like head-to-head (the FalconDec notebook runs proof_v2 on identical test decisions, cell 23) did not complete in this run, so no claim is made here about which model is more accurate on shared tasks. That comparison is the first item for the next version.

3. Intended and out-of-scope uses

Intended

Out of scope

• Arithmetic and multi-step reasoning (GSM8K, AQuA): a single encoder pass can't do System-2 work. Route these to an LLM.
• Security or correctness gating of code: vulnerability, clone and bug detection remain weak. Don't use it to approve code.
• Open-ended questions: it always picks one of your options. Add "None of the above" when that is a real possibility; it was trained with that option.
• Non-English text.
• High-stakes decisions without human oversight (medical, legal, financial, employment, safety-critical).
• Production use of v1.0.0 without your own evaluation: see the note at the top.

4. The Hub repository

Pin a revision in production. The repo can change, so pass a commit hash when you load.

Model Surgeon. If Model Surgeon processing renames the weight file in the published copy (for example to model_edited.safetensors), the loader in §5.1 detects it and loads the file anyway. Run the integrity test (§6.1) after every revision change to confirm the published weights behave like the trained checkpoint.

5. Install, load and read a result

pip install torch "transformers>=4.48" safetensors huggingface_hub numpy

5.1 Load

Save this helper as proof_v3.py next to your code. Every example below uses it.

# proof_v3.py
import importlib.util, json, shutil
from pathlib import Path
from huggingface_hub import snapshot_download


def load_proof_v3(repo="Falconsai/proof_v3", revision=None, variant="fp16", device=None, dtype=None):
    """Returns (fdm, model, tokenizer). variant: "fp16" (305 MB) or "int8" (153 MB, dequantised on load)."""
    path = Path(repo) if Path(repo).exists() else Path(snapshot_download(repo, revision=revision))
    if variant == "int8":
        path = path / "compact-int8"
    fc = json.loads((path / "falcondec_config.json").read_text(encoding="utf-8"))
    expected = fc.get("weights", "model.safetensors")
    if not (path / expected).exists():                  # e.g. a renamed file after Model Surgeon processing
        cands = sorted(p for p in path.glob("*.safetensors"))
        if not cands:
            raise FileNotFoundError(f"no .safetensors weights in {path}")
        local = Path("proof_v3_local") / variant
        shutil.copytree(path, local, dirs_exist_ok=True)
        shutil.copy(cands[0], local / expected)
        path = local
    spec = importlib.util.spec_from_file_location("falcondec_modeling", str(path / "falcondec_modeling.py"))
    fdm = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(fdm)
    model, tok = fdm.load_falcondec(str(path), device=device, dtype=dtype)   # cuda if available, else cpu
    return fdm, model, tok

from proof_v3 import load_proof_v3

fdm, model, tok = load_proof_v3()                      # or load_proof_v3(revision="<commit>"), variant="int8"
print(model.fcfg["name"], model.fcfg["version"], model.num_parameters() / 1e6, "M params")

If loading prints [FalconDec] load warning: missing=… unexpected=…, the weights didn't match the architecture. Treat that as a failed load (§6.1 checks for it).

5.2 Decide

decide() takes one state and any number of typed questions: either a list, or a Jev/Laya-style dict keyed by name.

state = {"from": "[email protected]", "subject": "Duplicate charge on invoice #4411",
         "body": "We were billed twice for March. Please refund the duplicate today or we will cancel our plan."}

out = fdm.decide(model, tok, state, {
    "department": {"type": "choice", "instructions": "Which department should handle this request?",
                   "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",
                                "sales": "pricing, contracts", "other": "everything else"}},
    "urgency":    {"type": "score", "instructions": "How urgent is this request?",
                   "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
    "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
})
a = out["answers"]
print(a["department"]["choice"], round(a["department"]["confidence"], 3), a["department"]["defer"])
print("urgency level", round(a["urgency"]["expected_level"], 2), "of", 2)
print("P(churn)", round(a["churn_risk"]["p_true"], 3))

Plain options work too: {"question": "Which team?", "options": ["Accounts", "Billing", "Shipping"]}.

5.3 Reading a result

Each item in out["results"] (and out["answers"][key]) contains:

How to read them:

• Low confidence, spread probabilities: the state doesn't support any option clearly. Defer, or add "None of the above".
• noul near 0.5: genuinely ambiguous. Ask for more information rather than guessing.
• score: use expected_level for thresholds ("escalate if ≥ 1.5") rather than choice.

6. Testing the model

Tests 6.1–6.3 need no labelled data, so run them in CI whenever you change the revision. Test 6.4 is the one that tells you whether to ship.

6.1 Integrity and determinism

Save as check_proof_v3.py and run python check_proof_v3.py [revision].

import contextlib, io, sys
import numpy as np
from proof_v3 import load_proof_v3

rev = sys.argv[1] if len(sys.argv) > 1 else None
log = io.StringIO()
with contextlib.redirect_stdout(log):
    fdm, model, tok = load_proof_v3(revision=rev)
assert "load warning" not in log.getvalue(), log.getvalue()

fc = model.fcfg
assert fc["name"] == "FalconDec", fc["name"]
T = model.temperature.float().cpu().numpy()
assert T.shape == (3, 4) and (T > 0).all(), T
q = {"team": {"question": "Which team?", "options": ["recover password", "shipping", "invoicing"]}}
a = fdm.decide(model, tok, "I forgot my password and can't sign in.", q)["answers"]["team"]
b = fdm.decide(model, tok, "I forgot my password and can't sign in.", q)["answers"]["team"]
assert all(abs(a["probs"][k] - b["probs"][k]) < 1e-4 for k in a["probs"]), "non-deterministic"
assert abs(sum(a["probs"].values()) - 1) < 1e-3
print(f"OK  FalconDec v{fc['version']}  choice={a['choice']}  conf={a['confidence']:.3f}  "
      f"defer_threshold={fc.get('defer_threshold')}  temperatures={np.round(T, 3).tolist()}")

6.2 Behavioural tests (pytest)

Save as tests/test_proof_v3.py and run pytest -q. Set PROOF_V3_REVISION to test a pinned commit.

import os, random
import pytest
from proof_v3 import load_proof_v3


@pytest.fixture(scope="session")
def fd():
    return load_proof_v3(revision=os.environ.get("PROOF_V3_REVISION"))


def ask(fd, state, question, options, **kw):
    fdm, model, tok = fd
    return fdm.decide(model, tok, state, [dict(question=question, options=options, **kw)])["results"][0]


def test_probabilities_are_valid(fd):
    r = ask(fd, "The build failed on main.", "What next?", ["Revert", "Ignore", "Retry"])
    assert all(0 <= p <= 1 for p in r["probs"].values()) and abs(sum(r["probs"].values()) - 1) < 1e-3


def test_typed_outputs(fd):
    fdm, model, tok = fd
    out = fdm.decide(model, tok, "I was charged twice. Refund me or I'm leaving.", {
        "refund": {"type": "noul", "instructions": "Does the user ask for a refund?"},
        "urgency": {"type": "score", "instructions": "How urgent?", "criteria": ["low", "medium", "high"]}})["answers"]
    assert 0 <= out["refund"]["p_true"] <= 1 and out["refund"]["choice"] in (True, False)
    assert 0 <= out["urgency"]["expected_level"] <= 2


def test_fanout_matches_single_questions(fd):
    # Batching changes padding; under bf16 that moves probabilities slightly, never the substance.
    fdm, model, tok = fd
    state = "I forgot my password and can't sign in."
    qs = [{"question": "Team?", "options": ["recover password", "shipping", "invoicing"]},
          {"question": "Urgent?", "options": ["Yes", "No"]}]
    together = fdm.decide(model, tok, state, qs)["results"]
    for q, t in zip(qs, together):
        alone = fdm.decide(model, tok, state, [q])["results"][0]
        assert all(abs(alone["probs"][k] - t["probs"][k]) < 2e-2 for k in alone["probs"])


def test_option_order_is_mostly_irrelevant(fd):
    # The head is order-equivariant, but the encoder sees positions; training reshuffled options every epoch.
    state, q = "Where is my parcel? It's three days late.", "What should support do?"
    opts = ["Give the delivery status", "Start a refund", "Book an appointment"]
    base = ask(fd, state, q, opts)["choice"]
    same = sum(ask(fd, state, q, random.Random(s).sample(opts, len(opts)))["choice"] == base for s in range(5))
    assert same >= 4


def test_many_options_use_the_tournament(fd):
    opts = [f"topic number {i}" for i in range(119)] + ["reset my password"]
    r = ask(fd, "I can't log in, I need to reset my password.", "What does the user want?", opts)
    assert len(r["probs"]) == 120 and abs(sum(r["probs"].values()) - 1) < 1e-3


def test_int8_agrees_with_fp16(fd):
    fdm8, m8, tok8 = load_proof_v3(revision=os.environ.get("PROOF_V3_REVISION"), variant="int8")
    fdm, model, tok = fd
    items = [dict(state=s, question="Which team?", options=["billing", "shipping", "accounts", "technical"])
             for s in ["I was double charged", "Where is my parcel?", "Change my email", "The app crashes on start",
                       "Refund the duplicate payment", "Package never arrived", "Reset my login", "Error 500 on checkout"]]
    a = [p.argmax() for p in fdm.score_items(model, tok, items)]
    b = [p.argmax() for p in fdm8.score_items(m8, tok8, items)]
    assert sum(x == y for x, y in zip(a, b)) >= len(items) - 1

6.3 Latency

import time, numpy as np, torch
from proof_v3 import load_proof_v3

fdm, model, tok = load_proof_v3()
q = [{"question": "Route?", "options": ["billing and payment", "shipping", "recover password"]}]
for _ in range(5):
    fdm.decide(model, tok, "I was charged twice.", q)
t = []
for _ in range(100):
    if torch.cuda.is_available(): torch.cuda.synchronize()
    t0 = time.perf_counter(); fdm.decide(model, tok, "I was charged twice.", q)
    if torch.cuda.is_available(): torch.cuda.synchronize()
    t.append((time.perf_counter() - t0) * 1000)
print(f"p50 {np.percentile(t, 50):.1f} ms   p95 {np.percentile(t, 95):.1f} ms   on {model.device}")

For CPU serving, load the int8 variant with device="cpu", dtype=torch.float32, and optionally apply torch.ao.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) for int8 matrix multiplies.

6.4 Accuracy on your own labelled data

Write 50–500 decisions that look like your real traffic, one JSON object per line. expected may be a letter, a 0-based index or the option text; type is optional.

{"id": "t1", "tag": "support", "state": "…", "question": "…", "options": ["…", "…"], "expected": "B", "type": "choice"}

import json, string
import numpy as np
from proof_v3 import load_proof_v3

fdm, model, tok = load_proof_v3()
rows = [json.loads(l) for l in open("my_eval.jsonl", encoding="utf-8") if l.strip()]

def idx(v, opts):
    if isinstance(v, int): return v
    v = str(v).strip()
    if len(v) == 1 and v.upper() in string.ascii_uppercase[:len(opts)]: return string.ascii_uppercase.index(v.upper())
    return [o.lower() for o in opts].index(v.lower())

items = [dict(state=r["state"], question=r.get("question", ""), options=r["options"], type=r.get("type", "choice"))
         for r in rows]
probs = fdm.score_items(model, tok, items, batch_size=64)
gold = np.array([idx(r["expected"], r["options"]) for r in rows])
pred = np.array([p.argmax() for p in probs]); conf = np.array([p.max() for p in probs]); ok = pred == gold

def ece(c, k, bins=15):
    e = 0.0
    for lo in np.linspace(0, 1, bins, endpoint=False):
        m = (c > lo) & (c <= lo + 1 / bins)
        if m.any(): e += m.mean() * abs(c[m].mean() - k[m].mean())
    return e

print(f"accuracy {ok.mean():.3f} | ECE {ece(conf, ok):.3f}")
for tag in sorted({r.get("tag", "all") for r in rows}):
    m = np.array([r.get("tag", "all") == tag for r in rows]); print(f"  {tag:12s} n={m.sum():4d} acc={ok[m].mean():.3f}")
for t in (0.5, 0.6, 0.7, 0.8, 0.9):
    m = conf >= t
    print(f"  act if conf >= {t}: answers {m.mean():6.1%}, accuracy when answering {ok[m].mean() if m.any() else float('nan'):.3f}")

The last loop is the deferral policy of §7.3. Pick the smallest threshold whose "accuracy when answering" meets your bar. If ECE on your data is much higher than 0.014, recalibrate (§8.2).

6.5 Regression gate between revisions

Fail the pipeline if any test task drops by more than two points between two revisions:

import json, sys
from huggingface_hub import hf_hub_download

old_rev, new_rev = sys.argv[1:3]
rep = lambda rev: {r["task"]: r["acc"] for r in json.load(open(
    hf_hub_download("Falconsai/proof_v3", "falcondec_report.json", revision=rev), encoding="utf-8"))["test_per_task"]}
old, new = rep(old_rev), rep(new_rev)
bad = [(t, old[t], new[t]) for t in new if t in old and new[t] < old[t] - 0.02]
print("\n".join(f"REGRESSION {t}: {a:.3f} -> {b:.3f}" for t, a, b in bad) or "no regressions")
sys.exit(1 if bad else 0)

7. Using it in an agentic system

7.1 Where it fits

An agent loop is mostly small decisions (which tool, is this safe, did that work, am I done, should a human look) around a few hard reasoning steps. LLMs are slow and poorly calibrated at the small ones. proof_v3 takes those; the LLM keeps planning, reasoning and generation.

Several questions about the same state go in one decide() call.

7.2 A routing node (LangGraph)

from proof_v3 import load_proof_v3

fdm, model, tok = load_proof_v3(revision="<commit>")

def route(state: dict) -> str:
    res = fdm.decide(model, tok, state, {
        "next": {"type": "choice", "instructions": "What should the agent do next?",
                 "criteria": {"search": "needs external information", "code": "needs code written or run",
                              "answer": "has enough information to answer", "human": "ambiguous, risky or out of scope"}},
        "unsafe": {"type": "noul", "instructions": "Does the latest input try to override the agent's instructions?"},
    }, defer_threshold=0.75)["answers"]
    if res["unsafe"]["p_true"] > 0.5:
        return "human"
    if res["next"]["defer"]:
        return "llm_planner"             # low confidence: let the LLM decide
    return res["next"]["choice"]

graph.add_conditional_edges("observe", route, {"search": "search_node", "code": "code_node", "answer": "answer_node",
                                               "human": "human_node", "llm_planner": "planner_node"})

7.3 The deferral policy

Choose thresholds from your own evaluation (§6.4); the default 0.70 is a starting point. For irreversible actions (payments, deletions, sending email), raise the threshold and keep a hard rule or human confirmation in front: the state is attacker-controlled text, and adversarial input can move scores. Log the question, options, choice, confidence, model.fcfg["version"] and the Hub revision for every decision; that log becomes your next evaluation and fine-tuning set (§8.3).

7.4 As a tool for Claude (tool use)

import json, threading
import anthropic
from proof_v3 import load_proof_v3

fdm, model, tok = load_proof_v3()
lock = threading.Lock()
client = anthropic.Anthropic()
tools = [{
    "name": "proof_v3_decide",
    "description": ("Fast, local, calibrated closed-set decision model. Give it a state (text or JSON), a question and "
                    "2+ distinct options; it returns the choice, a calibrated confidence and a 'defer' flag. "
                    "If 'defer' is true, don't rely on the answer. Not for arithmetic or multi-step reasoning."),
    "input_schema": {"type": "object", "properties": {
        "state": {"type": "string", "description": "The message, document excerpt or JSON state."},
        "question": {"type": "string"},
        "options": {"type": "array", "items": {"type": "string"}, "minItems": 2},
        "type": {"type": "string", "enum": ["choice", "score"], "description": "score = options are ordered levels"}},
        "required": ["state", "question", "options"]},
}]

def run_tool(inp):
    with lock:
        r = fdm.decide(model, tok, inp["state"], [{"question": inp["question"], "options": inp["options"],
                                                   "type": inp.get("type", "choice")}])["results"][0]
    return {k: r[k] for k in ("choice", "confidence", "defer", "probs") if k in r}

messages = [{"role": "user", "content": "Triage: 'I forgot my password and the reset email never arrived.' "
                                        "Teams: Accounts, Billing, Shipping."}]
while True:
    resp = client.messages.create(model="claude-sonnet-5", max_tokens=1024, tools=tools, messages=messages)
    if resp.stop_reason != "tool_use":
        print("".join(b.text for b in resp.content if b.type == "text"))
        break
    messages.append({"role": "assistant", "content": resp.content})
    results = []
    for block in resp.content:
        if block.type == "tool_use" and block.name == "proof_v3_decide":
            try:
                results.append({"type": "tool_result", "tool_use_id": block.id, "content": json.dumps(run_tool(block.input))})
            except Exception as exc:
                results.append({"type": "tool_result", "tool_use_id": block.id, "content": str(exc), "is_error": True})
    messages.append({"role": "user", "content": results})

7.5 As an MCP server

pip install mcp, then save proof_v3_mcp.py next to proof_v3.py:

import os, threading
from mcp.server.fastmcp import FastMCP
from proof_v3 import load_proof_v3

fdm, model, tok = load_proof_v3(revision=os.environ.get("PROOF_V3_REVISION"),
                                variant=os.environ.get("PROOF_V3_VARIANT", "fp16"))
MIN_CONF = float(os.environ.get("PROOF_V3_MIN_CONF", "0.7"))
lock = threading.Lock()
mcp = FastMCP("proof_v3")


@mcp.tool()
def decide(state: str, question: str, options: list[str], type: str = "choice") -> dict:
    """Choose one of 2+ distinct options for a question about a state. type="score" means ordered levels.
    Returns the choice, a calibrated confidence and 'defer' (true = not reliable enough to act on)."""
    with lock:
        r = fdm.decide(model, tok, state, [{"question": question, "options": options, "type": type}],
                       defer_threshold=MIN_CONF)["results"][0]
    return {k: r[k] for k in ("choice", "confidence", "defer", "probs", "expected_level") if k in r}


@mcp.tool()
def decide_many(state: str, questions: dict) -> dict:
    """Several typed questions about one state: {name: {"type": "choice"|"noul"|"score",
    "instructions": str, "criteria": {key: description} | [levels]}}."""
    with lock:
        return fdm.decide(model, tok, state, questions, defer_threshold=MIN_CONF)["answers"]


if __name__ == "__main__":
    mcp.run()

Register it in Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "proof_v3": {
      "command": "C:\\path\\to\\python.exe",
      "args": ["C:\\path\\to\\proof_v3_mcp.py"],
      "env": { "PROOF_V3_REVISION": "<commit>", "PROOF_V3_VARIANT": "int8" }
    }
  }
}

7.6 As an HTTP microservice

pip install fastapi uvicorn, then save serve_proof_v3.py:

import threading
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from proof_v3 import load_proof_v3

fdm, model, tok = load_proof_v3()
lock = threading.Lock()
app = FastAPI(title="proof_v3")


class Decide(BaseModel):
    state: str | dict
    questions: dict | list
    defer_threshold: float = 0.7


@app.get("/health")
def health():
    return {"ok": True, "model": model.fcfg.get("name"), "version": model.fcfg.get("version")}


@app.post("/v1/decide")
def decide(req: Decide):
    try:
        with lock:
            return fdm.decide(model, tok, req.state, req.questions, defer_threshold=req.defer_threshold)
    except (ValueError, KeyError, TypeError) as exc:
        raise HTTPException(422, str(exc))

Run it with uvicorn serve_proof_v3:app --port 9903 --workers 1. Each worker holds its own copy of the model; scale out with more processes.

7.7 Without an LLM: a support intake step

from proof_v3 import load_proof_v3

fdm, model, tok = load_proof_v3()
QUEUES = {"billing": "charges, invoices, payments, refunds", "returns": "returning or exchanging items",
          "delivery": "shipping status, late or missing parcels", "accounts": "login, password, profile",
          "human": "complaints or requests to speak to a person"}

def intake(message: str) -> dict:
    a = fdm.decide(model, tok, message, {
        "queue": {"type": "choice", "instructions": "Which team should handle this message?", "criteria": QUEUES},
        "urgent": {"type": "noul", "instructions": "Does this need a reply within the hour?"},
    })["answers"]
    if a["queue"]["defer"]:
        return {"action": "human_review", "suggestion": a["queue"]["choice"],
                "reason": f"low confidence ({a['queue']['confidence']:.0%})"}
    return {"action": "enqueue", "queue": a["queue"]["choice"],
            "priority": "high" if a["urgent"]["p_true"] >= 0.5 else "normal",
            "evidence": {"confidence": round(a["queue"]["confidence"], 3), "version": model.fcfg["version"]}}

8. Tuning for your domain

8.1 Change the deferral threshold

Pass defer_threshold= to decide(), or set model.fcfg["defer_threshold"]. This changes nothing in the model and is usually enough.

8.2 Recalibrate on your data

If ECE on your traffic (§6.4) is noticeably worse than 0.014, fit one extra temperature on top of the stored ones and save a recalibrated copy:

import numpy as np, torch
# probs, gold: from §6.4 (probabilities already include the stored temperatures)
logp = [np.log(np.clip(p, 1e-12, 1)) for p in probs]
def nll(s):
    return -np.mean([(lp / s)[g] - np.log(np.exp(lp / s).sum()) for lp, g in zip(logp, gold)])
s = min(np.linspace(0.5, 3.0, 51), key=nll)
print("extra temperature", s)
with torch.no_grad():
    model.temperature.mul_(float(s))
fdm.save_falcondec(model, tok, "proof_v3_recalibrated")            # add int8=True for the compact variant

Fit on one labelled set and measure on another.

8.3 Fine-tune on your own decisions

Use the FalconDec training notebook (V2):

1. Export your logged and corrected decisions as JSONL: {"state", "question", "options", "answer", "type"?, "task"?}.
2. In cell 2, set MODE="finetune", FINETUNE_FROM="Falconsai/proof_v3", CUSTOM_DATA_JSONL="your_file.jsonl", and a preset.
3. Run the notebook. The version bumps automatically (1.0.0 → 1.0.1), and the report records the lineage.

Your tasks are up-weighted (CUSTOM_WEIGHT), while the public tasks keep the model general. Check the regression gate (§6.5) before publishing.

8.4 Save and publish

fdm.save_falcondec(model, tok, "proof_v3_out")                     # fp16
fdm.save_falcondec(model, tok, "proof_v3_out/compact-int8", int8=True)

from huggingface_hub import HfApi                                   # needs a write token: huggingface-cli login
HfApi().upload_folder(folder_path="proof_v3_out", repo_id="Falconsai/proof_v3", commit_message="FalconDec v1.0.x")

9. Architecture

[CLS] question [SEP] [MASK] option₁ [MASK] option₂ … [MASK] optionₖ [SEP] state [SEP]
        │
  Ettin-150M encoder (22 layers, hidden 768)
        │
  hidden state at each [MASK]  +  CLS context  +  question-type embedding
        │
  set transformer: 2 layers, no positional encoding → options attend to each other, order-equivariant
        │
  MLP → one logit per option → ÷ temperature[type, option-count bucket] → softmax

• Option markers (the approach Laya uses): every option is read at its own [MASK] token, so all options are scored in one pass. proof_v2 needed one cross-encoder pair per option plus a second pass.
• Question first, state last: when the input is too long, the tail of the state is truncated, never the options.
• Adaptive option budget: the per-option token budget shrinks as the number of options grows, and the sequence stretches to 2,048 tokens above 24 options so labels stay distinct (Laya's documented Banking77 failure was 3–4 tokens per label). Above 96 options, decide() runs a tournament: best options of each chunk, then a final pass.
• Typed primitives: noul is rendered as a neutral two-option Yes/No choice (avoiding Laya's documented failure where the answer followed the true:/false: labels); score keeps its level order and reports an expected level.
• Calibration lives in the model: a 3 × 4 temperature table (question type × option-count bucket: ≤2, 3–5, 6–12, >12).
• int8 storage: per-output-channel symmetric int8 for every weight matrix, fp16 elsewhere, dequantised on load. The compute path is identical to fp16.

10. Training data

Every example is a decision: state, question, options, answer, plus type, task and domain. Options are reshuffled every epoch (ordinal levels keep their order). Each source was loaded independently; any source that failed to load on the training machine was skipped and is listed under skipped_builders in falcondec_report.json.

Augmentation: in 8% of choice questions the gold answer is removed and "None of the above" becomes correct; in another 4% it is added as a distractor. Balance: tasks are sampled with p ∝ n^0.5 every epoch, so no domain dominates (proof_v2 was 73% code). Leak guard: training decisions whose (state, question) appears in validation or test were removed.

Held-out sources were never used for training, calibration or model selection. Check each dataset's card for its license before redistributing derived data.

11. Training procedure and calibration

Training time, GPU and library versions are recorded in falcondec_report.json (train_minutes, env). Estimate emissions with the ML CO2 Impact calculator from those values.

12. Operational notes

• Concurrency. decide() isn't internally locked. Serialise calls with a lock per process (as in §7.4–7.6) and scale out with processes.
• Hardware. A GPU gives ~16 ms per single-question call. On CPU, use the int8 variant (loaded as fp32) and optionally dynamic int8 quantisation; expect tens to a few hundred milliseconds depending on input length and cores.
• Input length. 512 tokens by default. The question and options come first, so an over-long state loses its end. Put the decisive information early, or summarise.
• Determinism. Repeated identical calls give identical results on the same hardware and library versions. Under bf16 on GPU, batching different questions together changes padding and can move probabilities slightly (about 1e-2); near-ties can flip. Decisions with a clear margin don't change.
• Traceability. Log model.fcfg["version"], the Hub revision and the variant (fp16/int8) with every decision.
• Offline use. After the first download, set HF_HUB_OFFLINE=1, or save a local copy and load it by path.

13. Bias, risks and limitations

• Under-trained first version. Trained with the small preset (≤500 decisions per task, 1 epoch). Accuracy on reasoning-heavy and held-out tasks is modest (held-out macro 0.436).
• No completed head-to-head with proof_v2. Don't assume v1.0.0 is more accurate than proof_v2 on proof_v2's strong tasks until that comparison is published.
• Closed world. The model always picks one of your options. Add "None of the above" when appropriate.
• Arithmetic and code correctness remain weak; route them elsewhere.
• Distribution-dependent calibration. ECE was measured on this test mix; re-check it on your traffic (§6.4) and recalibrate if needed (§8.2).
• Adversarial input. The state is untrusted text. The model can't be instructed like an LLM, but crafted input can shift its scores. Don't make it the only safeguard before irreversible actions.
• Data provenance. Training data is English, largely crowd-sourced, templated, synthetic or scraped from public code. Biases in these sources and in the backbone's pre-training can carry into decisions. Automated routing can systematically misroute users whose phrasing differs from the training data (dialects, non-native speakers, assistive phrasing); monitor misroutes by group where possible.
• Oversight. Not for high-stakes decisions without human review.

14. Versioning and lineage

proof_v3 is a new architecture trained from the pretrained Ettin backbone (MODE="scratch"); it doesn't inherit proof_v2's weights. FalconDec versions follow their own numbering: fine-tuning a FalconDec checkpoint bumps the patch version (1.0.0 → 1.0.1); a new scratch run bumps the major version.

Changelog. v1.0.0: first release, small preset, seed 42.

15. API reference

All functions live in falcondec_modeling.py.

Question fields: type (choice / noul / score, default choice); question or instructions; options (list) or criteria (dict {key: description} for choice, list of levels for score); labels ({"true": …, "false": …} wording for noul); option_tokens and seq_len (optional per-question budgets).

Model attributes: model.fcfg (the live config: layout, special, defer_threshold, version, lineage); model.temperature (3 × 4 tensor); model.num_parameters(); model.device.

16. Citation and references

@misc{falconsai_proof_v3_2026,
  title        = {proof_v3 (FalconDec v1.0.0): a single-pass, typed, calibrated decision model},
  author       = {{Falconsai}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Falconsai/proof_v3}},
  note         = {Ettin-150M backbone; successor to Falconsai/proof_v2}
}

Methods. Warner et al. (2024), ModernBERT. Weller et al. (2025), Ettin encoders. Gneiting and Raftery (2007), Strictly Proper Scoring Rules. Guo et al. (2017), On Calibration of Modern Neural Networks. Geifman and El-Yaniv (2017), Selective Classification. Zaheer et al. (2017), Deep Sets; Lee et al. (2019), Set Transformer. Williams (1992), REINFORCE; Shao et al. (2024), GRPO. Hinton, Vinyals and Dean (2015), Distillation. Related decision models: Laya (Convai Innovations), TypeSafe Jev, Together Tev1.

Data. ARC, OpenBookQA, SciQ, CommonsenseQA, QASC, HellaSwag, WinoGrande, MMLU, BoolQ, GSM8K, AQuA-RAT, SNLI, MultiNLI, ANLI, SciTail, CLINC150, MASSIVE, Bitext, Banking77, AG News, Yelp, DAIR Emotion, SST-5, jailbreak-classification, Civil Comments, deepset prompt-injections, CodeXGLUE, MBPP, HumanEval, LocalLLaMA/typed-decisions.

Architecture

• Identification: NLP · Small Language Model (SLM) (98% confidence)
• Source format: safetensors · Intended task: not declared
• config.json: synthesized from the anatomy (no source config.json); model_type omitted — no architecture name in the source (QA-F-126)
• Source license: not recorded (uploaded file)
• Lineage chain: 1 surgery (no prior attestation reachable)
• Post-surgery totals: 159,654,157 parameters · 168 tensors
• Compute estimate: 15.02439 GFLOPs (comparison metric, not a measurement)

Provenance & operations

• Parents: model.safetensors
• Operations performed: load×1
• Weight merges recorded: 0
• Quantized tensors (F32→F16): 0

Surgery Log (ordered)

1. load — model.safetensors (319.3 MB, safetensors)

Validation

• Tissue imaging: not run
• Structural integrity is testable offline via the packaged load_and_test.py.

Compliance note

The signed attestation + this card together document model composition, modification history, and validation evidence — the record structure technical-documentation obligations (e.g. EU AI Act Annex IV) ask for. This is evidence, not legal advice.

Operated with Model Surgeon — verify this package at https://surgeon.falcons.ai/verify © 2026 FALCONS.AI — Model Surgeon record format. The model weights remain their owner's.