Hierarchical document classifier for the LLM-Mailroom intake pipeline: a
fine-tuned ModernBERT-base encoder with a doc_type head plus one
subclass head per document class. It is the deterministic pre-check in the
BERT-coupled intake overhaul (mailroom-issues #85).
Architecture
- Backbone:
answerdotai/ModernBERT-base — 22 layers, 768 hidden,
8,192-token context, bf16.
- Heads: one head per class —
doc_type (6 classes) plus 5 subclass
heads (contract, corporate_record, correspondence, insurance_claim,
merger_agreement). MLP heads with dropout 0.1.
- Windowing: token-level, 8,192 tokens with 512-token overlap.
doc_type
by plurality vote over windows; subclass by plurality over windows whose
doc_type vote is the winning class.
- Calibration: per-head temperature scaling (
temperatures.json).
Files
| file |
purpose |
model.safetensors |
ModernBERT backbone weights (bf16, ~298 MB) |
heads.pt |
hierarchical head state dicts |
labels.json |
head vocabularies (labels / label2id / id2label / weights) |
temperatures.json |
per-head calibration temperatures |
train_counts.json |
per-(doc_type, subclass) authentic train-row counts (support gate) |
config.json, tokenizer.json, tokenizer_config.json |
backbone config + tokenizer |
summary.json |
full run summary (hyperparameters, per-epoch metrics, selection, test metrics) |
Labels
doc_type (6): contract, merger_agreement, corporate_record,
correspondence, insurance_claim, unknown (inference-only abstention —
not a trained class)
contract (24): agency, co_branding, collaboration, consulting,
development, distributor, endorsement, franchise, hosting, ip, joint_venture,
license, maintenance, manufacturing, marketing, other, outsourcing, promotion,
reseller, service, sponsorship, strategic_alliance, supply, transportation
corporate_record (10): articles_of_incorporation, board_resolution,
bylaws, charter_amendment, indenture, officer_certificate, other,
powers_of_attorney, rights_instrument, subsidiary_list
correspondence (7): demand, email, letter, meeting_request, memo, notice,
press_release
insurance_claim (6): auto, carrier, inpatient, outpatient, pde, property
merger_agreement (5): all_cash, all_stock, mixed_cash_stock,
mixed_cash_stock_election, other
Results
Trained 2 epochs on Lucius-Morningstar/mailroom-modernbert-training
@ 5b72a345cd3c057b736bea4910fdbef6509ad1c3 — 4,497 train / 489 validation
windows; 323 held-out test documents (never used for training, calibration, or
threshold tuning).
Validation
| epoch |
val_loss |
doc_acc |
macro-F1 (observed) |
ECE (calibrated) |
| 1 |
1.0041 |
0.8859 |
0.8524 |
0.0201 |
| 2 |
0.9121 |
0.9195 |
0.9051 |
0.0205 |
Selected epoch: 2 — best observed doc_type macro-F1 subject to
calibrated ECE ≤ 0.05 (gate_met: true).
Held-out test (323 docs)
| metric |
value |
doc_type accuracy |
0.9319 (301/323) |
subclass accuracy (given correct doc_type) |
0.5449 (164/301) |
Per-head (epoch 2, validation)
| head |
window acc |
macro-F1 (observed) |
ECE (calibrated) |
doc_type |
0.9141 |
0.9051 |
0.0205 |
insurance_claim |
0.8889 |
0.8085 |
0.0500 |
corporate_record |
0.5405 |
0.2218 |
0.1037 |
merger_agreement |
0.4656 |
0.1988 |
0.0350 |
correspondence |
0.5161 |
0.0980 |
0.1050 |
contract |
0.1630 |
0.0899 |
0.0630 |
Usage
The backbone is a standard ModernBertModel; the hierarchical heads are a
custom bundle. Load it with the mailroom-ml inference layer:
from mailroom_ml.inference import load_bundle, classify_document
bundle = load_bundle("Lucius-Morningstar/mailroom-modernbert-classifier")
result = classify_document(
title="Notice of Default",
text=document_text,
bundle=bundle,
)
# result["doc_type"], result["subclass"], result["confidence"], result["route"]
route == "fast_path" means the calibrated gate passed and the LLM sorter may
be skipped (skip mode + allowlist only); otherwise route the document to the
LLM sorter with the BERT triage as an advisory prior.
Provenance
- Training run:
20260920-173810 — 2 epochs, ~3.78 h on a Modal L4.
- Hyperparameters: batch 4, grad-accum 8, lr 2e-5, seed 42,
λ_dt 0.65,
label smoothing 0.05, sqrt-inverse class weights (cap 10), MLP heads
(dropout 0.1), weight decay 0.01, subclass support floor 12.
- Dataset revision:
5b72a345cd3c057b736bea4910fdbef6509ad1c3.
- Calibration temperatures:
doc_type 0.484, contract 0.705,
corporate_record 0.596, correspondence 0.960, insurance_claim 0.122,
merger_agreement 0.679.
Limitations
- Subclass heads are weak for
contract / correspondence /
corporate_record / merger_agreement (macro-F1 0.09–0.22). Use the
doc_type head for routing; route subclass-ambiguous documents to the LLM
sorter (the Tier-1 prior-scoped lane, mailroom-issues #108). Do not
trust a skip-mode subclass for these classes yet.
- Trained on a curated legal-document corpus; not a substitute for legal
review.
unknown is an inference-only abstention label, not a trained class.- Long documents are windowed (8,192 tokens, 512 overlap); the model never
truncates silently — oversize documents fall back to the LLM path.
License
Apache-2.0 (inherits answerdotai/ModernBERT-base).
