finecat-nli-ultra-l by Lee Miller: Open-Weight Model

Runs On

What it takes to serve finecat-nli-ultra-l (378M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

Precision	Weights	Memory needed	Cheapest setup	Per hour	Also fits
16-bit	0.8 GB	0.9 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
8-bit	0.4 GB	0.5 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00
4-bit	0.2 GB	0.2 GB	1x MI300X (192 GB) Vultr	$1.85	1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 20, 2026.

finecat-nli-ultra-l on every accelerator the SAVRN Index prices, at every precision

Model Card

By Lee Miller, published under apache-2.0, revision 92c53207956e.

This model is a fine-tune of dmis-lab/UltraBERT-380M, trained on the dleemiller/FineCat-NLI-Reason dataset, a compilation of several high-quality NLI data sources with quality screening and reduction of easy samples in the training split. Also includes additional NLI, fact verification and long document sources (2.05M pairs in total). The training uses logit distillation from an NLI finetune of Qwen3.8-27B. \begin{equation} \mathcal{L} = \alpha \cdot \mathcal{L}{\text{CE}}(z^{(s)}, y) + \beta \cdot T^2 \cdot \mathrm{KL}\left(\sigma(z^{(t)}/T) \,\|\, \sigma(z^{(s)}/T)\right) \end{equation} where \$z^{(s)}\$ and \$z^{(t)}\$ are the student and teacher logits, \$y\$ are the ground truth…

Read Lee Miller's full model card

FineCat-NLI Ultra Large

Overview

This model is a fine-tune of dmis-lab/UltraBERT-380M, trained on the dleemiller/FineCat-NLI-Reason dataset, a compilation of several high-quality NLI data sources with quality screening and reduction of easy samples in the training split. Also includes additional NLI, fact verification and long document sources (2.05M pairs in total). The training uses logit distillation from an NLI finetune of Qwen3.8-27B.

Distillation loss looks like this: $$ \begin{equation} \mathcal{L} = \alpha \cdot \mathcal{L}_{\text{CE}}(z^{(s)}, y) + \beta \cdot T^2 \cdot \mathrm{KL}\left(\sigma(z^{(t)}/T) \,|\, \sigma(z^{(s)}/T)\right) \end{equation} $$

where $z^{(s)}$ and $z^{(t)}$ are the student and teacher logits, $y$ are the ground truth labels, $\alpha = 0.2$, $\beta = 0.8$ and $T = 1$.

Training inputs keep the full documents the teacher labelled (up to 8k tokens). The model was trained for 3 epochs at a constant learning rate followed by a 1-epoch cooldown. The released weights are an average of the final weights and an earlier exponential-moving-average checkpoint, with the output temperature calibrated on the validation split. UltraBERT is fine-tuned with its native classification head (first-token pooling) and the UltraBERT authors' optimizer settings (weight decay 0.1, 5% warmup), at a learning rate of 1.5e-5.

NLI Evaluation Results

F1-Micro scores (equivalent to accuracy) for each dataset. Performance was measured at bs=32 using a Nvidia Blackwell PRO 6000 Max-Q.

Model	finecat	mnli	mnli_mismatched	snli	anli_r1	anli_r2	anli_r3	wanli	lingnli	Throughput (samples/s)	Peak GPU Mem (MB)
`dleemiller/finecat-nli-v2-xl`	0.8445	0.9202	0.9191	0.9356	0.7860	0.6600	0.6067	0.7710	0.8834	254.42	4649.37
`dleemiller/finecat-nli-ultra-l`	0.8388	0.9127	0.9131	0.9303	0.7910	0.6400	0.5833	0.7694	0.8714	624.67	1857.54
`dleemiller/finecat-nli-v2-l`	0.8359	0.9165	0.9127	0.9328	0.7740	0.6020	0.5733	0.7674	0.8776	636.71	1982.06
`MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli`	0.8233	0.9121	0.9079	0.8898	0.7960	0.6830	0.6400	0.7700	0.8821	454.96	3250.44
`dleemiller/finecat-nli-l`	0.8227	0.9152	0.9265	0.9162	0.7480	0.5700	0.5433	0.7706	0.8742	539.04	1838.06
`tasksource/ModernBERT-large-nli`	0.7959	0.8983	0.9229	0.9188	0.7260	0.5110	0.4925	0.6978	0.8504	543.44	1838.06
`dleemiller/ModernCE-large-nli`	0.7811	0.9088	0.9205	0.9273	0.6630	0.4860	0.4408	0.6576	0.8566	540.74	1838.06
`cross-encoder/nli-deberta-v3-large`	0.7618	0.9019	0.9049	0.9220	0.5300	0.4170	0.3758	0.6548	0.8466	448.35	3250.44

Usage

Label Map:

entailment: 0
neutral: 1
contradiction: 2

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers "transformers>=5"

Then you can load the model and run inference. UltraBERT uses custom model code, so pass trust_remote_code=True.

from sentence_transformers import CrossEncoder
import numpy as np

model = CrossEncoder("dleemiller/finecat-nli-ultra-l", trust_remote_code=True)
id2label = model.model.config.id2label  # {0:'entailment', 1:'neutral', 2:'contradiction'}

pairs = [
    ("The glass fell off the counter and shattered on the tile.",
     "The glass broke when it hit the floor."),          # E
    ("The store opens at 9 a.m. every day.",
     "The store opens at 7 a.m. on weekdays."),          # C
    ("A researcher presented results at the conference.",
     "The presentation won the best paper award."),       # N
    ("It started raining heavily, so the match was postponed.",
     "The game was delayed due to weather."),             # E
    ("Every seat on the flight was taken.",
     "There were several empty seats on the plane."),     # C
]

logits = model.predict(pairs)  # shape: (5, 3)

for (prem, hyp), row in zip(pairs, logits):
    pred_idx = int(np.argmax(row))
    pred = id2label[pred_idx]
    print(f"[{pred}]  Premise: {prem}  |  Hypothesis: {hyp}")

Recommended Applications

Frame each check as a premise (text) and hypothesis (statement), and use the entailment probability.

Application	Example hypothesis	Supporting results
Zero-shot tagging / topic classification	"This text is about billing."	zs_meta, fin_topics
Routing and decisions in agent pipelines	"The user wants to cancel an order."	zs_meta, mnli, snli
RAG grounding / hallucination checks	Each answer sentence vs. retrieved passages	ragtruth, wice, vitaminc
Contract and policy review	"The agreement permits sharing data with third parties."	contractnli_a/b, control, docnli
Biomedical and health claim checking	Claim vs. abstract, clinical note or trial eligibility criteria	sdoh_nli, biosift, bionli, scifact, pubhealth, nli4ct
Logical / rule-based reasoning over documents	"Under these rules, the applicant qualifies."	fol, control
Dialogue fact-checking	Assistant reply vs. knowledge source	dialfact, vitaminc
Guardrails / policy matching	Policy statement vs. user or model text	crossingguard
Stance detection	"The author supports the proposal."	vast

Less suited: financial and general legal verification (findver, legal_domain), and presupposition reasoning (nope_adv).

Extended Evaluation (45 datasets)

Accuracy on the test splits of 45 NLI and adjacent datasets. 2-way datasets (entailed vs. not) are scored as entailment vs. any non-entailment prediction. DocNLI and VitaminC are evaluated on fixed random subsets of 1,000 and 10,000 pairs. FRANK (news-summary factuality) is scored per summary sentence: each sentence vs. its source article, labelled by the majority annotator judgment.

Dataset	n	`finecat-nli-ultra-l`	27B teacher (fast mode)
finecat	18012	0.8387	0.8656
mnli	9815	0.9127	0.9311
mnli_mismatched	9830	0.9133	0.9305
snli	9824	0.9303	0.9399
anli_r1	1000	0.7910	0.8630
anli_r2	1000	0.6400	0.7490
anli_r3	1200	0.5808	0.7100
wanli	5000	0.7694	0.7796
lingnli	7361	0.8709	0.8958
glue_diagnostic	1100	0.7473	0.8118
med	5380	0.7939	0.8039
nan_nli	258	0.7209	0.7403
scone_test	1000	1.0000	0.9980
nmonli_test	200	1.0000	1.0000
conjnli	623	0.7464	0.7335
nope_adv	345	0.5159	0.5768
nli4ct	199	0.7990	0.8693
sdoh_nli	990	0.9818	0.9848
healthver	230	0.7739	0.7565
scifact	300	0.8667	0.8433
biosift	1000	0.9280	0.9210
scientsbank	971	0.7456	0.7683
beetle	819	0.6117	0.6325
legal_domain	1163	0.5494	0.6389
findver	500	0.6500	0.7660
dialfact	1499	0.8312	0.8586
frank_sent	3176	0.7554	0.8234
ragtruth	1497	0.8430	0.8691
vast	1500	0.7800	0.8053
zs_meta	5400	0.9278	0.9344
seahorse	800	0.8500	0.8638
pubhealth	600	0.8383	0.8650
bionli	600	0.8983	0.9333
fin_topics	600	0.9683	0.9633
vitaminc	10000	0.8905	0.9351
docnli	1000	0.8350	—
contractnli_a	1707	0.9320	0.9285
contractnli_b	2074	0.8954	0.9021
control	800	0.7338	0.8087
fol	10277	0.8562	0.8430
scitail	2126	0.9633	0.9652
wice	958	0.8716	0.9144
sick	4906	0.9311	0.9213
crossingguard	5397	0.9198	0.9344
hans	15000	0.8551	0.9209

Macro average over the 45 datasets: 0.8234.

Acknowledgments

We thank the creators and contributors of dmis-lab (UltraBERT) and the Qwen team for making their work available. This model would not be possible without their efforts and open source contributions.

Citation

@misc{nli-compiled-2025,
  title = {FineCat NLI Dataset},
  author = {Lee Miller},
  year = {2025},
  howpublished = {Refined compilation of 6 major NLI datasets}
}

Configuration

Architecture: UltrabertForSequenceClassification
Layers: 24
Hidden size: 1,024
Feed-forward size: 3,072
Attention heads: 8
Key/value heads: 8
Head dimension: 128
Vocabulary size: 49,152
Model type: ultrabert

Identity and Version

Repository: dleemiller/finecat-nli-ultra-l
Publisher: Lee Miller
Task: Zero-shot classification
Modality: Text
Library: sentence-transformers
Parameters: 378M parameters
Languages: en
Revision: 92c53207956efe726ae5d4192f1555bdd3c9b2e1
First published: 2026-10-16
Last updated: 2026-10-17

Files and Weights

17 files, 1.5 GB in total. The weights are 1 file totalling 1.5 GB in safetensors.

Weights1 file · 1.5 GB

Configuration8 files · 42.9 KB

Tokenizer2 files · 3.5 MB

Documentation1 file · 9.7 KB

Other4 files · 1.0 MB

Repository1 file · 1.7 KB

Every file

File	Type	Size	SHA-256
model.safetensors	Weights	1.5 GB	92b459f6c659
calibration.json	Configuration	499 B	—
config.json	Configuration	1.9 KB	—
config_sentence_transformers.json	Configuration	259 B	—
configuration_ultrabert.py	Configuration	2.2 KB	—
greedy_soup.json	Configuration	7.1 KB	—
modeling_ultrabert.py	Configuration	30.6 KB	—
modules.json	Configuration	131 B	—
sentence_bert_config.json	Configuration	225 B	—
README.md	Documentation	9.7 KB	—
finecat_ultra.jpeg	Other	100.9 KB	191bd2d741e0
finecat_ultra_dark.png	Other	362.6 KB	819f74747bfc
finecat_ultra_light.png	Other	264.3 KB	0ef6244980da
finecatv2.jpeg	Other	321.9 KB	c6e5f5562cb2
.gitattributes	Repository	1.7 KB	—
tokenizer.json	Tokenizer	3.5 MB	—
tokenizer_config.json	Tokenizer	1.0 KB	—

License and Download

License: apache-2.0
Access: Open weights, no gate
Download size: 1.5 GB

Download from Lee Miller

Released by Lee Miller through its official repository on Hugging Face. Read the license.

Built From

Derived from dmis-lab/UltraBERT-380M
Trained on (disclosed) dleemiller/FineCat-NLI

Evaluations

Each result is shown as reported, with the conditions its reporter stated. None is a SAVRN measurement. A comparison lines two results up only when their configuration, unit and setup are all stated and identical.

Benchmark	Conditions	Result	Reported by	Revision	Date
dleemiller FineCat NLI test	Task Cross Encoder ClassificationMetric AccuracyComparison conditions not established	0.8388	dleemiller Publisher reported	Evaluated revision not stated	—

Memory Requirements

Precision	Weights in memory
As published	1.5 GB
16-bit	0.8 GB
8-bit	0.4 GB
4-bit	0.2 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About finecat-nli-ultra-l

How much GPU memory does finecat-nli-ultra-l need?

About 0.9 GB at 16-bit and 0.2 GB at 4-bit: the weights (378M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run finecat-nli-ultra-l on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use finecat-nli-ultra-l commercially?

Yes. finecat-nli-ultra-l is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

Similar Models

Model · Zero-shot classification

nitzotz

Netanel Elyasi

What it is. Nitzotz reads a Hebrew message and answers questions you type about it: pick one of several options, give a score on a scale, or say yes or no to a claim. For every answer it gives a probability you can trust, so you know when it is sure and when it is guessing. It does not write text, so it cannot make things up. It runs on a normal laptop, with no internet connection and no cost per question. What it is for. Deciding what to do with incoming messages: is this a scam, what kind of message is it, which department should get it, how urgent is it. It is not a chatbot and it is not built for long documents (see Limitations). In numbers. On 298 Hebrew messages it says correctly…

Open weights apache-2.0 384M parameters laya

View model

Model · Zero-shot classification

ModernBERT-large-nli

Tasksource

This model is ModernBERT multi-task fine-tuned on tasksource NLI tasks, including MNLI, ANLI, SICK, WANLI, doc-nli, LingNLI, FOLIO, FOL-NLI, LogicNLI, Label-NLI and all datasets in the below table). This is the equivalent of an "instruct" version. The model was trained for 200k steps on an Nvidia A30 GPU. It is very good at reasoning tasks (better than llama 3.1 8B Instruct on ANLI and FOLIO), long context reasoning, sentiment analysis and zero-shot classification with new labels. The following table shows model test accuracy. These are the scores for the same single transformer with different classification heads on top. Further gains can be obtained by fine-tuning on a single-task, e.g.…

Open weights apache-2.0 396M parameters 2,048 tokens transformers

View model

Model · Zero-shot classification

finecat-nli-l

Lee Miller

This model is a fine-tune of the excellent tasksource/ModernBERT-large-nli, trained on the dleemiller/FineCat-NLI dataset—a compilation of several high-quality NLI data sources with quality screening and reduction of easy samples in the training split. The training also incorporates logit distillation from MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli, a top-performing NLI model, particularly on ANLI benchmarks. \begin{equation} \mathcal{L} = \alpha \cdot \mathcal{L}{\text{CE}}(z^{(s)}, y) + \beta \cdot \mathcal{L}{\text{MSE}}(z^{(s)}, z^{(t)}) \end{equation} where \$z^{(s)}\$ and \$z^{(t)}\$ are the student and teacher logits, \$y\$ are the ground truth labels, and…

Open weights 396M parameters 2,048 tokens sentence-transformers

View model

Model · Zero-shot classification

finecat-nli-v2-l

Lee Miller

This model is a fine-tune of jhu-clsp/ettin-encoder-400m, trained on the dleemiller/FineCat-NLI-Reason dataset, a compilation of several high-quality NLI data sources with quality screening and reduction of easy samples in the training split. Also includes additional NLI, fact verification and long document sources (2.05M pairs in total). The training uses logit distillation from an NLI finetune of Qwen3.8-27B. \begin{equation} \mathcal{L} = \alpha \cdot \mathcal{L}{\text{CE}}(z^{(s)}, y) + \beta \cdot T^2 \cdot \mathrm{KL}\left(\sigma(z^{(t)}/T) \,\|\, \sigma(z^{(s)}/T)\right) \end{equation} where \$z^{(s)}\$ and \$z^{(t)}\$ are the student and teacher logits, \$y\$ are the ground…

Open weights apache-2.0 396M parameters 7,999 tokens sentence-transformers

View model

Model · Zero-shot classification

scandi-nli-large

Alexandra Institute

This model is a fine-tuned version of NbAiLab/nb-bert-large for Natural Language Inference in Danish, Norwegian Bokmål and Swedish. We have released three models for Scandinavian NLI, of different sizes: - alexandrainst/scandi-nli-large (this) A demo of the large-v2 model can be found in this Hugging Face Space - check it out! The performance and model size of each of them can be found in the Performance section below. You can use this model in your scripts as follows: We assess the models both on their aggregate Scandinavian performance, as well as their language-specific Danish, Swedish and Norwegian Bokmål performance. In all cases, we report Matthew's Correlation Coefficient (MCC)…

Open weights apache-2.0 355M parameters 512 tokens transformers

View model

Model · Zero-shot classification

bart-large-mnli-yahoo-answers

Joe Davison

This model takes facebook/bart-large-mnli and fine-tunes it on Yahoo Answers topic classification. It can be used to predict whether a topic label can be assigned to a given sequence, whether or not the label has been seen before. You can play with an interactive demo of this zero-shot technique with this model, as well as the non-finetuned facebook/bart-large-mnli, here. This model was fine-tuned on topic classification and will perform best at zero-shot topic classification. Use hypothesistemplate="This text is about {}." as this is the template used during fine-tuning. For settings other than topic classification, you can use any model pre-trained on MNLI such as facebook/bart-large-mnli…

Open weights apache-2.0 407M parameters 1,024 tokens transformers

View model

Runs On

Model Card

FineCat-NLI Ultra Large

Table of Contents

Overview

NLI Evaluation Results

Usage

Label Map:

Direct Usage (Sentence Transformers)

Recommended Applications

Extended Evaluation (45 datasets)

Acknowledgments

Citation

Configuration

Identity and Version

Files and Weights

License and Download

Built From

Evaluations

Memory Requirements

Questions About finecat-nli-ultra-l

How much GPU memory does finecat-nli-ultra-l need?

What is the cheapest GPU to run finecat-nli-ultra-l on?

Can I use finecat-nli-ultra-l commercially?

Similar Models

nitzotz

ModernBERT-large-nli

finecat-nli-l

finecat-nli-v2-l

scandi-nli-large

bart-large-mnli-yahoo-answers