Table of Contents
- Overview
- NLI Evaluation Results
- Usage
- Recommended Applications
- Extended Evaluation (45 datasets)
- Acknowledgments
- Citation
Overview
This model is a fine-tune of jhu-clsp/ettin-encoder-1b,
trained on the dleemiller/FineCat-NLI-Reason dataset, a compilation of several high-quality
NLI data sources with quality screening and reduction of easy samples in the training split. Also includes additional
NLI, fact verification and long document sources (2.05M pairs in total).
The training uses logit distillation from an NLI finetune of Qwen3.8-27B.
Distillation loss looks like this:
$$
\begin{equation}
\mathcal{L} = \alpha \cdot \mathcal{L}_{\text{CE}}(z^{(s)}, y) + \beta \cdot T^2 \cdot \mathrm{KL}\left(\sigma(z^{(t)}/T) \,|\, \sigma(z^{(s)}/T)\right)
\end{equation}
$$
where \(z^{(s)}\) and \(z^{(t)}\) are the student and teacher logits, \(y\) are the ground truth labels,
\(\alpha = 0.2\), \(\beta = 0.8\) and \(T = 1\).
Training inputs keep the full documents the teacher labelled (up to 8k tokens). The model was trained for 3 epochs
at a constant learning rate followed by a 1-epoch cooldown. The released weights are an average of the final weights
and earlier exponential-moving-average checkpoints, with the output temperature calibrated on the validation split.
NLI Evaluation Results
F1-Micro scores (equivalent to accuracy) for each dataset.
Performance was measured at bs=32 using a Nvidia Blackwell PRO 6000 Max-Q.
| Model |
finecat |
mnli |
mnli_mismatched |
snli |
anli_r1 |
anli_r2 |
anli_r3 |
wanli |
lingnli |
Throughput (samples/s) |
Peak GPU Mem (MB) |
dleemiller/finecat-nli-v2-xl |
0.8445 |
0.9202 |
0.9191 |
0.9356 |
0.7860 |
0.6600 |
0.6067 |
0.7710 |
0.8834 |
254.42 |
4649.37 |
dleemiller/finecat-nli-ultra-l |
0.8388 |
0.9127 |
0.9131 |
0.9303 |
0.7910 |
0.6400 |
0.5833 |
0.7694 |
0.8714 |
624.67 |
1857.54 |
dleemiller/finecat-nli-v2-l |
0.8359 |
0.9165 |
0.9127 |
0.9328 |
0.7740 |
0.6020 |
0.5733 |
0.7674 |
0.8776 |
636.71 |
1982.06 |
MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli |
0.8233 |
0.9121 |
0.9079 |
0.8898 |
0.7960 |
0.6830 |
0.6400 |
0.7700 |
0.8821 |
454.96 |
3250.44 |
dleemiller/finecat-nli-l |
0.8227 |
0.9152 |
0.9265 |
0.9162 |
0.7480 |
0.5700 |
0.5433 |
0.7706 |
0.8742 |
539.04 |
1838.06 |
tasksource/ModernBERT-large-nli |
0.7959 |
0.8983 |
0.9229 |
0.9188 |
0.7260 |
0.5110 |
0.4925 |
0.6978 |
0.8504 |
543.44 |
1838.06 |
dleemiller/ModernCE-large-nli |
0.7811 |
0.9088 |
0.9205 |
0.9273 |
0.6630 |
0.4860 |
0.4408 |
0.6576 |
0.8566 |
540.74 |
1838.06 |
cross-encoder/nli-deberta-v3-large |
0.7618 |
0.9019 |
0.9049 |
0.9220 |
0.5300 |
0.4170 |
0.3758 |
0.6548 |
0.8466 |
448.35 |
3250.44 |
Usage
Label Map:
entailment: 0neutral: 1contradiction: 2
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load the model and run inference.
from sentence_transformers import CrossEncoder
import numpy as np
model = CrossEncoder("dleemiller/finecat-nli-v2-xl")
id2label = model.model.config.id2label # {0:'entailment', 1:'neutral', 2:'contradiction'}
pairs = [
("The glass fell off the counter and shattered on the tile.",
"The glass broke when it hit the floor."), # E
("The store opens at 9 a.m. every day.",
"The store opens at 7 a.m. on weekdays."), # C
("A researcher presented results at the conference.",
"The presentation won the best paper award."), # N
("It started raining heavily, so the match was postponed.",
"The game was delayed due to weather."), # E
("Every seat on the flight was taken.",
"There were several empty seats on the plane."), # C
]
logits = model.predict(pairs) # shape: (5, 3)
for (prem, hyp), row in zip(pairs, logits):
pred_idx = int(np.argmax(row))
pred = id2label[pred_idx]
print(f"[{pred}] Premise: {prem} | Hypothesis: {hyp}")
Recommended Applications
Frame each check as a premise (text) and hypothesis (statement), and use the entailment probability.
| Application |
Example hypothesis |
Supporting results |
| Zero-shot tagging / topic classification |
"This text is about billing." |
zs_meta, fin_topics |
| Routing and decisions in agent pipelines |
"The user wants to cancel an order." |
zs_meta, mnli, snli |
| RAG grounding / hallucination checks |
Each answer sentence vs. retrieved passages |
ragtruth, wice, vitaminc |
| Contract and policy review |
"The agreement permits sharing data with third parties." |
contractnli_a/b, control, docnli |
| Biomedical and health claim checking |
Claim vs. abstract, clinical note or trial eligibility criteria |
sdoh_nli, biosift, bionli, scifact, pubhealth, nli4ct |
| Logical / rule-based reasoning over documents |
"Under these rules, the applicant qualifies." |
fol, control |
| Dialogue fact-checking |
Assistant reply vs. knowledge source |
dialfact, vitaminc |
| Guardrails / policy matching |
Policy statement vs. user or model text |
crossingguard |
| Stance detection |
"The author supports the proposal." |
vast |
Less suited: financial and general legal verification (findver, legal_domain), and presupposition reasoning (nope_adv).
Extended Evaluation (45 datasets)
Accuracy on the test splits of 45 NLI and adjacent datasets.
2-way datasets (entailed vs. not) are scored as entailment vs. any non-entailment prediction.
DocNLI and VitaminC are evaluated on fixed random subsets of 1,000 and 10,000 pairs.
FRANK (news-summary factuality) is scored per summary sentence: each sentence vs. its source article, labelled by the majority annotator judgment.
| Dataset |
n |
finecat-nli-v2-xl |
27B teacher (fast mode) |
| finecat |
18012 |
0.8445 |
0.8656 |
| mnli |
9815 |
0.9202 |
0.9311 |
| mnli_mismatched |
9830 |
0.9191 |
0.9305 |
| snli |
9824 |
0.9356 |
0.9399 |
| anli_r1 |
1000 |
0.7860 |
0.8630 |
| anli_r2 |
1000 |
0.6600 |
0.7490 |
| anli_r3 |
1200 |
0.6067 |
0.7100 |
| wanli |
5000 |
0.7710 |
0.7796 |
| lingnli |
7361 |
0.8834 |
0.8958 |
| glue_diagnostic |
1100 |
0.7582 |
0.8118 |
| med |
5380 |
0.8147 |
0.8039 |
| nan_nli |
258 |
0.7171 |
0.7403 |
| scone_test |
1000 |
1.0000 |
0.9980 |
| nmonli_test |
200 |
1.0000 |
1.0000 |
| conjnli |
623 |
0.7191 |
0.7335 |
| nope_adv |
345 |
0.5739 |
0.5768 |
| nli4ct |
199 |
0.8141 |
0.8693 |
| sdoh_nli |
990 |
0.9879 |
0.9848 |
| healthver |
230 |
0.7609 |
0.7565 |
| scifact |
300 |
0.8733 |
0.8433 |
| biosift |
1000 |
0.9200 |
0.9210 |
| scientsbank |
971 |
0.7508 |
0.7683 |
| beetle |
819 |
0.6422 |
0.6325 |
| legal_domain |
1163 |
0.4918 |
0.6389 |
| findver |
500 |
0.6560 |
0.7660 |
| dialfact |
1499 |
0.8412 |
0.8586 |
| frank_sent |
3176 |
0.6870 |
0.8234 |
| ragtruth |
1497 |
0.8544 |
0.8691 |
| vast |
1500 |
0.7933 |
0.8053 |
| zs_meta |
5400 |
0.9304 |
0.9344 |
| seahorse |
800 |
0.8562 |
0.8638 |
| pubhealth |
600 |
0.8500 |
0.8650 |
| bionli |
600 |
0.9133 |
0.9333 |
| fin_topics |
600 |
0.9767 |
0.9633 |
| vitaminc |
10000 |
0.9062 |
0.9351 |
| docnli |
1000 |
0.8530 |
— |
| contractnli_a |
1707 |
0.9279 |
0.9285 |
| contractnli_b |
2074 |
0.8954 |
0.9021 |
| control |
800 |
0.7525 |
0.8087 |
| fol |
10277 |
0.8727 |
0.8430 |
| scitail |
2126 |
0.9661 |
0.9652 |
| wice |
958 |
0.8810 |
0.9144 |
| sick |
4906 |
0.9317 |
0.9213 |
| crossingguard |
5397 |
0.9250 |
0.9344 |
| hans |
15000 |
0.8463 |
0.9209 |
Macro average over the 45 datasets: 0.8282.
Acknowledgments
We thank the creators and contributors of jhu-clsp (Ettin) and the Qwen team for making their work available.
This model would not be possible without their efforts and open source contributions.
Citation
@misc{nli-compiled-2025,
title = {FineCat NLI Dataset},
author = {Lee Miller},
year = {2025},
howpublished = {Refined compilation of 6 major NLI datasets}
}
