gliner_small-v2.5-onnx

gliner_small-v2.5 — ONNX / INT8

ONNX conversion of gliner-community/gliner_small-v2.5, dynamically quantised to INT8, packaged as a self-contained bundle for offline NER.

This is a re-serialisation, not a fine-tune: the weights are the upstream ones. Only the format (PyTorch → ONNX) and the precision (fp32 → INT8) are ours.

Model summary

Files

The fp32 reference graph (model.onnx, sha256 5245733ccb2b75072cce0b4bbb14424988f92f9daf775d97bdf0de74be28df63) is not shipped — it is only needed to reproduce the INT8 graph. Its hash is recorded in NOTICE.

Quick start

from huggingface_hub import hf_hub_download

REPO = "GG-QandV/gliner_small-v2.5-onnx"
for f in ("model_quantized.onnx", "tokenizer.json", "tokenizer_config.json", "gliner_config.json"):
    hf_hub_download(REPO, filename=f, local_dir="models/gliner_small-v2.5-onnx")

The graph has six inputs, fed per span-encoded prompt:

input_ids, attention_mask, words_mask, text_lengths, span_idx, span_mask

span_mask is bool (not int64) — the graph declares tensor(bool). The prompt follows the GLiNER label format, using the model's own special tokens from gliner_config.json: <<ENT>> <type> … <<SEP>> <text words>. Word splitting is whitespace-based (words_splitter_type: whitespace), subtoken_pooling: first. Inference in ONNX Runtime on CPU; result is a logits tensor of shape (batch, words, max_width, num_types); entities are sigmoid(logits) ≥ threshold (the reference runtime uses ner_score_threshold = 0.5).

Export & quantisation

# dev-only, requires torch; the ONNX bundle is what ships, torch is never a runtime dependency
python scripts/export_gliner_onnx.py \
    --src      gliner_small_v25_hf \
    --out      gliner_small_v25_bundle \
    --revision f227d3cd637bd4e6757ae143935316d062393341

Evaluation

Measured on the source bundle during acceptance (ONNX Runtime CPU, 12 vCPU).

Parity with upstream

20 texts (10 ru + 10 uk), 6 entity types, identical session options on both paths, same model_quantized.onnx:

The port is numerically exact against the upstream implementation, at both ORT_ENABLE_ALL and ORT_DISABLE_ALL.

INT8 vs fp32 reference

32 texts, organisation/person/location/date/technology/product, threshold 0.5:

The two optimisation levels produce identical entity sets (0 differences). All 7 vs-fp32 "misses" resolve to type reassignment (ORG → LOC, technology → product), span splits (Наталія Бондаренко → two tokens; 15 марта 2027 → date + year) or dropped low-score noise — no entity is genuinely lost. Large score deltas (up to 0.40) occur only on sub-threshold noise spans.

Latency & memory (INT8, 6 types, p50, 12 vCPU)

Both levels are numerically equivalent; the choice is purely memory vs speed.

Languages

10 PER/ORG/LOC sentences per language, threshold 0.5:

All 19 languages score ≥ 0.77. Degradation tracks sub-token fragmentation (up to 2.9 sub-tokens per word in Turkish) rather than language distance.

Entity types (measured, not asserted)

Of the eleven candidate types tested, six were kept for production: person, organization, location, date, technology, product.

Changing the type list changes the input prompt and therefore shifts scores; a type set must be validated as a whole.

Intended use & limitations

• Intended: offline, open-vocabulary NER on European languages inside a local runtime.
• Not intended: as a legal, medical or safety-critical extractor; as a source of ground truth.
• Organisation recall is weak and organization counters should not be treated as authoritative.
• This is an INT8 graph: expect small, threshold-adjacent differences from the fp32 reference.
• Dates may be split into components; downstream code should normalise before storing them.

Licensing

Weights: Apache-2.0, inherited from gliner-community/gliner_small-v2.5 (the upstream model card declares Apache-2.0; the GLiNER project itself is Apache-2.0). The full license text is in LICENSE; provenance and all hashes are in NOTICE. No modification of the model itself was made — only ONNX serialisation and INT8 quantisation.

Citation

@inproceedings{zaratiana-etal-2024-gliner,
  title     = {{GLiNER}: Generalist Model for Named Entity Recognition using Bidirectional Transformer},
  author    = {Zaratiana, Urchade and Tomeh, Nadi and Holat, Pierre and Charnois, Thierry},
  booktitle = {Proceedings of NAACL 2024},
  year      = {2024}
}