embeddinggemma-2-text-int8-onnx

EmbeddingGemma 2 text encoder, int8 ONNX for CPU

The text encoder of google/embeddinggemma-2 (Google DeepMind, Apache-2.0), exported to ONNX and quantized to int8 for ONNX Runtime on CPU. It returns the same kind of vector as the original model: a 768-dimensional, L2-normalised text embedding. Its output for a text does not depend on what else is in the batch: the vectors are bitwise identical at every batch size we tested.

This is a modified version of the original model, not a retrained one. It is the dense retrieval model of Påmin Memory, and it can be used on its own with any ONNX Runtime application.

Why use this model

Sizes are file sizes in bytes / 10^6. The text tower sizes come from the upstream checkpoint's safetensors header: the language_model.* tensors hold 271,002,648 parameters, stored as bf16; fp32 is 4 bytes per parameter.

When not to use it

• On a GPU or NPU, use the original model. The int8 graph is built for the CPU execution provider. The upstream card recommends bf16 where the hardware supports it, else fp32, and warns that float16 can return NaN.
• If you need image, video or audio embeddings, use the original model. This encoder has no other towers. Its text vectors are meant to match the original's text vectors, but we have not measured how well they align with the original's image, video or audio vectors.
• If you need vectors that are bitwise identical across ONNX Runtime versions. The vectors are identical across batch sizes and thread counts within one ONNX Runtime version, not across versions: ORT 1.28 and 1.31 differ by up to 0.0111 per element. Keep the runtime version fixed for the life of an index, or re-embed when you change it.
• If you need the full 8,192-token context. The longest input in our tests has 471 tokens, and Påmin Memory truncates at 512. Longer inputs have not been validated for this conversion.

Usage

Inputs are input_ids and attention_mask (int64, [batch, sequence]), output is sentence_embedding ([batch, 768], float32, unit length). Mean pooling and normalisation are inside the graph. Use the bundled tokenizer.json (the upstream tokenizer, unchanged) and the upstream task prompts:

# pip install onnxruntime tokenizers numpy huggingface_hub
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

path = snapshot_download("pamin-labs/embeddinggemma-2-text-int8-onnx")
session = ort.InferenceSession(f"{path}/model.onnx", providers=["CPUExecutionProvider"])
tokenizer = Tokenizer.from_file(f"{path}/tokenizer.json")
tokenizer.enable_truncation(max_length=512)
tokenizer.enable_padding(pad_id=tokenizer.token_to_id("<pad>"), pad_token="<pad>")  # right padding

def embed(texts):
    enc = tokenizer.encode_batch(texts)
    feeds = {"input_ids": np.array([e.ids for e in enc], dtype=np.int64),
             "attention_mask": np.array([e.attention_mask for e in enc], dtype=np.int64)}
    return session.run(["sentence_embedding"], feeds)[0]   # [batch, 768], L2-normalised

queries = embed(["task: search result | query: What causes the northern lights?"])
documents = embed([
    "title: none | text: The northern lights are caused by charged particles from the sun.",
    "title: none | text: Bread rises because yeast produces carbon dioxide.",
])
print(queries.shape, np.linalg.norm(queries, axis=1))  # (1, 768) [~1.0]
print(queries @ documents.T)                           # cosine similarities; the first document scores higher

The upstream model supports Matryoshka truncation to 512, 256 or 128 dimensions: keep the leading dimensions and L2-normalise again. We have not measured truncated vectors of this int8 graph separately.

Påmin Memory runs this model one text per forward pass, with the search query and document prompts above.

What was changed

The weights are the upstream text tower at revision 914f7f8. Nothing was retrained or fine-tuned.

1. Text tower only. The vision and audio towers are not included.
2. ONNX export of the text tower with masked mean pooling and L2 normalisation, using the PyTorch exporter.
3. Graph rewrites with identical arithmetic in fp32: RMSNorm as SimplifiedLayerNormalization, tanh-GELU as FastGelu, rotary embeddings as com.microsoft.RotaryEmbedding, and grouped-query attention without the Expand that copies key/value heads (the shared heads are reached by reshaping the queries).
4. int8 quantization. - Weights of every linear layer: signed int8, one scale per output channel. - Activations: unsigned 8-bit, quantized at run time per token, one scale per row. For mlp.down_proj there is one scale per token and per group of 512 inputs. No scale is shared between the texts of a batch. - A per-token scale multiply is dropped where the result only feeds an RMSNorm, which removes any per-row factor. The up_proj scale is folded into the down_proj group scales. - The per-layer embedding projection is split into 24 per-layer linears. - The token embedding table is int8 with one scale per row, and the embedding scale is folded into those row scales.
5. Batch invariance. Two fp32 steps still depended on the padded batch shape on ONNX Runtime's CPU kernels: - The attention keys are padded inside the graph to at least 65 columns: zero keys, zero values and masked scores. Every attention product then takes the same kernel path whatever the batch's padded length. - Mean pooling uses a cumulative sum along the sequence, read at the last position. This is exact for padding rows, which are zero.

The released model.onnx keeps the per-node metadata the PyTorch exporter writes. That metadata includes source file paths from the machine that ran the export, and has no effect on inference.

Evaluation

All results below are for 768 dimensions on the CPU execution provider of a shared 4-vCPU Intel Xeon (2.10 GHz, AVX-512 VNNI and AMX). Timing was not part of any result. Measurements are labelled by scope:

• Component measurements run the model alone.
• Product entry point measurements run Påmin Memory's full search path. The model is one stage of that path, together with lexical retrieval, fusion and a reranker.

Fidelity to the original (component)

parity200: 200 texts from XQuAD in 9 languages (en, zh, de, es, ar, ru, th, vi, hi), half questions with the query prompt and half paragraph openings with the document prompt. The reference is the original model in fp32 through sentence-transformers with its documented prompts, truncated at 1,024 tokens.

Release gate, pipeline/verify.py (ONNX Runtime 1.28, 2 threads, tokenizers with right padding):

Earlier comparison with the previous revision, using the same 200 texts and fp32 reference but an earlier evaluation script outside pipeline/. That script's embedding path gives a lower minimum for this revision than verify.py (0.99552 against 0.99723). The means agree. Read the two columns against each other, not against the table above:

Batch invariance

36 texts of 17–471 tokens: 32 XQuAD paragraphs with the document prompt and 4 questions with the query prompt. Each text is embedded alone. It is then embedded in batches of 2, 8 and 32, in corpus order and in a shuffled order, and once as the shortest text in a batch with the 31 longest. Padding is on the right, up to 239 extra positions at 256-token truncation and up to 454 at 512.

Also measured:

• A text embedded alone gives identical vectors with 1 and with 2 threads (ORT 1.31).
• The same text differs by up to 0.0111 per element between ORT 1.28 and ORT 1.31.

Retrieval in Påmin Memory (product entry point)

These are measured through Påmin Memory's full search path (search: lexical and dense retrieval, fusion and a bge-reranker-v2-m3 int8 reranker), on the CPU, with 768-dimensional vectors.

• XQuAD-R: 1,190 queries per group over 13,014 paragraphs in 11 languages, scored in two groups. A cross-lingual query's relevant paragraph is in another language; a same-language query's is in its own.
• MuSiQue: the first 600 questions (all 2-hop) over their 7,274 paragraphs.

These runs used the previous revision of this graph (2d2e08e). On the embedder-only screen below, this revision gives the same results within noise. A product entry-point run of this revision has not been published yet.

• Against BGE-M3, which Påmin Memory used before, this model is lower cross-lingually (−0.0567 raw) and higher on same-language XQuAD-R (+0.0106) and on MuSiQue (+0.0208).
• The language transform is not part of this model. It is a query-side step in Påmin Memory. It blends each query vector with a copy from which language directions have been removed, using constants fitted to this model's vectors.
• Paired sign-flip tests of the transform against raw vectors:
• cross-lingual +0.0218, p = 0.0001;
• same-language −0.0018, p = 0.38;
• MuSiQue −0.0042, p = 0.14.
• The MuSiQue difference between BGE-M3 and this model is between two single runs, with no paired test.

To use the transform outside Påmin Memory, fit its constants for your exact model files with pipeline/fit_language_constants.py.

Embedder alone, this revision against the previous one (component)

The model ranks alone (no lexical channel, fusion or reranker) on pooled XQuAD-R (1,190 queries per group) and on MuSiQue (first 200 questions). Paired sign-flip tests:

No difference is significant.

Speed and memory

Not yet measured on a quiet machine. All runs so far were on a shared host under load, which is not good enough to publish latency. Speed depends heavily on the CPU's int8 support (AVX-512 VNNI, AVX-VNNI or AMX on x86).

Files

• model.onnx and model.onnx.data must sit in the same directory.
• tokenizer.json, tokenizer_config.json and config.json are the upstream files, unchanged. config.json describes the full multimodal model and is included for reference; the ONNX graph does not read it.
• checksums.sha256 lists every file in this repository except itself.
• pipeline/ rebuilds the model; CHANGELOG.md records the revisions; NOTICE holds the attribution.

Revisions

See CHANGELOG.md.

Reproduce and keeping up with upstream

pipeline/ rebuilds this model from the pinned upstream revision with one command, pipeline/build.sh, then runs the release gates: bitwise batch invariance, fidelity thresholds against fp32, and a NaN check. We rebuilt it from scratch with that pipeline:

• model.onnx.data came out byte-identical to the published file.
• model.onnx came out identical except for the exporter's source-path metadata. That metadata records where the build ran.

How this repository follows changes to the official google/embeddinggemma-2:

1. Pinned, not automatic. The build uses one exact upstream commit, recorded with file checksums in pipeline/upstream.json, currently 914f7f89142e33e77833254d9c9b90c3cef7303b. Upstream changes do not reach this repository by themselves.
2. Rebuild and gate. When upstream publishes a new revision, we bump the pin and rebuild with pipeline/build.sh. The new build must pass the release gates before it is published: bitwise batch invariance, the fidelity thresholds against the new upstream fp32 reference, and the NaN check. It must also pass a retrieval check.
3. New revision, never a silent replacement. The result is published as a new revision of this repository with a CHANGELOG entry and a git tag. The entry states whether the vectors changed; if they did, indexes must be re-embedded. Published files are never replaced without a new entry.
4. Language constants refit. Påmin Memory's query-side language constants are refitted for the new vectors with pipeline/fit_language_constants.py.
5. Pin a revision yourself so that an update never changes your vectors unexpectedly:

python snapshot_download("pamin-labs/embeddinggemma-2-text-int8-onnx", revision="<tag or commit sha>")

The full checklist and the resource needs are in the pipeline README.

Limitations and bias

This model inherits the limitations, biases and responsible-use considerations of the base model. Read the EmbeddingGemma 2 model card, in particular its Usage and Limitations and Ethical Considerations and Risks sections. The upstream card asks deployments to follow the Gemma Prohibited Use Policy.

Limitations specific to this conversion:

• Approximation. int8 quantization moves every vector slightly; see the fidelity table above.
• Cross-lingual retrieval through Påmin Memory is lower than with BGE-M3, which Påmin Memory used before. Its language transform recovers part of the gap.
• Runtime-dependent vectors. Vectors are not bitwise identical across ONNX Runtime versions or across different CPU kernels.
• Evaluated scope. Evaluation covers retrieval in 11 languages (XQuAD-R) and English multi-hop QA (MuSiQue). Other languages, domains and tasks were not measured for this conversion.

License and attribution

Apache License 2.0, the license of the base model (google/embeddinggemma-2, by Google DeepMind). This repository contains a modified version of that model; see NOTICE for the attribution and a summary of the modifications. The build pipeline in pipeline/ is by Påmin Labs under the same license.

Citation

If you use this conversion, please cite the base model and this repository. The upstream card does not provide a BibTeX entry at the time of writing; cite it by its model card.

@misc{embeddinggemma2,
  title        = {{EmbeddingGemma 2}},
  author       = {{Google DeepMind}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/google/embeddinggemma-2}}
}

@misc{pamin2026embeddinggemma2int8onnx,
  title        = {{EmbeddingGemma 2 text encoder, int8 ONNX for CPU}},
  author       = {{P{\aa}min Labs}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/pamin-labs/embeddinggemma-2-text-int8-onnx}},
  note         = {Batch-invariant int8 ONNX conversion of google/embeddinggemma-2}
}

Contact

Maintained by Påmin Labs. Questions and problem reports go to the discussions of this repository or to GitHub issues of Påmin Memory.