EmbeddingGemma-2-GGUF

Community GGUF quantizations of google/embeddinggemma-2.

About EmbeddingGemma 2

EmbeddingGemma 2 is a multilingual, multimodal embedding model from Google DeepMind. It maps text and code, images, video, and audio into a shared 768-dimensional vector space and has an 8,192-token shared context window. The upstream checkpoint has a 270M-parameter text path plus optional vision and audio encoders. This release contains a quantized text backbone and a shared BF16 vision/audio projector. Text, image, and audio inputs returned normalized 768-dimensional vectors in local llama-server CPU smoke checks. See the official model card for supported task prefixes, modalities, and input guidance.

Embedding evaluation

All values below were measured locally from the locked BF16 GGUF and quantized artifacts; they are not copied from the upstream model card. Additional quant formats are being added incrementally and will appear in the comparison table after running this same hold-out evaluation. The fixed hold-out evaluation uses the 1,379-pair test split of MTEB STSBenchmark STS, dataset revision 96943a16ea6a35129e253c659081cb59daf81b30. It covers 2,552 unique sentences with the task: sentence similarity | query: prefix, an 8,192-token context, and the CPU llama.cpp runtime at commit 9c2e0e491a822adae1f0b1c831adb4160057d24f. Spearman and Pearson measure correlation between cosine similarity and the human scores. Mean and 5th-percentile cosine measure vector agreement with the BF16 GGUF reference. Higher values indicate stronger task correlation or closer vector agreement; smaller files use less disk. This is a task-specific embedding evaluation, not the full MTEB benchmark. Token-level KLD, next-token Top-1, PPL, and RMS probability deltas do not apply to this embedding model.

BF16 reference baseline

Quantized variants compared with BF16

Q5_K_M and Q4_K_M are memory/fidelity candidates from this measured size and task-correlation trade-off; no VRAM fit or throughput claim is made. Q1_0 loads and returns embeddings, but its near-zero STS correlation and much lower BF16 vector agreement indicate severe quality loss on this task. The machine-readable evaluation summary and reproducibility manifest include the full measurements and method. The initial Q8_0 artifact and BF16 vision/audio projector also passed local text, image, and audio embedding smoke checks.

Quick start

Start an embedding endpoint with llama.cpp:

llama-server \
  -m EmbeddingGemma-2-Q8_0.gguf \
  --mmproj mmproj-EmbeddingGemma-2-BF16.gguf \
  --embedding \
  --pooling mean \
  --ctx-size 8192 \
  --port 8080

Send an English search query using the model's task prefix:

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"EmbeddingGemma-2","input":"task: search result | query: What causes the northern lights?"}'

For text-only workloads, omit --mmproj. For documents, use title: {title} | text: {content} or title: none | text: {content} when there is no title. Keep query and document vectors at the same dimension. The server returns normalized vectors by default.

Reproducibility and validation

See the reproducibility manifest, artifact measurements, and SHA256 checksums. The manifest records the locked upstream revision and model hash, BF16 GGUF source, converter/runtime revision, model-specific calibration/imatrix, quantization commands, and smoke profile. Raw conversion, imatrix, quantization, and validation logs are kept locally and are not part of this repository.

License and attribution

The upstream model is licensed under Apache 2.0. These are community GGUF quantizations, not an official Google or Google DeepMind release or endorsement.