Exported to ONNX from perplexity-ai/pplx-embed-v2-late-0.6b (by Perplexity, MIT license). This is an unofficial conversion; all credit for the model goes to the original authors.
The model is a late-interaction (multi-vector) retriever: it outputs one L2-normalized 128-d vector per token or merged image patch, and documents are scored against queries with MaxSim.
Files (onnx/)
embed_tokens: input_ids -> inputs_embedsvision_encoder: pixel_values, grid_thw (one image per call) -> image embeddings that replace the image tokenstext_model: inputs_embeds, attention_mask, position_ids [3,B,T] -> token_embeddings [B,T,128]
| precision |
suffix |
size (3 files) |
| fp32 |
(none) |
2.38 GB |
| fp16 |
_fp16 |
1.19 GB |
| q4 |
_q4 |
0.38 GB |
Not included:
- q8: retrieval dropped more than 0.02 (Recall@1 -0.10, MRR -0.084).
Browser runtime: the fp32, fp16, q4 files loaded and ran with onnxruntime-web 1.30.0 (wasm backend), tested in Node.js. WebGPU was not tested.
Prompts
- Query:
[Q] {text}
- Text document:
[D] {text}
- Image document:
[D] <|vision_start|>{image}<|vision_end|> ({image} = the image token repeated once per merged patch)
Encode text and images in separate calls. For documents, drop tokens listed in pplx_onnx_config.json skiplist_token_ids; queries keep every token.
Usage (MaxSim)
from PIL import Image
from onnx_inference import PplxLateONNX, maxsim
m = PplxLateONNX(".", precision="fp32") # fp32 | fp16 | q4
q = m.encode_query("a red bicycle") # (n_query_tokens, 128)
d = m.encode_image(Image.open("photo.jpg")) # (n_image_tokens, 128)
score = maxsim(q, d) # sum over query tokens of the max dot product over document tokens
onnx_inference.py (in this repo) does the tokenization, image preprocessing, positions and MaxSim with numpy + onnxruntime.
Parity with the PyTorch model
Measured on CPU with onnxruntime against Sentence Transformers (PyTorch, CPU) on 4 queries and 4 photos: per-token cosine between ONNX and PyTorch vectors.
| precision |
query cos mean / min |
image cos mean / min |
MaxSim top-1 agreement |
| fp32 |
1.0000 / 1.0000 |
1.0000 / 1.0000 |
4/4 |
| fp16 * |
1.0000 / 1.0000 |
1.0000 / 0.9999 |
4/4 |
| q4 |
0.9979 / 0.9964 |
0.9974 / 0.9422 |
4/4 |
* fp16 was compared with the fp32 ONNX files on the same 4 queries and 4 photos (the fp32 files match PyTorch at cosine 1.0000 above).
Retrieval check
Tested by encoding 1,000 Flickr8k images and 100 of their captions (each caption's own image is the target) and comparing MaxSim retrieval with the PyTorch model. ONNX ran with onnxruntime on CPU. The PyTorch reference is Sentence Transformers MultiVectorEncoder with bf16 compute on a GPU, on the same images and queries.
| precision |
images |
Recall@1 |
Recall@5 |
Recall@10 |
MRR |
PyTorch Recall@1 / MRR (same images) |
pass |
| fp32 |
1000 |
0.74 |
0.91 |
0.97 |
0.821 |
0.72 / 0.809 |
yes |
| fp16 |
1000 |
0.75 |
0.91 |
0.97 |
0.824 |
0.72 / 0.809 |
yes |
| q8 |
1000 |
0.62 |
0.85 |
0.90 |
0.725 |
0.72 / 0.809 |
no |
| q4 |
1000 |
0.76 |
0.92 |
0.96 |
0.825 |
0.72 / 0.809 |
yes |
Pass rule: Recall@1 and MRR no more than 0.02 below PyTorch on the same images; fp32 and fp16 also need a per-token cosine of at least 0.99 in the parity check. With 100 queries, differences of a few points in either direction are within noise.
Notes
- The Qwen3.5 linear-attention layers were rewritten with standard ops so they could be exported (scan mode:
scan). Only full-sequence encoding is supported, which is all an embedding model needs.
- fp16: weights are fp16; inputs and outputs stay fp32. The norms, rotary embeddings and the linear-attention recurrence run in fp32, because squaring hidden values in fp16 overflows.
- transformers.js has no class for this architecture; use onnxruntime directly.
- Export script:
export_scripts/. Full numbers: export_report.json, functional_report.json.
License
MIT, same as perplexity-ai/pplx-embed-v2-late-0.6b.
