Vela-1.0-Omni-Mini

Vela Omni Mini

Vela Omni Mini maps text, images, and speech into a shared embedding space for multimodal search, routing, and clustering.

1044M parameters · 768 dimensions · L2-normalized embeddings.

Try it in Vela Studio.

Evaluation

Scores use a 0–100 scale; higher is better. All applicable models use the same examples and retrieval pools. N/A denotes a modality the text-only model does not support. Bold Vela scores improve on multi-modal-embed-large.

Macro-F1 gives equal weight to every intent class (77 for Banking77 and 60 for MASSIVE), complementing the query-weighted accuracy; undefined class F1 is zero. Text evaluation uses fixed class prototypes: 3,080 Banking77 and 2,972 MASSIVE English queries. Vela Omni is adapted using training examples and intent labels from these two datasets; comparison models are evaluated as published without additional adaptation here. These scores measure supervised task adaptation with nearest-prototype classification, not zero-shot transfer or the MTEB classification protocol.

Image retrieval uses the COCO Karpathy CC-BY2 image subset: 823 images and 4,115 captions. Speech retrieval uses 2,611 LibriSpeech test-clean clips and 2,610 unique transcripts, with clips limited to 30 seconds. Recall is measured against each complete candidate pool, accepting all matching positives.

Evaluation uses FP32 embeddings, cosine similarity, and a common 128-token text limit. Full scores and protocol include model revisions, sample counts, and paired uncertainty against the original and previous models. These fixed evaluation pools are reused across releases.

Standard benchmarks

The following selected tasks use MTEB 2.21.0, frozen encoders and each task's official evaluation protocol. They cover text similarity, retrieval, classification and clustering, plus image and audio classification, clustering and zero-shot label matching. Scores are multiplied by 100; higher is better. A dash means not measured here. These results do not represent a complete MTEB, MMTEB, MIEB or MAEB score. Bold Mini scores improve on the previous Mini release.

Evaluation uses FP32, batch size 8 and seed 20260918. Text limits are 512 tokens for Nano, 32,768 for Mini and 128 for the separate Embedding reference. The benchmark adapter truncates longer inputs: on ArguAna this affects 222 of 10,080 Nano inputs and none for Mini. Text, image and audio classification use task-specific labeled probes; zero-shot tasks compare media with fixed text labels. CREMA-D classification uses the official five-fold protocol on its named train split, with the encoder frozen. Official clustering subsampling is preserved.

Full metrics and protocol include dataset and model revisions, evaluation counts, repeated-probe results, classification F1/precision/recall, retrieval metrics and clustering statistics. MTOP validation results are included there separately. These measurements show that fine-grained image label matching and audio emotion recognition need stronger alignment; speech retrieval scores should not be read as general audio understanding.

Audio benchmarks

Complete MAEB(beta) audio-only panel: 19 tasks and 134 subset/split combinations, evaluated with MTEB 2.21.0.

Scores ×100; bold compares Nano and Mini, not a previous release. Each group follows the official mean of task scores. Classification includes official supervised probes; joint text/audio tasks are excluded.

All 19 task metrics and exact evaluated revisions. The current models retain the evaluated audio weights and processing.

Quick start

Use PyTorch, Transformers 4.57.6, Hugging Face Hub, safetensors, NumPy, and Pillow:

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("llm-semantic-router/Vela-1.0-Omni-Mini")
sys.path.insert(0, path)
from vela_omni import VelaOmni

model = VelaOmni.from_pretrained(path, device="cpu")
vectors = model.encode_text(["A bicycle beside a building.", "Someone is reading aloud."])
print(vectors.shape)  # (2, 768)

Pass a list of Pillow images to model.encode_image(images). Pass a list of mono NumPy waveforms to model.encode_audio(waveforms, sampling_rate=16000); each waveform must be at most 30 seconds. Compare normalized vectors with their dot product. To route media, embed each destination’s name and description, then select the closest vector. To discover routing categories, cluster media embeddings and inspect each group. Similarity scores are rankings, not calibrated probabilities.

Cross-modal alignment uses COCO image–caption pairs from the CC-BY 2.0 image subset and LibriSpeech speech–transcript pairs (CC-BY 4.0). COCO annotations, Banking77, and MASSIVE are provided under CC-BY 4.0.

The repository includes the native model code, component configurations, and tokenizer and processor files. See NOTICE and LICENSE for component attribution and license terms.

Explore the Vela model collection