Skip to page content
SAVRN
Search Plan a project

Open-weight model · Feature extraction

w2v-bert-2.0

by AI at Meta facebook/w2v-bert-2.0

w2v-bert-2.0 is an open-weight model for feature extraction from AI at Meta, released under MIT License. It has 580M parameters. At 16-bit it needs about 1.4 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 1.1M downloads a month.

We are open-sourcing our Conformer-based W2v-BERT 2.0 speech encoder as described in Section 3.2.1 of the paper, which is at the core of our Seamless models. This model was pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages.

Parameters580M
Context—
Weights4.7 GB
Licensemit
AccessOpen weights
Monthly Downloads1.1M

Runs On

What it takes to serve w2v-bert-2.0 (580M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 1.2 GB 1.4 GB 1x MI300X (192 GB)
Vultr
$1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 0.6 GB 0.7 GB 1x MI300X (192 GB)
Vultr
$1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 0.3 GB 0.3 GB 1x MI300X (192 GB)
Vultr
$1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 16, 2026.

w2v-bert-2.0 on every accelerator the SAVRN Index prices, at every precision

SAVRN's Notes on w2v-bert-2.0

580M parameters producing speech representations rather than transcripts: the page classifies it as feature extraction, the front end you fine-tune or probe for downstream speech tasks. Published weights are 4.7 GB of float32 in pt and safetensors — 1.4 GB at 16-bit, 0.7 GB at 8-bit, 0.3 GB at 4-bit — with a cheapest listed setup of one MI300X at $1.85 an hour. Downloads run 1.8M a month.

MIT permits commercial use with notices kept. It is the multilingual w2v-BERT generation described by arXiv:2312.05187, its disclosed language list opening at Afrikaans, Amharic, Arabic, Assamese and Azerbaijani. The real question is whether you still want a feature extractor: end-to-end recognizers like Qwen3-ASR-1.7B on this list cover most transcription needs, leaving w2v-bert-2.0 for custom head training and speech research. Validate on your own data either way.

Model Card

By AI at Meta, published under mit, revision da985ba0987f.

W2v-BERT 2.0 speech encoder

We are open-sourcing our Conformer-based W2v-BERT 2.0 speech encoder as described in Section 3.2.1 of the paper, which is at the core of our Seamless models.

This model was pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages. It requires finetuning to be used for downstream tasks such as Automatic Speech Recognition (ASR), or Audio Classification.

Model Name #params checkpoint
W2v-BERT 2.0 600M checkpoint

This model and its training are supported by Transformers, more on it in thedocs.

Transformers usage

This is a bare checkpoint without any modeling head, and thus requires finetuning to be used for downstream tasks such as ASR. You can however use it to extract audio embeddings from the top layer with this code snippet:

Read the full model card (322 words)

Configuration

Architecture
Wav2Vec2BertModel
Layers
24
Hidden size
1,024
Feed-forward size
4,096
Attention heads
16
Stored precision
float32
Model type
wav2vec2-bert

Identity and Version

Repository
facebook/w2v-bert-2.0
Publisher
AI at Meta
Task
Feature extraction
Modality
Text
Library
transformers
Parameters
580M parameters
Languages
af, am, ar, as, az, be, bn, bs
Revision
da985ba0987f70aaeb84a80f2851cfac8c697a7b
First published
2023-12-19
Last updated
2024-01-25

Files and Weights

6 files, 4.7 GB in total. The weights are 2 files totalling 4.7 GB in pt, safetensors.

Weights2 files · 4.7 GB
Configuration2 files · 2.1 KB
Documentation1 file · 4.8 KB
Repository1 file · 1.5 KB
Every file
FileTypeSizeSHA-256
conformer_shaw.ptWeights2.3 GB 8310b4270a5b
model.safetensorsWeights2.3 GB eb890c9660ed
config.jsonConfiguration1.9 KB —
preprocessor_config.jsonConfiguration275 B —
README.mdDocumentation4.8 KB —
.gitattributesRepository1.5 KB —

License and Download

License
mit
Access
Open weights, no gate
Download size
4.7 GB
Download from AI at Meta

Released by AI at Meta through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published4.7 GB
16-bit1.2 GB
8-bit0.6 GB
4-bit0.3 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Compare w2v-bert-2.0

Questions About w2v-bert-2.0

How much GPU memory does w2v-bert-2.0 need?

About 1.4 GB at 16-bit and 0.3 GB at 4-bit: the weights (580M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run w2v-bert-2.0 on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use w2v-bert-2.0 commercially?

Yes. w2v-bert-2.0 is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.

Similar Models

Model · Feature extraction

d1-600m-text2sql

Jirameth

Fine-tuned from LiquidAI/d1-omni-600M to select one of four supplied SQL queries using a schema, question, and conversation history. Your application supplies and executes the queries. On 1,000 synthetic development turns, v1 scored 45.6% versus 24.9% for the unchanged base, a gain of 20.7 percentage points. Both used FP32, complete inputs, original query order, batch size 4, and highest-score selection. The set contains 250 four-turn conversations across 10 synthetic clinical task families excluded from training. Each turn receives a fixed supplied history, independent of earlier predictions. This development set was reused for model selection. Results do not establish independent-test or…

Open weights other 587M parameters 128,000 tokens transformers

Model · Feature extraction

jina-embeddings-v3

Jina AI

jina-embeddings-v3 is a multilingual multi-task text embedding model designed for a variety of NLP applications. Based on the Jina-XLM-RoBERTa architecture, this model supports Rotary Position Embeddings to handle long input sequences up to 8192 tokens. Additionally, it features 5 LoRA adapters to generate task-specific embeddings efficiently. - retrieval.query: Used for query embeddings in asymmetric retrieval tasks - retrieval.passage: Used for passage embeddings in asymmetric retrieval tasks - separation: Used for embeddings in clustering and re-ranking applications - classification: Used for embeddings in classification tasks - text-matching: Used for embeddings in tasks that quantify…

Open weights cc-by-nc-4.0 572M parameters 8,194 tokens transformers

Model · Feature extraction

Qwen3-Embedding-0.6B

Qwen

The Qwen3 Embedding model series is the latest proprietary model of the Qwen family, specifically designed for text embedding and ranking tasks. Building upon the dense foundational models of the Qwen3 series, it provides a comprehensive range of text embeddings and reranking models in various sizes (0.6B, 4B, and 8B). This series inherits the exceptional multilingual capabilities, long-text understanding, and reasoning skills of its foundational model. The Qwen3 Embedding series represents significant advancements in multiple text embedding and ranking tasks, including text retrieval, code retrieval, text classification, text clustering, and bitext mining. Exceptional Versatility: The…

Open weights apache-2.0 596M parameters 32,768 tokens sentence-transformers

We have updated the new reranker, supporting larger lengths, more languages, and achieving better performance. More details please refer to our Github: FlagEmbedding. FlagEmbedding focuses on retrieval-augmented LLMs, consisting of the following projects currently: - 3/18/2024: Release new rerankers, built upon powerful M3 and LLM (GEMMA and MiniCPM, not so large actually) backbones, supporitng multi-lingual processing and larger inputs, massive improvements of ranking performances on BEIR, C-MTEB/Retrieval, MIRACL, LlamaIndex Evaluation. - 3/18/2024: Release Visualized-BGE, equipping BGE with visual capabilities. Visualized-BGE can be utilized to generate embeddings for hybrid image-text…

Open weights mit 560M parameters 514 tokens transformers

Model · Feature extraction

multilingual-e5-large

Liang Wang

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024 This model has 24 layers and the embedding size is 1024. Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset. This model is initialized from xlm-roberta-large and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation. For all labeled datasets, we only use its training set for fine-tuning. For other training details, please refer to our paper at https://arxiv.org/pdf/2402.05672. Check out unilm/e5 to reproduce evaluation results on the BEIR and MTEB…

Open weights mit 560M parameters 514 tokens sentence-transformers

Model · Feature extraction

multilingual-e5-large-instruct

Liang Wang

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024 This model has 24 layers and the embedding size is 1024. Below are examples to encode queries and passages from the MS-MARCO passage ranking dataset. Usage with Infinity: This model is initialized from xlm-roberta-large and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation. First stage: contrastive pre-training with 1 billion weakly supervised text pairs. Check out unilm/e5 to reproduce evaluation results on the BEIR and MTEB benchmark. 1. Do I need to add instructions to the query? Yes, this…

Open weights mit 560M parameters 514 tokens sentence-transformers