# w2v-bert-2.0 by AI at Meta: Open-Weight Model Source: https://savrn.com/models/w2v-bert-2-0 Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- ## Runs On What it takes to serve w2v-bert-2.0 (580M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them. | Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits | | --- | --- | --- | --- | --- | --- | | 16-bit | 1.2 GB | 1.4 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 8-bit | 0.6 GB | 0.7 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 4-bit | 0.3 GB | 0.3 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the [SAVRN Index](https://savrn.com/ai-index/pricing/gpus), read Oct 23, 2026. [w2v-bert-2.0 on every accelerator the SAVRN Index prices, at every precision](https://savrn.com/models/w2v-bert-2-0/gpus) ## SAVRN's Notes on w2v-bert-2.0 580M parameters producing speech representations rather than transcripts: the page classifies it as feature extraction, the front end you fine-tune or probe for downstream speech tasks. Published weights are 4.7 GB of float32 in pt and safetensors — 1.4 GB at 16-bit, 0.7 GB at 8-bit, 0.3 GB at 4-bit — with a cheapest listed setup of one MI300X at $1.85 an hour. Downloads run 1.8M a month. MIT permits commercial use with notices kept. It is the multilingual w2v-BERT generation described by arXiv:2312.05187, its disclosed language list opening at Afrikaans, Amharic, Arabic, Assamese and Azerbaijani. The real question is whether you still want a feature extractor: end-to-end recognizers like Qwen3-ASR-1.7B on this list cover most transcription needs, leaving w2v-bert-2.0 for custom head training and speech research. Validate on your own data either way. SAVRN Research ## Model Card By AI at Meta, published under mit, revision da985ba0987f. ### W2v-BERT 2.0 speech encoder We are open-sourcing our Conformer-based W2v-BERT 2.0 speech encoder as described in Section 3.2.1 of the [paper](https://savrn.com/papers/seamless-multilingual-expressive-and-streaming-speech-translation), which is at the core of our Seamless models. This model was pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages. It requires finetuning to be used for downstream tasks such as Automatic Speech Recognition (ASR), or Audio Classification. | Model Name | #params | checkpoint | | --- | --- | --- | | W2v-BERT 2.0 | 600M | [checkpoint](https://huggingface.co/reach-vb/conformer-shaw/resolve/main/conformer_shaw.pt) | This model and its training are supported by Transformers, more on it in the[docs](https://huggingface.co/docs/transformers/main/en/model_doc/wav2vec2-bert). ### Transformers usage This is a bare checkpoint without any modeling head, and thus requires finetuning to be used for downstream tasks such as ASR. You can however use it to extract audio embeddings from the top layer with this code snippet: [Read the full model card (322 words)](https://savrn.com/models/w2v-bert-2-0/card) ## Configuration Architecture Wav2Vec2BertModel Layers 24 Hidden size 1,024 Feed-forward size 4,096 Attention heads 16 Stored precision float32 Model type wav2vec2-bert ## Identity and Version Repository facebook/w2v-bert-2.0 Publisher AI at Meta Task Feature extraction Modality Text Library transformers Parameters 580M parameters Languages af, am, ar, as, az, be, bn, bs Revision da985ba0987f70aaeb84a80f2851cfac8c697a7b First published 2023-12-19 Last updated 2024-01-25 ## Files and Weights 6 files, 4.7 GB in total. The weights are 2 files totalling 4.7 GB in pt, safetensors. Weights2 files · 4.7 GB Configuration2 files · 2.1 KB Documentation1 file · 4.8 KB Repository1 file · 1.5 KB Every file | File | Type | Size | SHA-256 | | --- | --- | --- | --- | | conformer_shaw.pt | Weights | 2.3 GB | 8310b4270a5b | | model.safetensors | Weights | 2.3 GB | eb890c9660ed | | config.json | Configuration | 1.9 KB | — | | preprocessor_config.json | Configuration | 275 B | — | | README.md | Documentation | 4.8 KB | — | | .gitattributes | Repository | 1.5 KB | — | ## License and Download License mit Access Open weights, no gate Download size 4.7 GB [Download from AI at Meta](https://huggingface.co/facebook/w2v-bert-2.0) Released by AI at Meta through its official repository on Hugging Face. [Read the license](https://opensource.org/license/mit). ## Built From - Described by [arXiv:2312.05187](https://savrn.com/papers/seamless-multilingual-expressive-and-streaming-speech-translation) ## Memory Requirements | Precision | Weights in memory | | --- | --- | | As published | 4.7 GB | | 16-bit | 1.2 GB | | 8-bit | 0.6 GB | | 4-bit | 0.3 GB | Weights only, from the published parameter count; the key-value cache and runtime add to this. ## Compare w2v-bert-2.0 - [multilingual-e5-large vs w2v-bert-2.0](https://savrn.com/models/compare/multilingual-e5-large-vs-w2v-bert-2-0) - [Qwen3-Embedding-0.6B vs w2v-bert-2.0](https://savrn.com/models/compare/qwen3-embedding-0-6b-vs-w2v-bert-2-0) ## Questions About w2v-bert-2.0 ### How much GPU memory does w2v-bert-2.0 need? About 1.4 GB at 16-bit and 0.3 GB at 4-bit: the weights (580M parameters) plus a working margin. A long context needs more. ### What is the cheapest GPU to run w2v-bert-2.0 on? At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists. ### Can I use w2v-bert-2.0 commercially? Yes. w2v-bert-2.0 is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included. ## Similar Models Model · Feature extraction ### [d1-600m-text2sql](https://savrn.com/models/d1-600m-text2sql) [Jirameth](https://savrn.com/model-publishers/chulinz) Fine-tuned from LiquidAI/d1-omni-600M to select one of four supplied SQL queries using a schema, question, and conversation history. Your application supplies and executes the queries. On 1,000 synthetic development turns, v1 scored 45.6% versus 24.9% for the unchanged base, a gain of 20.7 percentage points. Both used FP32, complete inputs, original query order, batch size 4, and highest-score selection. The set contains 250 four-turn conversations across 10 synthetic clinical task families excluded from training. Each turn receives a fixed supplied history, independent of earlier predictions. This development set was reused for model selection. Results do not establish independent-test or… Open weights other 587M parameters 128,000 tokens transformers [View model](https://savrn.com/models/d1-600m-text2sql) Model · Feature extraction ### [jina-embeddings-v3](https://savrn.com/models/jina-embeddings-v3) [Jina AI](https://savrn.com/model-publishers/jinaai) jina-embeddings-v3 is a multilingual multi-task text embedding model designed for a variety of NLP applications. Based on the Jina-XLM-RoBERTa architecture, this model supports Rotary Position Embeddings to handle long input sequences up to 8192 tokens. Additionally, it features 5 LoRA adapters to generate task-specific embeddings efficiently. - retrieval.query: Used for query embeddings in asymmetric retrieval tasks - retrieval.passage: Used for passage embeddings in asymmetric retrieval tasks - separation: Used for embeddings in clustering and re-ranking applications - classification: Used for embeddings in classification tasks - text-matching: Used for embeddings in tasks that quantify… Open weights cc-by-nc-4.0 572M parameters 8,194 tokens transformers [View model](https://savrn.com/models/jina-embeddings-v3) Model · Feature extraction ### [Qwen3-Embedding-0.6B](https://savrn.com/models/qwen3-embedding-0-6b) [Qwen](https://savrn.com/model-publishers/qwen) The Qwen3 Embedding model series is the latest proprietary model of the Qwen family, specifically designed for text embedding and ranking tasks. Building upon the dense foundational models of the Qwen3 series, it provides a comprehensive range of text embeddings and reranking models in various sizes (0.6B, 4B, and 8B). This series inherits the exceptional multilingual capabilities, long-text understanding, and reasoning skills of its foundational model. The Qwen3 Embedding series represents significant advancements in multiple text embedding and ranking tasks, including text retrieval, code retrieval, text classification, text clustering, and bitext mining. Exceptional Versatility: The… Open weights apache-2.0 596M parameters 32,768 tokens sentence-transformers [View model](https://savrn.com/models/qwen3-embedding-0-6b) Model · Feature extraction ### [social-jev-0.6B-preview](https://savrn.com/models/social-jev-0-6b-preview) [Social Jev](https://savrn.com/model-publishers/social-jev) SocialJev is a family of classification foundation models for the social sciences, inspired by TypeSafe’s Jev. The models combine the flexibility of modern language models pretrained on trillions of tokens of text with the speed of lightweight NLP methods. Define your task through a simple typed interface and SocialJev returns typed outputs. Note: SocialJev is under active development. Interfaces may change between early releases. The model can be used directly through Transformers without installing the SocialJev package: Results are ordered first by document, then by query for that document. Each result provides.choice and.probabilities; the probabilities above are illustrative. The… Open weights 596M parameters transformers [View model](https://savrn.com/models/social-jev-0-6b-preview) Model · Feature extraction ### [bge-reranker-large](https://savrn.com/models/bge-reranker-large) [Beijing Academy of Artificial Intelligence](https://savrn.com/model-publishers/baai) We have updated the new reranker, supporting larger lengths, more languages, and achieving better performance. More details please refer to our Github: FlagEmbedding. FlagEmbedding focuses on retrieval-augmented LLMs, consisting of the following projects currently: - 3/18/2024: Release new rerankers, built upon powerful M3 and LLM (GEMMA and MiniCPM, not so large actually) backbones, supporitng multi-lingual processing and larger inputs, massive improvements of ranking performances on BEIR, C-MTEB/Retrieval, MIRACL, LlamaIndex Evaluation. - 3/18/2024: Release Visualized-BGE, equipping BGE with visual capabilities. Visualized-BGE can be utilized to generate embeddings for hybrid image-text… Open weights mit 560M parameters 514 tokens transformers [View model](https://savrn.com/models/bge-reranker-large) Model · Feature extraction ### [multilingual-e5-large](https://savrn.com/models/multilingual-e5-large) [Liang Wang](https://savrn.com/model-publishers/intfloat) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024 This model has 24 layers and the embedding size is 1024. Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset. This model is initialized from xlm-roberta-large and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation. For all labeled datasets, we only use its training set for fine-tuning. For other training details, please refer to our paper at https://arxiv.org/pdf/2402.05672. Check out unilm/e5 to reproduce evaluation results on the BEIR and MTEB… Open weights mit 560M parameters 514 tokens sentence-transformers [View model](https://savrn.com/models/multilingual-e5-large) ## AI at Meta [All models and datasets](https://savrn.com/model-publishers/facebook) ## Versions - [da985ba0987f](https://savrn.com/models/w2v-bert-2-0/versions/da985ba0987f) · current 2026-10-23 ## Explore More - [All feature extraction models](https://savrn.com/models/tasks/feature-extraction) - [All models under mit](https://savrn.com/models/licenses/mit) - [Model comparisons](https://savrn.com/models/comparisons) - [The model directory](https://savrn.com/models) - [Open model prices by host](https://savrn.com/ai-index/pricing/open-models) ## Source - Repository metadata, read 2026-10-19. - [AI at Meta website](https://ai.meta.com/resources/models-and-libraries/) - [Hugging Face record](https://huggingface.co/facebook/w2v-bert-2.0) - [How the hub is built](https://savrn.com/model-hub/methodology)