This model has been trained without supervision following the approach described in Towards Unsupervised Dense Information Retrieval with Contrastive Learning. The associated GitHub repository is available here https://github.com/facebookresearch/contriever.
SAVRN's Notes on contriever
Start with what the page does not say about AI at Meta's contriever: no parameter count, no task label, no license, no run-on setup. What it does say suffices: a 12-layer BERT-type model, hidden size 768, a 512-token window, trained by unsupervised contrastive learning for dense retrieval, shipped as 438 MB of pytorch weights at float32. Sentence embeddings over a document store is the use, and at that size any accelerator already in the rack holds it.
The blank license comes first: nothing on the page grants commercial use or redistribution, so get terms from the publisher's GitHub repository before shipping. Second, in transformers it returns no sentence embedding by itself; you add a mean pooling step, which belongs in your serving code and tests. Third, the last update on record is January 2022, so treat it as a fixed artifact that 8,071,813 downloads a month depend on.
Model Card
This model has been trained without supervision following the approach described in Towards Unsupervised Dense Information Retrieval with Contrastive Learning. The associated GitHub repository is available here https://github.com/facebookresearch/contriever. Using the model directly available in HuggingFace transformers requires to add a mean pooling operation to obtain a sentence embedding.
Excerpt from the card by AI at Meta.
Configuration
- Architecture
- Contriever
- Context length (tokens)
- 512
- Layers
- 12
- Hidden size
- 768
- Feed-forward size
- 3,072
- Attention heads
- 12
- Vocabulary size
- 30,522
- Stored precision
- float32
- Model type
- bert
Identity and Version
- Repository
- facebook/contriever
- Publisher
- AI at Meta
- Task
- Not stated by the source
- Modality
- Other
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- Not stated by the source
- Revision
- 2bd46a25019aeea091fd42d1f0fd4801675cf699
- First published
- 2022-03-02
- Last updated
- 2022-01-19
Files and Weights
8 files, 438.7 MB in total. The weights are 1 file totalling 438.0 MB in bin.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| pytorch_model.bin | Weights | 438.0 MB | d0b6e2516913 |
| config.json | Configuration | 619 B | — |
| special_tokens_map.json | Configuration | 112 B | — |
| README.md | Documentation | 1.4 KB | — |
| .gitattributes | Repository | 1.2 KB | — |
| tokenizer.json | Tokenizer | 466.1 KB | — |
| tokenizer_config.json | Tokenizer | 321 B | — |
| vocab.txt | Tokenizer | 231.5 KB | — |
License and Download
- License
- Not stated by the source
- Access
- Open weights, no gate
- Download size
- 438.0 MB
Released by AI at Meta through its official repository on Hugging Face.
Built From
- Described by arXiv:2112.09118
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 438.0 MB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About contriever
What is contriever's context length?
512 tokens, from the maximum position embeddings in its published configuration.