Building open-domain chatbots is a challenging area for machine learning research.
Model Card
By AI at Meta, published under apache-2.0, revision bbf60f5f68fd.
Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to their partners, both asking and answering questions, and displaying knowledge, empathy and personality appropriately, depending on the situation. We show that large scale models can learn these skills when given appropriate training…
Read AI at Meta's full model card
Model description
Abstract
Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to their partners, both asking and answering questions, and displaying knowledge, empathy and personality appropriately, depending on the situation. We show that large scale models can learn these skills when given appropriate training data and choice of generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter neural models, and make our models and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing failure cases of our models.
Configuration
- Architecture
- BlenderbotSmallForConditionalGeneration
- Context length (tokens)
- 512
- Layers
- 8
- Vocabulary size
- 54,944
- Model type
- blenderbot-small
Identity and Version
- Repository
- facebook/blenderbot_small-90M
- Publisher
- AI at Meta
- Task
- Not stated by the source
- Modality
- Other
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- en
- Revision
- bbf60f5f68fd8789ac04bd1c20712233f3dc899f
- First published
- 2022-03-02
- Last updated
- 2024-02-29
Files and Weights
11 files, 1.1 GB in total. The weights are 3 files totalling 1.1 GB in bin, h5, msgpack.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| flax_model.msgpack | Weights | 350.3 MB | e33ca6a0da27 |
| pytorch_model.bin | Weights | 350.4 MB | 9f1a4ebe4627 |
| tf_model.h5 | Weights | 350.4 MB | 92d6158f02e4 |
| config.json | Configuration | 1.5 KB | — |
| generation_config.json | Configuration | 311 B | — |
| special_tokens_map.json | Configuration | 99 B | — |
| README.md | Documentation | 1.5 KB | — |
| .gitattributes | Repository | 391 B | — |
| merges.txt | Tokenizer | 344.6 KB | — |
| tokenizer_config.json | Tokenizer | 230 B | — |
| vocab.json | Tokenizer | 963.5 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 1.1 GB
Released by AI at Meta through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:1907.06616
- Trained on (disclosed) blended_skill_talk
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 1.1 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About blenderbot_small-90M
Can I use blenderbot_small-90M commercially?
Yes. blenderbot_small-90M is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is blenderbot_small-90M's context length?
512 tokens, from the maximum position embeddings in its published configuration.