FinBERT is a pre-trained NLP model to analyze sentiment of financial text. It is built by further training the BERT language model in the finance domain, using a large financial corpus and thereby fine-tuning it for financial sentiment classification. Financial PhraseBank by Malo et al. (2014) is used for fine-tuning. For more details, please see the paper FinBERT: Financial Sentiment Analysis with Pre-trained Language Models and our related blog post on Medium. The model will give softmax outputs for three labels: positive, negative or neutral. About Prosus Prosus is a global consumer internet group and one of the largest technology investors in the world. Operating and investing globally…
Open-weight model · Text classification
LinkBERT-large
by Michihiro Yasunaga michiyasunaga/LinkBERT-large
LinkBERT-large model pretrained on English Wikipedia articles along with hyperlink information. It is introduced in the paper LinkBERT: Pretraining Language Models with Document Links (ACL 2022). The code and data are available in this repository.
Parameters—
Context512
Weights1.3 GB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads2.1k
Model Card
By Michihiro Yasunaga, published under apache-2.0, revision d7bc175578ab.
LinkBERT-large model pretrained on English Wikipedia articles along with hyperlink information. It is introduced in the paper LinkBERT: Pretraining Language Models with Document Links (ACL 2022). The code and data are available in this repository. LinkBERT is a transformer encoder (BERT-like) model pretrained on a large corpus of documents. It is an improvement of BERT that newly captures document links such as hyperlinks and citation links to include knowledge that spans across multiple documents. Specifically, it was pretrained by feeding linked documents into the same language model context, besides a single document. LinkBERT can be used as a drop-in replacement for BERT. It achieves…
Read Michihiro Yasunaga's full model card
LinkBERT-large model pretrained on English Wikipedia articles along with hyperlink information. It is introduced in the paper LinkBERT: Pretraining Language Models with Document Links (ACL 2022). The code and data are available in this repository.
Model description
LinkBERT is a transformer encoder (BERT-like) model pretrained on a large corpus of documents. It is an improvement of BERT that newly captures document links such as hyperlinks and citation links to include knowledge that spans across multiple documents. Specifically, it was pretrained by feeding linked documents into the same language model context, besides a single document.
LinkBERT can be used as a drop-in replacement for BERT. It achieves better performance for general language understanding tasks (e.g. text classification), and is also particularly effective for knowledge-intensive tasks (e.g. question answering) and cross-document tasks (e.g. reading comprehension, document retrieval).
Intended uses & limitations
The model can be used by fine-tuning on a downstream task, such as question answering, sequence classification, and token classification. You can also use the raw model for feature extraction (i.e. obtaining embeddings for input text).
How to use
To use the model to get the features of a given text in PyTorch:
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('michiyasunaga/LinkBERT-large')
model = AutoModel.from_pretrained('michiyasunaga/LinkBERT-large')
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state
For fine-tuning, you can use this repository or follow any other BERT fine-tuning codebases.
Evaluation results
When fine-tuned on downstream tasks, LinkBERT achieves the following results.
General benchmarks (MRQA and GLUE):
| HotpotQA | TriviaQA | SearchQA | NaturalQ | NewsQA | SQuAD | GLUE | |
|---|---|---|---|---|---|---|---|
| F1 | F1 | F1 | F1 | F1 | F1 | Avg score | |
| BERT-base | 76.0 | 70.3 | 74.2 | 76.5 | 65.7 | 88.7 | 79.2 |
| LinkBERT-base | 78.2 | 73.9 | 76.8 | 78.3 | 69.3 | 90.1 | 79.6 |
| BERT-large | 78.1 | 73.7 | 78.3 | 79.0 | 70.9 | 91.1 | 80.7 |
| LinkBERT-large | 80.8 | 78.2 | 80.5 | 81.0 | 72.6 | 92.7 | 81.1 |
Citation
If you find LinkBERT useful in your project, please cite the following:
@InProceedings{yasunaga2022linkbert,
author = {Michihiro Yasunaga and Jure Leskovec and Percy Liang},
title = {LinkBERT: Pretraining Language Models with Document Links},
year = {2022},
booktitle = {Association for Computational Linguistics (ACL)},
}
Configuration
- Architecture
- BertModel
- Context length (tokens)
- 512
- Layers
- 24
- Hidden size
- 1,024
- Feed-forward size
- 4,096
- Attention heads
- 16
- Vocabulary size
- 28,996
- Model type
- bert
Identity and Version
- Repository
- michiyasunaga/LinkBERT-large
- Publisher
- Michihiro Yasunaga
- Task
- Text classification
- Modality
- Text
- Library
- transformers
- Parameters
- Not stated by the source
- Languages
- en
- Revision
- d7bc175578ab3361bd13908e01f342e2dfbdba7c
- First published
- 2022-03-08
- Last updated
- 2022-03-31
Files and Weights
8 files, 1.3 GB in total. The weights are 1 file totalling 1.3 GB in bin.
Weights1 file · 1.3 GB
Configuration2 files · 863 B
Tokenizer3 files · 649.6 KB
Documentation1 file · 3.5 KB
Repository1 file · 1.2 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| pytorch_model.bin | Weights | 1.3 GB | b3280d127791 |
| config.json | Configuration | 751 B | — |
| special_tokens_map.json | Configuration | 112 B | — |
| README.md | Documentation | 3.5 KB | — |
| .gitattributes | Repository | 1.2 KB | — |
| tokenizer.json | Tokenizer | 435.8 KB | — |
| tokenizer_config.json | Tokenizer | 321 B | — |
| vocab.txt | Tokenizer | 213.4 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 1.3 GB
Download from Michihiro Yasunaga
Released by Michihiro Yasunaga through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:2203.15827
- Trained on (disclosed) bookcorpus
- Trained on (disclosed) wikipedia
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 1.3 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About LinkBERT-large
Can I use LinkBERT-large commercially?
Yes. LinkBERT-large is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is LinkBERT-large's context length?
512 tokens, from the maximum position embeddings in its published configuration.
Similar Models
This is a RoBERTa-base model trained on ~124M tweets from January 2018 to December 2021, and finetuned for sentiment analysis with the TweetEval benchmark. The original Twitter-based RoBERTa model can be found here and the original reference paper is TweetEval. This model is suitable for English. 0 -> Negative; 1 -> Neutral; 2 -> Positive This sentiment analysis model has been integrated into TweetNLP. You can access the demo here.
https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2 with ONNX weights to be compatible with Transformers.js. If you haven't already, you can install the Transformers.js JavaScript library from NPM using: Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).
This is a multilingual XLM-roBERTa-base model trained on ~198M tweets and finetuned for sentiment analysis. The sentiment fine-tuning was done on 8 languages (Ar, En, Fr, De, Hi, It, Sp, Pt) but it can be used for more languages (see paper for details). This model has been integrated into the TweetNLP library.
FinBERT is a BERT model pre-trained on financial communication text. The purpose is to enhance financial NLP research and practice. It is trained on the following three financial communication corpus. The total corpora size is 4.9B tokens. More technical details on FinBERT: Click Link This released finbert-tone model is the FinBERT model fine-tuned on 10,000 manually annotated (positive, negative, neutral) sentences from analyst reports. This model achieves superior performance on financial tone analysis task. If you are simply interested in using FinBERT for financial tone analysis, give it a try. If you use the model in your academic work, please cite the following paper: Huang, Allen H.…
With this model, you can classify emotions in English text data. The model was trained on 6 diverse datasets (see Appendix below) and predicts Ekman's 6 basic emotions, plus a neutral class: 1) anger 2) disgust 3) fear 4) joy 5) neutral 6) sadness 7) surprise The model is a fine-tuned checkpoint of DistilRoBERTa-base. For a 'non-distilled' emotion model, please refer to the model card of the RoBERTa-large version. a) Run emotion model with 3 lines of code on single text example using Hugging Face's pipeline command on Google Colab: b) Run emotion model on multiple examples and full datasets (e.g.,.csv files) on Google Colab: Please reach out to [email protected] if you have any…