finbert-crypto-stock-news-sentiment

FinBERT for crypto and stock news sentiment

Classifies a news headline as positive, negative or neutral for holders of the asset it is filed under. Fine-tuned from ProsusAI/finbert on headlines filed under Bitcoin, gold (the GLD fund) and US stocks (the SPY fund). Internal version name: finbert-tone-2.

It describes the tone of the news. It does not forecast prices.

Results

Scored once on a held-out test set of 854 headlines from 2026-04-01 to 2026-10-14, after every choice had been made on validation data. The test set holds more positive and negative headlines than the news feed does, so the figures are shown as drawn and also re-weighted to the feed's natural mix.

McNemar's exact test against the base model: p = 9e-77.

On headlines where both the label and the prediction are positive or negative, the model picks the same direction 99.0% of the time (296 headlines).

An earlier natural-mix test set of 826 headlines, already used to evaluate the previous version, gives accuracy 87.3% and macro-F1 0.814; because it was reused it is a secondary figure.

Training

• Method: LoRA r=16, lr=0.0001 with 10% linear warm-up, class-weighted cross-entropy, merged.
• Head: dense 768->256, ReLU, dropout 0.3, dense 256->3; set model.classifier to this before loading.
• Input: the headline, at most 128 tokens.
• Training window: 2015-01-02 to 2026-03-24; test window 2026-04-01 to 2026-10-14. Splits are in time order, never shuffled.
• Labels: written by language-model labellers (claude sonnet subagent, rubric v2, claude subagent, rubric v2) under a fixed written rubric, with agreement checks. The headlines themselves are not redistributed.
• Calibration: divide the logits by a temperature of 1.5 before the softmax.

Usage

The classifier head has two layers, which from_pretrained does not build by itself. Loaded the usual way, the head would be new random weights and the scores meaningless, so put the saved head back after loading:

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "HuangYiYang/finbert-crypto-stock-news-sentiment"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

state = load_file(hf_hub_download(repo, "model.safetensors"))
hidden, width = state["classifier.0.weight"].shape
head = torch.nn.Sequential(
    torch.nn.Linear(width, hidden), torch.nn.ReLU(), torch.nn.Dropout(0.0),
    torch.nn.Linear(hidden, state["classifier.3.weight"].shape[0]),
)
prefix = "classifier."
head_state = {k.removeprefix(prefix): v for k, v in state.items() if k.startswith(prefix)}
head.load_state_dict(head_state)
model.classifier = head
model.eval()

text = "Bitcoin plunges 15% as exchange hack wipes out billions"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    probs = (model(**inputs).logits / 1.5).softmax(-1)[0]
print({model.config.id2label[i]: round(float(p), 3) for i, p in enumerate(probs)})

Limitations

• Trained on three markets: Bitcoin, gold and US stocks. The test set covers Bitcoin and US stocks only: after removing headlines too similar to ones already labelled, gold had almost no headlines left in the test window, so gold has no separate test figure. Other assets and other kinds of text (articles, social posts) are untested.
• Much of the news filed under the gold fund is general market commentary, so it is labelled for the market it describes, not for gold.
• The base FinBERT was fine-tuned on Financial PhraseBank: full sentences about companies, labelled for investors. This model's inputs are short headlines, including crypto news and economic-data templates, labelled for holders of the asset they are filed under. The gap between the two is why the base model scores low here.
• English headlines only.
• Labels come from language-model labellers under a rubric, not from human annotators.
• The tone of a headline says nothing about where the price goes next.

Paper

A paper describing this model is in preparation; the arXiv link will be added here. Code: https://github.com/LouSens/RADAR.