GPT-2 small, instruction-tuned on SmolTalk
The released GPT-2 small weights (124M parameters) fine-tuned
into a chat model on
smol-smoltalk,
with the loss on assistant replies only. Small, medium and large versions are
compared in the instruction-tuning write-up.
Results
|
|
| Validation loss (assistant tokens, 1,000 test conversations) |
1.446 |
| Training |
one epoch, 460,341 conversations, 14,386 steps of 32 |
| Batch × accumulation, peak learning rate |
2 × 16, 5e-5 |
| Hardware and time |
one RTX 2080 in fp32, 14.2 hours |
What is the capital of France?
The capital of France is Saint-Ouen-sur-Mer in Belgium.
Files
best.safetensors: the weights at the lowest validation loss (the final step).config.json: the training config.
Loading
These weights use the GPT2 class from gpt2-repro, not Hugging Face
Transformers, and a vocabulary of 50,263 tokens: GPT-2's plus six chat
tokens. Clone gpt2-repro and install its dependencies with
uv, then run the code below from the repository
root:
git clone https://github.com/rahuliyer/gpt2-repro
cd gpt2-repro
uv sync
from huggingface_hub import hf_hub_download
from inference import generate_text
from instruct import get_tokenizer
from instruct.chat import encode_prompt
from instruct.tokenizer import ASSISTANT_END_TOKEN
from model import GPT2, GPT2SmallConfig
tokenizer = get_tokenizer()
path = hf_hub_download("rahuliyer/gpt2-small-smoltalk", "best.safetensors")
model = GPT2.load_from_checkpoint(GPT2SmallConfig(vocab_size=tokenizer.n_vocab), path)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
]
prompt = encode_prompt(tokenizer, messages)
text = generate_text(
model, tokenizer, prompt, max_length=256,
stop_token_id=tokenizer.encode_single_token(ASSISTANT_END_TOKEN),
)
print(text[len(tokenizer.decode(prompt)):])
generate_text samples with temperature 1 and top-5 by default.
Chat format
<|system-start|>...<|system-end|><|user-start|>...<|user-end|><|assistant-start|>
The system message is optional. The model ends its reply with
<|assistant-end|>. Conversations were truncated to 1,024 tokens in training.
Limitations
It learned the chat format but often gets facts wrong, as above. English only. Not suitable for anything that relies on accurate
or safe output.
