Nebium-Small (88.1M)
Nebium-Small is a 117-million-parameter causal Transformer trained for autoregressive next-chess-move prediction over Lichess UCI move sequences.
Nebium Model Family Architecture Overview
| Model |
Params |
d_model |
Heads |
Layers |
max_seq_len |
Chinchilla-optimal tokens |
| Nebium-Small |
117M |
768 |
12 |
12 |
1024 |
~2.3B |
| Nebium-Medium |
345M |
1024 |
16 |
24 |
1024 |
~6.9B |
| Nebium-Large |
762M |
1280 |
20 |
36 |
1024 |
~15.2B |
Architectural Primitives:
- Rotary Position Embeddings (RoPE) on attention query and key projections ($\theta = 10000$)
- SwiGLU feed-forward transformation
- RMSNorm pre-normalization
- Causal mask with padding token masking
- Byte-Pair Encoding (BPE) tokenizer trained on UCI move plies
Architectural Specifications
| Hyperparameter |
Value |
| Model Tier |
Nebium-Small |
| Parameter Count |
88.1M |
| Hidden Dimension ($d_{model}$) |
768 |
| Attention Heads ($n_{heads}$) |
12 |
| Transformer Layers ($n_{layers}$) |
12 |
| Max Context Length ($L_{max}$) |
1024 |
| Vocabulary Size ($V$) |
2018 |
| Positional Embedding |
rope |
| Activation Function |
swiglu |
| Layer Normalization |
rmsnorm |
Validation & Benchmark Results
| Metric |
Measured Value |
| Validation Loss |
2.727808 |
| Validation Perplexity |
15.2993 |
| Next-Token Top-1 Accuracy |
32.35% |
| Next-Token Top-5 Accuracy |
63.83% |
| Empirical Move Legality Rate |
100.00% |
| Tactical Puzzle Accuracy |
9.76% |
Scaling Law Analysis (Hoffmann et al. 2022)
Chinchilla power-law formulation:
$$L(N, D) = 1.69 + \frac{406.4}{N^{0.34}} + \frac{410.7}{D^{0.28}}$$
| Parameter / Metric |
Value |
| Model Parameters ($N$) |
88.1M |
| Chinchilla-Optimal Token Budget ($D^*$) |
~2.3B tokens |
| Compute-Optimal Expected Loss ($L_{optimal}$) |
3.4809 nats |
| Approximate Trained Tokens ($D$) |
~12.3M tokens |
| Theoretical Loss at Current Tokens |
6.7517 nats |
| Empirical Validation Loss |
2.7278 nats |
Python Usage Example
import json
import torch
from src.models.transformer.nebium import Nebium
from src.data.tokenizer import ChessTokenizer
tokenizer = ChessTokenizer()
tokenizer.load("tokenizer.json")
with open("model_config.json", "r", encoding="utf-8") as f:
config = json.load(f)
model = Nebium(**config)
state_dict = torch.load("model.pt", map_location="cpu", weights_only=True)
model.load_state_dict(state_dict)
model.eval()
prompt = "e2e4 e7e5 g1f3"
input_ids = torch.tensor([[tokenizer.bos_id] + tokenizer.encode(prompt)], dtype=torch.long)
attention_mask = torch.ones_like(input_ids)
with torch.no_grad():
output = model.generate(input_ids, attention_mask, max_new_tokens=10, temperature=0.7)
print("Continuation:", tokenizer.decode(output[0].tolist()))
License
MIT License.
