thor-lm-1b

THOR-LM-1B: Frontier Language Model Pre-Trained with THOR

THOR-LM-1B is a ~915 Million parameter modern autoregressive transformer pre-trained from scratch on high-purity FineWeb-Edu tokens, optimized exclusively with THOR (Tempered High-velocity Orthogonal Representation).

Model Architecture

• Parameters: 914,696,192 (~1.0B parameters)
• Layers: 18 Transformer Decoder Blocks
• Hidden Dimension ($d_{\text{model}}$): 2,048
• Attention: Grouped Query Attention (GQA, 16 Query Heads, 4 KV Heads)
• Feed-Forward: SwiGLU Gated MLP ($d_{\text{ff}} = 5,632$)
• Norm: Pre-RMSNorm ($\epsilon = 10^{-6}$)
• Positional Encoding: Rotary Position Embeddings (RoPE, $\theta = 10,000$)
• Context Length: 2,048 tokens
• Precision: bfloat16

The Optimizer: THOR

Unlike traditional language models trained with AdamW or Muon, THOR-LM-1B was optimized with THOR: - Polar Orthogonal Spectral Descent: Condition number $\kappa = 1$ maintained across all 2D weight matrices via quintic Newton-Schulz iteration. - Calibrated Noise Floor Dynamic Braking: Automatically moderates step sizes during high stochastic noise via the theorem $\rho_{\text{floor}} = \sqrt{\frac{1-\beta}{1+\beta}} \approx 0.1601$. - Lie Manifold Commutator Torque: $\Omega = [W W^T, U]$ eliminates orthogonal representation drift across curved neural manifolds.

Quickstart & Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Or load directly using the AirBorne codebase:
from airborne import thor

To train your own models with THOR:

pip install airborne-thor

from airborne import thor

optimizer = thor(model.parameters(), lr=1e-3, weight_decay=0.1)

Citation & Credits

Engineered by Suryaansh Prithvijit Singh // ULTRON & AirBorne AI Research. GitHub Repository: https://github.com/AirBorneAI/airborne-thor