Pihu Pandey

AstralLM is a compact, efficient causal language model built for Hugging Face Transformers. It features a clean decoder-only transformer architecture with Grouped Query Attention (GQA), SwiGLU feed-forward networks, RoPE positional embeddings, and per-head QK-norm — delivering strong performance at a small parameter count (~138M). AstralLM is a decoder-only transformer with the following design choices: The model uses 8 query heads and 4 key/value heads (2:1 ratio), halving the KV cache memory footprint during inference without measurable quality degradation. Each head operates over a head dimension of 80. Per-head RMSNorm is applied to both query and key projections before the rotary…