A 69M parameter causal language model built on the Mixture-of-Attentions (MoA) architecture — distance-based metric attention that respects the triangle inequality by construction, not approximation. Every attention head operates in a proper metric space. The geometry is enforced, not hoped for. Standard transformers compute attention as a dot product: Q·Kᵀ. This has no geometric meaning — it's a bilinear form, not a distance. Two tokens can be "close" by dot product while violating basic metric properties. MoA replaces this with negative squared distance under a learned diagonal Mahalanobis metric, then enforces the triangle inequality through a regularizer over random triples sampled…
Open weights
cc
69M parameters
1,024 tokens
transformers
A 70M parameter causal language model built on the Mixture-of-Attentions (MoA) architecture — distance-based metric attention that respects the triangle inequality by construction, not approximation. Every attention head operates in a proper metric space. The geometry is enforced, not hoped for. Standard transformers compute attention as a dot product: Q·Kᵀ. This has no geometric meaning — it's a bilinear form, not a distance. Two tokens can be "close" by dot product while violating basic metric properties. MoA replaces this with negative squared distance under a learned diagonal Mahalanobis metric, then enforces the triangle inequality through a regularizer over random triples sampled…
Open weights
cc
71M parameters
2,048 tokens
transformers
Model · Text generation
AobanZ
Lightning is a small, autoregressive transformer which utilizes FlashAttention and MHA. This model is trained on a variety of books from a dataset(300 MB). This is the expanded version of Lightning-30m. Lightning utilizes FlashAttention and AdamW for performance and capability. Lightning is designed to provide quick, coherent outputs, improved with a larger size and weight. Lightning is intended to be used for research, analysis and fine-tuning, stories and other. It is not intended to be used for professional advice, real writing or any kind of heavy work as generated outputs may be incorrect. Lightning can be used directly for text generation, experimentation, and conversational…
Open weights
mit
65M parameters
transformers
Model · Text generation
Jiwubu
minimind-math-v3 是一个基于 MiniMind 项目从零训练的小型中文数学语言模型，权重已转换为 Qwen3 兼容格式，可直接使用 HuggingFace transformers 加载推理。 - 架构：Dense Transformer（Decoder-only），未使用 MoE - 数据：以数学/推理语料为主的预训练数据（pretraint2tmini.jsonl），并经过数学 SFT 训练 - Tokenizer：基于 MiniMind 自训 tokenizer 修改的 patched tokenizer，详见下方说明 本模型的预训练与 SFT 均基于 jiwubu/minimind 代码仓库完成。相对上游 jingyaogong/minimind 的主要修改是新增 math/ 目录，用于生成数学 SFT 用的加减乘除算术数据（genmathdataaddsub.py、genmathdatamuldiv.py 等），并提供配套评测脚本（evalmath.py）。 最简单的方式是用仓库自带的交互式验证脚本（chat template + 贪心解码 + 流式输出）： 或直接用 transformers： - 预训练 2 个 epoch，batch size 32，最大序列长度 340 - 优化器 AdamW，初始学习率 5e-4（带 warmup + 余弦衰减） - 混合精度：bfloat16 本模型使用基于 MiniMind 原版 tokenizer 修改的 patched tokenizer（Sequence[Digits, ByteLevel]）：…
Open weights
apache-2.0
64M parameters
32,768 tokens
transformers
cRia-LM-75M is a 75.7M-parameter base language model built as a Relaxed Recursive Transformer (RRT). It uses a shared 11-layer recurrent block evaluated twice, with pass-specific LoRA parameters on the second traversal. Training was carried out in three stages. Stage 1 established the 2K base model over 10B tokens. Stage 2 continued training with a 2B-token budget and a capability-focused data curriculum; the released Stage 2 checkpoint is step 10,000, corresponding to about 1.31B continuation tokens. Stage 3 extended the context window from 2,048 to 4,096 tokens with a 50M-token run on codelion/sutra-1B. The released checkpoint continues from that long-context stage with 35 learned…
Open weights
apache-2.0
76M parameters
4,096 tokens
transformers
cRia-LM-75M-Instruct is a 75.7M-parameter instruction-tuned language model built as a Relaxed Recursive Transformer (RRT). It uses a shared 11-layer recurrent block evaluated twice, with pass-specific LoRA parameters on the second traversal. The model starts from cRia-LM-75M, then adds supervised instruction tuning and preference optimization. It ships with a native chat template using and markers. cRia-LM-75M-Instruct keeps the base model's 13 unique Transformer layers: src="https://hfviewer.com/api/card.svg?source=sz14%2FcRia-LM-75M-Instruct&granularity=auto&v=20260516-title-pills-card" alt="Architecture graph for sz14/cRia-LM-75M-Instruct. Open in hfviewer" width="100%" This gives an…
Open weights
apache-2.0
76M parameters
4,096 tokens
transformers
