Fabryka AI

Model · Text generation

gollem-v5-ckpts

Fabryka AI

Research checkpoints of sub-100M-parameter English language models, GPT-style decoders (nanoGPT lineage) trained for the This repository is a controlled single-factor scaling study: identical architecture/hyper-parameters/seed, varying only tokens and model width. † crown = expanded 8.29B-token corpus (~1.9 epochs); board-recon #14/74 (up from #18 via ARC). These are raw nanoGPT-lineage checkpoints (plain torch state dicts), not transformers AutoModel weights. The model class and a ready board-scoring harness are included in this repo: - traingptref.py — GPT definition (rebuild the GPT of the tabled shape, loadstatedict, trim logits to vocab 12288). - glintparityeval.py — the exact Glint…

Dataset

minimal-en-corpus-5b

Fabryka AI

An English-language pretraining corpus prepared for controlled experiments with approximately 125M-parameter GPT-2 models based on karpathy/nanoGPT. The name refers to the approximately 5B-token mixture selected before final BPE tokenization. With the included 12,288-token BPE tokenizer, the packaged nanoGPT training split contains 5,396,605,407 tokens. The dataset provides both reusable source text and ready-to-train nanoGPT binaries. - documentid: string - sourceid: string - text: largestring - 12,029 merges - 256 byte tokens - 3 reserved special tokens -: ID 12,285 -: ID 12,286 -: ID 12,287 - One separator appended after every packaged document The tokenizer was trained on a…