NeoLLM is an open-weight model from Kitsun, released under Apache License 2.0. It has 86M parameters and a 512-token context. At 16-bit it needs about 0.2 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 343 downloads a month.
NeoLLM is a 85.50 M parameter decoder-only language model trained from scratch on FineWeb-Edu with BF16 compute, completing training in approximately ~1h 16m on NVIDIA GeForce RTX 5090.
Runs On
What it takes to serve NeoLLM (86M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 0.2 GB | 0.2 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 0.1 GB | 0.1 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 0.0 GB | 0.1 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 7, 2026.
NeoLLM on every accelerator the SAVRN Index prices, at every precision
Model Card
By Kitsun, published under apache-2.0, revision 7d6c1c8474a3.
NeoLLM is a 85.50 M parameter decoder-only language model trained from scratch on FineWeb-Edu with BF16 compute, completing training in approximately ~1h 16m on NVIDIA GeForce RTX 5090. It integrates a collection of recently published attention and normalization techniques into a single architecture, with the goal of studying how they interact during pretraining. The model is actively being developed and the current checkpoint represents an intermediate training state.
Author / contact: @Kyokopom on X Repository: KitsuVp/NeoLLM
Architecture
NeoLLM is a decoder-only transformer with the following configuration:
| Parameter | Value |
|---|---|
| Hidden size | 512 |
| Layers | 12 |
| Attention heads | 8 |
| KV heads (GQA) | 4 |
| Head dim | 64 |
| Intermediate size | 1536 |
| Vocabulary | LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens) |
| Context length | 512 tokens |
| Leviathan | enabled=True; d_seed=128, modes=8, knots=16 |
| JTok / JTok-M | enabled=True / False; Torch continuous surfaces, experts=5, top_k=2 |
Parameter breakdown
| Parameter bucket | Count |
|---|---|
| Total parameters | 85.50M (85,500,216) |
| Embedding parameters (tied) | 32.97M (32,973,824) |
| Non-embedding parameters | 52.53M (52,526,392) |
| Effective trainable parameters | 85.50M (85,500,216) |
Configuration
- Architecture
- NeoLLMForCausalLM
- Context length (tokens)
- 512
- Layers
- 12
- Hidden size
- 512
- Feed-forward size
- 1,536
- Attention heads
- 8
- Key/value heads
- 4
- Head dimension
- 64
- Vocabulary size
- 64,402
- RoPE base
- 10000
- Model type
- neollm
Identity and Version
- Repository
- KitsuVp/NeoLLM
- Publisher
- Kitsun
- Task
- Not stated by the source
- Modality
- Other
- Library
- Not stated by the source
- Parameters
- 86M parameters
- Languages
- en
- Revision
- 7d6c1c8474a39bfcacf7a9989003f363663f8131
- First published
- 2025-09-16
- Last updated
- 2026-10-07
Files and Weights
17 files, 180.8 MB in total. The weights are 2 files totalling 171.1 MB in bin, safetensors.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model.safetensors | Weights | 171.1 MB | 483e1ee2897e |
| training_args.bin | Weights | 5.3 KB | 0f88bc2535cb |
| added_tokens.json | Configuration | 605 B | — |
| ademamix_precision.json | Configuration | 730 B | — |
| config.json | Configuration | 4.3 KB | — |
| configuration_neollm.py | Configuration | 83.7 KB | — |
| generation_config.json | Configuration | 205 B | — |
| modeling_neollm.py | Configuration | 448.0 KB | — |
| optimizer_precision.json | Configuration | 864 B | — |
| special_tokens_map.json | Configuration | 434 B | — |
| README.md | Documentation | 18.6 KB | — |
| chat_template.jinja | Other | 5.5 KB | — |
| .gitattributes | Repository | 1.6 KB | — |
| merges.txt | Tokenizer | 1.7 MB | — |
| tokenizer.json | Tokenizer | 4.7 MB | df1d8d5ec5d0 |
| tokenizer_config.json | Tokenizer | 553 B | — |
| vocab.json | Tokenizer | 2.8 MB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 171.1 MB
Released by Kitsun through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:2310.19531
- Described by arXiv:2409.03137
- Described by arXiv:2411.03884
- Described by arXiv:2411.07501
- Described by arXiv:2502.05795
- Described by arXiv:2502.07490
- Described by arXiv:2502.17055
- Described by arXiv:2502.20566
- Described by arXiv:2502.21309
- Described by arXiv:2504.01002
- Described by arXiv:2505.06708
- Described by arXiv:2505.13315
- Described by arXiv:2506.22049
- Described by arXiv:2510.12402
- Described by arXiv:2510.22777
- Described by arXiv:2511.05963
- Described by arXiv:2511.14721
- Described by arXiv:2511.23225
- Described by arXiv:2512.07805
- Described by arXiv:2512.08217
- Described by arXiv:2512.14391
- Described by arXiv:2601.02031
- Described by arXiv:2601.04890
- Described by arXiv:2601.15380
- Described by arXiv:2601.18030
- Described by arXiv:2601.19611
- Described by arXiv:2601.22040
- Described by arXiv:2602.00800
- Described by arXiv:2602.01212
- Described by arXiv:2602.04902
- Described by arXiv:2602.10410
- Described by arXiv:2602.21371
- Described by arXiv:2602.23057
- Described by arXiv:2603.08343
- Described by arXiv:2603.09078
- Described by arXiv:2603.14923
- Described by arXiv:2603.15031
- Described by arXiv:2605.24956
- Described by arXiv:2608.19491
- Trained on (disclosed) HuggingFaceFW/fineweb-edu
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 171.1 MB |
| 16-bit | 0.2 GB |
| 8-bit | 0.1 GB |
| 4-bit | 0.0 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About NeoLLM
How much GPU memory does NeoLLM need?
About 0.2 GB at 16-bit and 0.1 GB at 4-bit: the weights (86M parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run NeoLLM on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use NeoLLM commercially?
Yes. NeoLLM is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.
What is NeoLLM's context length?
512 tokens, from the maximum position embeddings in its published configuration.