# NeoLLM by Kitsun: Open-Weight Model
Source: https://savrn.com/models/neollm
Markdown alternate of the page above; the site index is https://savrn.com/llms.txt

---

## Runs On

What it takes to serve NeoLLM (86M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
| --- | --- | --- | --- | --- | --- |
| 16-bit | 0.2 GB | 0.2 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 |
| 8-bit | 0.1 GB | 0.1 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 |
| 4-bit | 0.0 GB | 0.1 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 |

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the [SAVRN Index](https://savrn.com/ai-index/pricing/gpus), read Oct 7, 2026.

[NeoLLM on every accelerator the SAVRN Index prices, at every precision](https://savrn.com/models/neollm/gpus)

## Model Card

By Kitsun, published under apache-2.0, revision 7d6c1c8474a3.

NeoLLM is a 85.50 M parameter decoder-only language model trained from scratch on [FineWeb-Edu](https://savrn.com/datasets/fineweb-edu) with BF16 compute, completing training in approximately ~1h 16m on NVIDIA GeForce RTX 5090. It integrates a collection of recently published attention and normalization techniques into a single architecture, with the goal of studying how they interact during pretraining. The model is actively being developed and the current checkpoint represents an intermediate training state.

Author / contact: [@Kyokopom](https://x.com/Kyokopom) on X Repository: [KitsuVp/NeoLLM](https://savrn.com/models/neollm)

### Architecture

NeoLLM is a decoder-only transformer with the following configuration:

| Parameter | Value |
| --- | --- |
| Hidden size | 512 |
| Layers | 12 |
| Attention heads | 8 |
| KV heads (GQA) | 4 |
| Head dim | 64 |
| Intermediate size | 1536 |
| Vocabulary | LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens) |
| Context length | 512 tokens |
| Leviathan | enabled=True; d_seed=128, modes=8, knots=16 |
| JTok / JTok-M | enabled=True / False; Torch continuous surfaces, experts=5, top_k=2 |

#### Parameter breakdown

| Parameter bucket | Count |
| --- | --- |
| Total parameters | 85.50M (85,500,216) |
| Embedding parameters (tied) | 32.97M (32,973,824) |
| Non-embedding parameters | 52.53M (52,526,392) |
| Effective trainable parameters | 85.50M (85,500,216) |

[Read the full model card (1,827 words)](https://savrn.com/models/neollm/card)

## Configuration

Architecture

NeoLLMForCausalLM

Context length (tokens)

512

Layers

12

Hidden size

512

Feed-forward size

1,536

Attention heads

8

Key/value heads

4

Head dimension

64

Vocabulary size

64,402

RoPE base

10000

Model type

neollm

## Identity and Version

Repository

KitsuVp/NeoLLM

Publisher

Kitsun

Task

Not stated by the source

Modality

Other

Library

Not stated by the source

Parameters

86M parameters

Languages

en

Revision

7d6c1c8474a39bfcacf7a9989003f363663f8131

First published

2025-09-16

Last updated

2026-10-07

## Files and Weights

17 files, 180.8 MB in total. The weights are 2 files totalling 171.1 MB in bin, safetensors.

Weights2 files · 171.1 MB

Configuration8 files · 538.8 KB

Tokenizer4 files · 9.2 MB

Documentation1 file · 18.6 KB

Other1 file · 5.5 KB

Repository1 file · 1.6 KB

Every file

| File | Type | Size | SHA-256 |
| --- | --- | --- | --- |
| model.safetensors | Weights | 171.1 MB | 483e1ee2897e |
| training_args.bin | Weights | 5.3 KB | 0f88bc2535cb |
| added_tokens.json | Configuration | 605 B | — |
| ademamix_precision.json | Configuration | 730 B | — |
| config.json | Configuration | 4.3 KB | — |
| configuration_neollm.py | Configuration | 83.7 KB | — |
| generation_config.json | Configuration | 205 B | — |
| modeling_neollm.py | Configuration | 448.0 KB | — |
| optimizer_precision.json | Configuration | 864 B | — |
| special_tokens_map.json | Configuration | 434 B | — |
| README.md | Documentation | 18.6 KB | — |
| chat_template.jinja | Other | 5.5 KB | — |
| .gitattributes | Repository | 1.6 KB | — |
| merges.txt | Tokenizer | 1.7 MB | — |
| tokenizer.json | Tokenizer | 4.7 MB | df1d8d5ec5d0 |
| tokenizer_config.json | Tokenizer | 553 B | — |
| vocab.json | Tokenizer | 2.8 MB | — |

## License and Download

License

apache-2.0

Access

Open weights, no gate

Download size

171.1 MB

[Download from Kitsun](https://huggingface.co/KitsuVp/NeoLLM)

Released by Kitsun through its official repository on Hugging Face. [Read the license](https://www.apache.org/licenses/LICENSE-2.0).

## Built From

- Described by arXiv:2310.19531
- Described by arXiv:2409.03137
- Described by arXiv:2411.03884
- Described by arXiv:2411.07501
- Described by arXiv:2502.05795
- Described by arXiv:2502.07490
- Described by arXiv:2502.17055
- Described by arXiv:2502.20566
- Described by arXiv:2502.21309
- Described by arXiv:2504.01002
- Described by arXiv:2505.06708
- Described by arXiv:2505.13315
- Described by arXiv:2506.22049
- Described by arXiv:2510.12402
- Described by arXiv:2510.22777
- Described by arXiv:2511.05963
- Described by arXiv:2511.14721
- Described by arXiv:2511.23225
- Described by arXiv:2512.07805
- Described by arXiv:2512.08217
- Described by arXiv:2512.14391
- Described by arXiv:2601.02031
- Described by arXiv:2601.04890
- Described by arXiv:2601.15380
- Described by arXiv:2601.18030
- Described by arXiv:2601.19611
- Described by arXiv:2601.22040
- Described by arXiv:2602.00800
- Described by arXiv:2602.01212
- Described by arXiv:2602.04902
- Described by arXiv:2602.10410
- Described by arXiv:2602.21371
- Described by arXiv:2602.23057
- Described by arXiv:2603.08343
- Described by arXiv:2603.09078
- Described by arXiv:2603.14923
- Described by arXiv:2603.15031
- Described by arXiv:2605.24956
- Described by arXiv:2608.19491
- Trained on (disclosed) [HuggingFaceFW/fineweb-edu](https://savrn.com/datasets/fineweb-edu)

## Memory Requirements

| Precision | Weights in memory |
| --- | --- |
| As published | 171.1 MB |
| 16-bit | 0.2 GB |
| 8-bit | 0.1 GB |
| 4-bit | 0.0 GB |

Weights only, from the published parameter count; the key-value cache and runtime add to this.

## Questions About NeoLLM

### How much GPU memory does NeoLLM need?

About 0.2 GB at 16-bit and 0.1 GB at 4-bit: the weights (86M parameters) plus a working margin. A long context needs more.

### What is the cheapest GPU to run NeoLLM on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

### Can I use NeoLLM commercially?

Yes. NeoLLM is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

### What is NeoLLM's context length?

512 tokens, from the maximum position embeddings in its published configuration.

## Kitsun

[All models and datasets](https://savrn.com/model-publishers/kitsuvp)

## Versions

- [7d6c1c8474a3](https://savrn.com/models/neollm/versions/7d6c1c8474a3) · current 2026-10-07

## Explore More

- [All models under apache-2.0](https://savrn.com/models/licenses/apache-2-0)
- [Model comparisons](https://savrn.com/models/comparisons)
- [The model directory](https://savrn.com/models)
- [Open model prices by host](https://savrn.com/ai-index/pricing/open-models)

## Source

- Repository metadata, read 2026-10-07.
- [Hugging Face record](https://huggingface.co/KitsuVp/NeoLLM)
- [How the hub is built](https://savrn.com/model-hub/methodology)