This repository contains a text-only Mini-K3-1H v2 pretraining checkpoint from a controlled 20-architecture comparison. The family retains Kimi-K3's KDA and Gated MLA operators, block Attention Residuals, Stable LatentMoE, SiTU activations, output gates, and Quantile Balancing at approximately one billion logical parameters. The exact architecture for this repository is listed below; some ablations deliberately replace the baseline KDA/MLA ratio, decay granularity, convolution length, or positional encoding. - Hidden width / attention heads / KDA head width: 1024 / 12 / 128 - Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839 control state retained in FP32 where…
Open-weight model · Text generation
Mini-K3-1H-decay-g64-v2_B
by Nkkbr nkkbr/Mini-K3-1H-decay-g64-v2_B
Mini-K3-1H-decay-g64-v2_B is an open-weight model for text generation from Nkkbr. It has 1B parameters and a 131,072-token context. At 16-bit it needs about 2.4 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 908 downloads a month.
This repository contains a text-only Mini-K3-1H v2 pretraining checkpoint from a controlled 20-architecture comparison.
Parameters1B
Context131,072
Weights2.0 GB
License—
AccessOpen weights
Monthly Downloads908
Runs On
What it takes to serve Mini-K3-1H-decay-g64-v2_B (1B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 2.0 GB | 2.4 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 1.0 GB | 1.2 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 0.5 GB | 0.6 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 16, 2026.
Mini-K3-1H-decay-g64-v2_B on every accelerator the SAVRN Index prices, at every precision
Model Card
This repository contains a text-only Mini-K3-1H v2 pretraining checkpoint from a controlled 20-architecture comparison. The family retains Kimi-K3's KDA and Gated MLA operators, block Attention Residuals, Stable LatentMoE, SiTU activations, output gates, and Quantile Balancing at approximately one billion logical parameters. The exact architecture for this repository is listed below; some ablations deliberately replace the baseline KDA/MLA ratio, decay granularity, convolution length, or positional encoding. This decay-group run was recovered after the original launch stopped before its initialization checkpoint because the training and portable checkpoint layouts of dtbias differed. It…
Excerpt from the card by Nkkbr.
Configuration
- Context length (tokens)
- 131,072
- Layers
- 13
- Hidden size
- 1,024
- Attention heads
- 12
- Vocabulary size
- 163,840
- Experts
- 64
- Model type
- mini_k3
Identity and Version
- Repository
- nkkbr/Mini-K3-1H-decay-g64-v2_B
- Publisher
- Nkkbr
- Task
- Text generation
- Modality
- Text
- Library
- pytorch
- Parameters
- 1B parameters
- Languages
- Not stated by the source
- Revision
- eb4270b389768ce301978933b1c46d09e262157c
- First published
- 2026-09-21
- Last updated
- 2026-09-24
Files and Weights
52 files, 2.2 GB in total. The weights are 1 file totalling 2.0 GB in safetensors.
Weights1 file · 2.0 GB
Configuration15 files · 445.5 KB
Tokenizer3 files · 2.8 MB
Documentation7 files · 14.7 KB
Other25 files · 121.5 MB
Repository1 file · 1.6 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model.safetensors | Weights | 2.0 GB | e6875ca43e24 |
| __init__.py | Configuration | 592 B | — |
| checkpoint_manifest.json | Configuration | 1.4 KB | — |
| config.json | Configuration | 1.6 KB | — |
| configuration_mini_k3.py | Configuration | 8.5 KB | — |
| export_parameter_schema.py | Configuration | 1.4 KB | — |
| initialize_model.py | Configuration | 1.5 KB | — |
| modeling_mini_k3.py | Configuration | 37.5 KB | — |
| parallelism.json | Configuration | 1.0 KB | — |
| smoke_test.py | Configuration | 3.8 KB | — |
| training/data_manifest.public.json | Configuration | 339.2 KB | — |
| training/environment.public.json | Configuration | 1.1 KB | — |
| training/experiment_manifest.public.json | Configuration | 1.5 KB | — |
| training/initialization_manifest.json | Configuration | 43.4 KB | — |
| training/training_recipe.public.json | Configuration | 737 B | — |
| verify_config.py | Configuration | 2.2 KB | — |
| ARCHITECTURE.md | Documentation | 1.3 KB | — |
| ARCHITECTURE_PACKAGE_README.md | Documentation | 1.7 KB | — |
| LICENSE | Documentation | 3.1 KB | — |
| PROVENANCE.md | Documentation | 927 B | — |
| README.md | Documentation | 4.9 KB | — |
| TRAINING.md | Documentation | 1.9 KB | — |
| VARIANT.md | Documentation | 895 B | — |
| parameter_schema.jsonl | Other | 349.8 KB | — |
| pyproject.toml | Other | 606 B | — |
| requirements-accelerated.txt | Other | 156 B | — |
| requirements.txt | Other | 25 B | — |
| training/training.jsonl | Other | 121.2 MB | 124b7c53c65c |
| training/validation.jsonl | Other | — | |
| training/validation_source_academic_arxiv.jsonl | Other | — | |
| training/validation_source_academic_pubmed.jsonl | Other | — | |
| training/validation_source_chinese_edu.jsonl | Other | — | |
| training/validation_source_chinese_general_existing.jsonl | Other | — | |
| training/validation_source_chinese_general_extra.jsonl | Other | — | |
| training/validation_source_code.jsonl | Other | — | |
| training/validation_source_english_nemotron.jsonl | Other | — | |
| training/validation_source_english_nemotron_extra.jsonl | Other | — | |
| training/validation_source_knowledge_en.jsonl | Other | — | |
| training/validation_source_knowledge_zh.jsonl | Other | — | |
| training/validation_source_longform_en.jsonl | Other | — | |
| training/validation_source_longform_zh.jsonl | Other | — | |
| training/validation_source_math.jsonl | Other | — | |
| training/validation_source_multi_deu.jsonl | Other | — | |
| training/validation_source_multi_fra.jsonl | Other | — | |
| training/validation_source_multi_jpn.jsonl | Other | — | |
| training/validation_source_multi_por.jsonl | Other | — | |
| training/validation_source_multi_rus.jsonl | Other | — | |
| training/validation_source_multi_spa.jsonl | Other | — | |
| .gitattributes | Repository | 1.6 KB | — |
| tiktoken.model | Tokenizer | 2.8 MB | b6c497a7469b |
| tokenizer.py | Tokenizer | 2.5 KB | — |
| tokenizer_config.json | Tokenizer | 3.5 KB | — |
License and Download
- License
- Not stated by the source
- Access
- Open weights, no gate
- Download size
- 2.0 GB
Download from Nkkbr
Released by Nkkbr through its official repository on Hugging Face.
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 2.0 GB |
| 16-bit | 2.0 GB |
| 8-bit | 1.0 GB |
| 4-bit | 0.5 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About Mini-K3-1H-decay-g64-v2_B
How much GPU memory does Mini-K3-1H-decay-g64-v2_B need?
About 2.4 GB at 16-bit and 0.6 GB at 4-bit: the weights (1B parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run Mini-K3-1H-decay-g64-v2_B on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
What is Mini-K3-1H-decay-g64-v2_B's context length?
131,072 tokens, from the maximum position embeddings in its published configuration.
Similar Models
This repository contains a text-only Mini-K3-1H checkpoint from the controlled Attention Residuals follow-up study. The comparison includes standard PreNorm, three block granularities, and full sublayer-level AttnRes. The original Block-4 baseline is the separately published nkkbr/Mini-K3-1H-v2; this model's only architectural change is the depth-wise residual mechanism. - Hidden width / attention heads / KDA head width: 1024 / 12 / 128 - Residual mode / decoder layers per block: block / 1 - Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839 control state retained in FP32 where defined by the implementation Every numbered checkpoint is an immutable Git tag. main…
This model is a fine-tuned version of None. It has been trained using TRL. This model was trained with SFT.
This model is a fine-tuned version of None. It has been trained using TRL. This model was trained with SFT.
This model is a fine-tuned version of None. It has been trained using TRL. This model was trained with SFT.
This model is a fine-tuned version of None. It has been trained using TRL. This model was trained with SFT.