The instruction-tuned release of Quartz Micro Preview Base — the same ~1B-parameter mixture-of-experts model, trained completely from scratch on a single consumer GPU, now given a full-parameter supervised fine-tune so it follows instructions instead of just completing text. This is a preview, not a finished product. It is small on every axis by design, and its evaluation below is reported honestly, including where it still falls short — that's this project's standing practice, not a disclaimer added after the fact. - Full-parameter SFT, not LoRA — the base model has zero instruction-following behavior, so a light adapter touch isn't the right tool; every parameter was updated.…
Open weights
apache-2.0
1.1B parameters
2,048 tokens
transformers
Model · Text generation
Apple
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari We introduce OpenELM, a family of Open Efficient Language Models. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. We pretrained OpenELM models using the CoreNet library. We release both pretrained and instruction tuned models with 270M, 450M, 1.1B and 3B parameters. We release the complete framework, encompassing data preparation, training, fine-tuning, and evaluation procedures, alongside multiple pre-trained…
Open weights
apple-amlr
1.1B parameters
transformers
N
Model · Text generation
Nkkbr
This repository contains a text-only Mini-K3-1H checkpoint from the controlled Attention Residuals follow-up study. The comparison includes standard PreNorm, three block granularities, and full sublayer-level AttnRes. The original Block-4 baseline is the separately published nkkbr/Mini-K3-1H-v2; this model's only architectural change is the depth-wise residual mechanism. - Hidden width / attention heads / KDA head width: 1024 / 12 / 128 - Residual mode / decoder layers per block: block / 1 - Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839 control state retained in FP32 where defined by the implementation Every numbered checkpoint is an immutable Git tag. main…
Open weights
1B parameters
131,072 tokens
pytorch
N
Model · Text generation
Nkkbr
This repository contains a text-only Mini-K3-1H v2 pretraining checkpoint from a controlled 20-architecture comparison. The family retains Kimi-K3's KDA and Gated MLA operators, block Attention Residuals, Stable LatentMoE, SiTU activations, output gates, and Quantile Balancing at approximately one billion logical parameters. The exact architecture for this repository is listed below; some ablations deliberately replace the baseline KDA/MLA ratio, decay granularity, convolution length, or positional encoding. - Hidden width / attention heads / KDA head width: 1024 / 12 / 128 - Vocabulary / BOS / generation EOS / PAD: 163840 / 163584 / 163586 / 163839 control state retained in FP32 where…
Open weights
1B parameters
131,072 tokens
pytorch
N
Model · Text generation
Nkkbr
This repository contains a text-only Mini-K3-1H v2 pretraining checkpoint from a controlled 20-architecture comparison. The family retains Kimi-K3's KDA and Gated MLA operators, block Attention Residuals, Stable LatentMoE, SiTU activations, output gates, and Quantile Balancing at approximately one billion logical parameters. The exact architecture for this repository is listed below; some ablations deliberately replace the baseline KDA/MLA ratio, decay granularity, convolution length, or positional encoding. This decay-group run was recovered after the original launch stopped before its initialization checkpoint because the training and portable checkpoint layouts of dtbias differed. It…
Open weights
1B parameters
131,072 tokens
pytorch
This model is a fine-tuned version of None. It has been trained using TRL. This model was trained with SFT.
Open weights
1B parameters
2,048 tokens
transformers
