K2-Horizon-MoVA-36B-A4B-MLA: Model Card

Research preview. main is now the healed checkpoint. It is measurably worse than the original (KLD 0.091, +0.058 nats) and loops occasionally in long generations (2 of 48, the original 0 of 48). Read "Known issues" first.

K2-Horizon-MoVA-36B-A4B-MLA (research preview)

IFM/K2-Horizon-MoVA-36B-A4B with its attention converted from grouped-query attention (GQA) to multi-head latent attention (MLA), DeepSeek-style, without retraining. The KV cache shrinks from 192 KiB to 84 KiB per token in BF16 (42 KiB with an 8-bit latent). The routed experts and the value experts are the original weights.

As far as we know this is the first MLA conversion of a GQA mixture-of-value-experts model. It is a research preview: calibrated, then healed by distillation of the attention projections only. See "Known issues".

Versions

KLD, NLL and top-1: NanoSeedLM engine against the original's MLX BF16 log-probs, 8 held-out 1,024-token windows (8,184 positions). Loops: 48 long generations in oMLX (see "Known issues").

The revisions share every base shard; only model-mla-delta.safetensors, mla_config.json and config.json (its mla_config name) differ. The training-free conversion stays available:

hf download txgsync/K2-Horizon-MoVA-36B-A4B-MLA --revision naive-j768 --local-dir K2-Horizon-MoVA-36B-A4B-MLA-naive

The healing pilot. Only the latent attention projections were trained (142 M parameters; everything else, experts included, stays IFM's). Teacher: the same weights with the original GQA attention, run in the same process. Loss: KL divergence from the teacher over the full vocabulary at every position. Data: 70% IFM/TxT360-3efforts (IFM's SFT conversations, rendered with this model's chat template, tool calls included) and 30% IFM/TxT360-v2 web text, 16,384-token sequences. 8 x RTX PRO 6000 for 1 h 56 min. It recovered 37-39% of the conversion's KL divergence on held-out prose and code (our pre-registered target was 50%), still improving when it stopped. A continuation (17 M more tokens, restarted optimizer) made held-out KLD slightly worse and was stopped; this checkpoint is the best so far.

Seed variants (branches)

The healed model with every weight matrix stored as SeedLM seeds (16-bit LFSR seeds, int4 coefficients and an exponent per 8 weights; no affine quantization), for Apple Silicon through MLX / oMLX (Trust Remote Code) and the NanoSeedLM C engine. Each branch has its own card.

Both are research previews released despite the coherence rule (the failures are reasoning loops to the token limit). Every searched block is in the public seed bank.

hf download txgsync/K2-Horizon-MoVA-36B-A4B-MLA --revision seeds-4.7bpw --local-dir K2-Horizon-MoVA-36B-A4B-MLA-seeds-4.7bpw

KV cache per token

How it was made (TransMLA, calibrated)

Per layer (method after Meng et al., TransMLA): 1. RoRoPE. For each RoPE frequency, the 8 KV heads are mixed by a unitary matrix (PCA of the keys across heads; it commutes with RoPE), concentrating positional information in one virtual head. That head keeps RoPE (128 dims); the other seven lose it. 2. Joint latent. PCA of calibration activations of [keys without RoPE ; values] gives a rank-768 latent, K and V balanced by energy. MoVA's values are a SiLU mixture of 4 of 64 routed value experts, not a projection: the experts run as before and their output is projected into the latent as it is cached. 3. Absorption. Key up-projections are folded into per-head query maps (q_lat); value up-projections (v_up) are applied per head after attention, before MoVA's softplus gate (the gate is why v_up is not folded into o_proj).

Calibration: ~160k tokens of mixed text; no gradient steps.

Quality (training-free, before any fine-tuning)

Against the original BF16 model on held-out text (4,096-token windows):

For scale, +0.26 nats means the model is about 1.3x more "surprised" by real text than the original. KLD and NLL are proxies: in short hands-on use (chat, agentic coding in pi with tool calls) the converted model has been hard to tell apart from the original, but long generations loop far more often (see "Known issues"). Your evaluations are welcome.

Running it (MLX / oMLX)

The architecture lives in k2_mla.py, loaded through config.json "model_file" with trust_remote_code. It needs the K2-Horizon MLX model (mlx_lm.models.k2_horizon), which oMLX provides.

hf download txgsync/K2-Horizon-MoVA-36B-A4B-MLA --local-dir ~/models/K2-Horizon-MoVA-36B-A4B-MLA

Then add the folder to oMLX and enable trust_remote_code for it. Recommended settings: temperature 1.0, top_p 0.95 (IFM's), reasoning effort high, a max_tokens cap (see "Known issues"); TurboQuant KV off for this model.

Memory: ~75 GB of BF16 weights (the original's tensors; k_proj is unused) plus the cache above.

Not supported: Hugging Face transformers (the bundled torch code is not included: it would run plain GQA), vLLM / SGLang, llama.cpp (an MLA variant of the K2-Horizon architecture would need C++ support there).

Known issues

• Looping: less than before healing, still more than the original. In oMLX at stock settings (40 mixed prompts plus 8 replays of an agentic coding turn), the healed main fell into degenerate repetition in 2 of 48 generations (a rewriting task's reasoning, and a large file write), against 7 of 48 for the training-free naive-j768 and 0 of 48 for the original MoVA. With 48 samples neither difference is statistically significant (Fisher one-sided p = 0.08 against naive-j768). Loops show up in long generations; short turns and tool calls are usually fine. top_k 20 + min_p 0.05 did not help the training-free model (10/48). Cap max_tokens (or use a thinking budget) so a loop ends on its own, and prefer the original model for long agentic or code-writing sessions.
• Prefill is slow. The attention keys are 896-dimensional (latent + RoPE), which MLX's fused attention kernels do not cover; prefill takes the unfused path, which also needs more transient memory at long context.
• oMLX memory accounting still assumes the original GQA cache (192 KiB/token), so it admits less context than MLA could hold.

Files

• model-000NN-of-00048.safetensors: the original BF16 weights (IFM, unchanged).
• model-mla-delta.safetensors: the new attention tensors per layer (kv_a_x, kv_a_v, k_rope_proj, q_rope_mix, q_lat, v_up).
• k2_mla.py: the MLX model; mla_config.json: per-layer ranks.

Credits

Base model: IFM, K2-Horizon-MoVA-36B-A4B (Apache-2.0). Method: TransMLA (Meng et al., 2025). Conversion by Matt Barnson (NanoSeedLM research), using IFM's open checkpoints and data.

Original model card

K2-Horizon-MoVA-36B-A4B

K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family: a Mixture-of-Experts model with Mixture-of-Values attention (MoVA) that stores 36B parameters and runs 4B per token. We have released the checkpoints, along with the data and the training code.

K2-Horizon-MoVA-36B-A4B Highlights

• Frontier-class results at 4B active parameters. On agentic and reasoning benchmarks it outscores open weight dense (approximately 30B model size) and MoE models up to 15× its size; and also performs competitively against closed frontier models (see Benchmark Results).
• 512K context. Native 524,288-token context from the midtraining stages onward.
• Intermediate checkpoints. Intermediate checkpoints are released so capability changes can be studied across training rather than at a single checkpoint.
• Fully open. Training data/recipe and the training code are public.

Benchmark Results

Scores in %. Bold marks the best score in each row. Sections follow the Artificial Analysis Intelligence Index categories. Baseline scores are from Artificial Analysis; Muse Glimmer-30B at high reasoning effort, all other open models in their reasoning mode.

Quickstart

Serving

vLLM, recipe at recipes.vllm.ai/IFM:

vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --trust-remote-code \
  --dtype bfloat16 \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --enable-auto-tool-choice

Use an exact branch name from the inventory with vLLM's --revision option. For example, --revision pretrain_1100000 selects the final checkpoint of Pretraining, at step 1,100,000.

SGLang recipe validated on 2× H200 in the SGLang K2 Horizon cookbook:

python3 -m sglang.launch_server \
  --model-path IFM/K2-Horizon-MoVA-36B-A4B \
  --revision main \
  --tp 2 \
  --ep 2 \
  --dtype bfloat16 \
  --attention-backend fa3 \
  --json-model-override-args '{"xllm_source_router_gemm_partitions":2}' \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --host 0.0.0.0 --port 30000

API Usage

[!Tip] Recommended settings: reasoning_effort="high", temperature=1.0, top_p=0.95. Reasoning depth is selected per request through chat_template_kwargs. Thinking is returned in reasoning_content and the answer in content.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="IFM/K2-Horizon-MoVA-36B-A4B",
    messages=[{"role": "user", "content": "Explain the result step by step."}],
    temperature=1.0,
    top_p=0.95,
    max_tokens=32768,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "high", "tool_call_format": "xml"}},
)
message = response.choices[0].message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)

Our model supports multiple tool-call formats, which can be changed with chat_template_kwargs. The supported values are json, xml, and xml_typed. The default is xml. Keep --tool-call-parser k2_horizon enabled to parse the selected format.

Transformers

Validated with Transformers 5.15.0, PyTorch 2.13.0, Safetensors 0.8.0.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "IFM/K2-Horizon-MoVA-36B-A4B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", dtype="bfloat16", low_cpu_mem_usage=True, trust_remote_code=True
)

inputs = tokenizer("Explain why long-context evaluation is difficult.", return_tensors="pt").to(model.device)
inputs.pop("token_type_ids", None)
outputs = model.generate(**inputs, max_new_tokens=32768, temperature=1.0, top_p=0.95, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Overview

The table below lists the training stages in order and the purpose of each stage.

Training steps are counted within each stage or phase. Token budgets cover only the additional training in that stage or phase. For example, the 50B tokens listed for SFT Phase 2 are additional to the 219B tokens in Phase 1, bringing the cumulative budget to 269B tokens by the end of Phase 2. Here, B and T denote billion and trillion tokens, respectively.

Each stage or phase continues from the final checkpoint of the preceding stage or phase.

Some stages, such as SFT, have multiple phases with slight changes to the data mix while retaining the same overall purpose.

Release Artifacts

The tables below list the release artifacts for K2-Horizon-MoVA-36B-A4B, their availability, and the expected release dates for remaining items.

Last updated: 2026-09-28

Status:

• Available — fully released for the scope listed;
• Partial — some items are available, with remaining items listed in the notes;
• In Progress — being prepared for release but not yet available.

Artifact Index

Checkpoint Inventory

Model repository: IFM/K2-Horizon-MoVA-36B-A4B

Branch names below refer to this repository. Patterns containing * group branches by training stage or phase. The * is a placeholder for a training-step number, not a literal branch name. Intermediate checkpoint groups exclude the final checkpoint listed separately; a pattern does not imply that a checkpoint is available at every step.

For example, sft_1_11000 is the checkpoint saved at training step 11,000 within SFT Phase 1, and is the final checkpoint of that phase. The numeric suffix is the step within the named stage or phase, not the cumulative step across all training. Thus, sft_2_2500 refers to step 2,500 within SFT Phase 2.

For a partially released group, the available checkpoints and the remaining checkpoints are listed in the notes.

[!IMPORTANT] Note: - Released K2-Horizon Hugging Face checkpoints (e.g. Huggingface) can be used for inference, evaluation, and downstream fine-tuning (including SFT). Training behavior in the bundled Hugging Face implementation may differ from native xLLM, including the auxiliary load-balancing loss. To continue the original pretraining with xLLM's training behavior, use the native xLLM checkpoint and xLLM runtime.

Best Practices

1. Reasoning effort: always high. All reported results use high reasoning effort. Pass {"chat_template_kwargs": {"reasoning_effort": "high"}} on every request. medium and low effort settings are not recommended except for research on reasoning efforts.
2. Sampling parameters. temperature=1.0, top_p=0.95.
3. Serving. Use the validated SGLang recipe above: BF16, TP=2, FlashAttention-3, and the xllm_source_router_gemm_partitions override, which preserves the checkpoint's router numerics. Full recipes for every K2-Horizon size, with measured H200 latency and throughput, are in the SGLang cookbook and the vLLM recipe.
4. Parsers. Enable the k2_horizon reasoning parser for chat, and add the k2_horizon tool-call parser for agent use. Leave both off for plain completion-style generation.

Citation

@misc{k2horizon2026,
  title  = {Introducing K2 Horizon: Frontier Performance, Radically Open},
  author = {{IFM Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/k2/},
}