Note: DeepSeek-V4-Flash-DSpark is not a new model. It is the same checkpoint with an additional speculative decoding module attached. A minimal inference example is available in the inference folder. For more details, refer to: https://github.com/deepseek-ai/DeepSpec We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) — both supporting a context length of one million tokens. DeepSeek-V4 series incorporate several key upgrades in architecture and optimization: 1. Hybrid Attention Architecture: We design a hybrid…
Open weights
mit
165.3B parameters
1,048,576 tokens
transformers
Scaling Reinforcement Learning Toward Self-Improvement MiMo-V2.6-Flash-RL is the efficiency-balanced checkpoint of the MiMo-V2.6 series. The series is built to scale reinforcement learning toward self-improvement — scaling RL compute, environment diversity, and grader compute together, so the model keeps expanding its capability frontier through exploration and feedback. Key features include: - Native Omnimodal + Long Horizon: Text, image, video, and audio in one model; 1M tokens for long repositories, tool traces, and multi-session agent runs. - Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2): After mixed RL, MOPD2 combines autonomous student rollouts with prefix-conditioned…
Open weights
mit
159.4B parameters
1,048,576 tokens
transformers
Scaling Reinforcement Learning Toward Self-Improvement MiMo-V2.6-Flash-RL is the efficiency-balanced checkpoint of the MiMo-V2.6 series. The series is built to scale reinforcement learning toward self-improvement — scaling RL compute, environment diversity, and grader compute together, so the model keeps expanding its capability frontier through exploration and feedback. Key features include: - Native Omnimodal + Long Horizon: Text, image, video, and audio in one model; 1M tokens for long repositories, tool traces, and multi-session agent runs. - Multi-Prefix Multi-Teacher On-Policy Distillation (MOPD2): After mixed RL, MOPD2 combines autonomous student rollouts with prefix-conditioned…
Open weights
mit
159.4B parameters
1,048,576 tokens
transformers
Hy3 with half of its routed experts removed by RAZOR, a training-free expert pruning method. Every MoE layer keeps 96 of its original 192 routed experts. No gradient updates or recovery training were applied: the retained weights are the base model's own weights. Pruning touches only the routed expert pool. Attention, shared experts, the embedding and the LM head are untouched, so the compute per token drops only by the share of expert FLOPs that the removed experts would have contributed. The other budget is Requires a Transformers build containing the native hyv3 implementation. Weights are bfloat16. RAZOR asks whether the surviving computation can replace an expert's function, rather…
Open weights
apache-2.0
153.8B parameters
262,144 tokens
transformers
DeepSeek-V4.1-Flash, ready to serve on two RTX PRO 6000 Blackwell cards (sm120, 96 GB each) with vLLM at tensor-parallel 2. The routed experts are - int4 Engram tables; - a 3-bit DSpark drafter; - a serving stack that keeps 30% of the routed experts (the ones agentic coding uses least) in pinned host RAM and runs them next to the VRAM experts; - a decode-once prefill kernel for the 3-bit experts. The recipe/ folder has the image build, the sm120 patches, the MoE kernel, and the serving, benchmark, KL and quantization scripts. We measured KL against the original model with uses the original model's top-512 logprobs on three corpora, 49,152 scored positions each. The neutral corpus is…
Open weights
mit
219B parameters
1,048,576 tokens
K
Model · Text generation
Keys
Full GLM-5.3 (753B, glmmoedsa) quantized to EXL3 with a per-expert mixed bit width averaging 2.75 bpw on the routed experts, sized to serve on four NVIDIA DGX Sparks (GB10, 128 GB unified memory each) with room for a 200K-token KV pool (or 1M tokens with decode-context-parallel 4). Unmodified weights of zai-org/GLM-5.3 (no abliteration, no fine-tuning), quantized from the BF16 checkpoint. For the earlier uniform-width quant see The fastest way to serve this checkpoint is TensorFold's native glmmoedsa engine — tensor parallel 4, one rank per Spark, no vLLM in the serving path; a drafted reply equals a serial one. Recipe, one-shot launcher, benches and an AGENTS.md needs no extra draft model…
Open weights
other
138.1B parameters
1,048,576 tokens
exllamav3
