SAVRN
Search Contact SAVRN

Open-weight model · Text generation

deepseek-v4-gguf

by Salvatore Sanfilippo antirez/deepseek-v4-gguf

This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).

Parameters
Context
Weights3.8 TB
Licensemit
AccessOpen weights
Monthly Downloads2M

Model Card

By Salvatore Sanfilippo, published under mit, revision f71f23d552d6.

This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader). https://github.com/antirez/ds4 Use q2 on 128 GB Mac machines, q4 on machines with ≥ 256 GB RAM, pair either with MTP for optional speculative decoding. The filename is the spec. In detail, for the q2 file: For the q4 file, only the three routed-expert classes change to Q4K. Everything else is byte-for-byte identical to the q2 recipe. The motivation behind the asymmetry: the routed experts are the majority of the parameter count but each individual expert handles only a fraction of tokens, so aggressive quantization on…

Read Salvatore Sanfilippo's full model card

DeepSeek V4 Flash — GGUF for ds4

This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).

https://github.com/antirez/ds4

Files

File Size Routed experts (ffn_{gate,up,down}_exps) Everything else
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf 80.8 GiB IQ2_XXS (gate, up) + Q2_K (down) Q8_0 attn proj / shared experts / output, F16 router + embed + indexer + compressor + HC, F32 norms / sinks / bias
DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.gguf 153.3 GiB Q4_K (all three) same as above
DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf 3.6 GiB MTP / speculative-decoding support (optional, not standalone).

Use q2 on 128 GB Mac machines, q4 on machines with ≥ 256 GB RAM, pair either with MTP for optional speculative decoding.

Quantization recipe

The filename is the spec. In detail, for the q2 file:

Tensor class Quant Notes
blk.*.ffn_gate_exps, blk.*.ffn_up_exps IQ2_XXS routed-expert up/gate
blk.*.ffn_down_exps Q2_K routed-expert down (K-quant for quality)
blk.*.ffn_{gate,up,down}_shexp Q8_0 shared experts
blk.*.attn_q_a, attn_q_b, attn_kv, attn_output_a, attn_output_b Q8_0 all attention projections (MLA + low-rank output)
output.weight Q8_0 output head
token_embd.weight F16 input embedding
blk.*.ffn_gate_inp (router) F16 learned router
blk.*.exp_probs_b (router bias), blk.*.attn_sinks, all *_norm.weight F32
blk.*.ffn_gate_tid2eid I32 hash-routing tables (first 3 layers only)
blk.*.attn_compressor_*, blk.*.indexer_*, blk.*.hc_*, blk.*.output_hc_* F16 / F32 DSv4-specific auxiliary blocks

For the q4 file, only the three routed-expert classes change to Q4_K. Everything else is byte-for-byte identical to the q2 recipe.

The motivation behind the asymmetry: the routed experts are the majority of the parameter count but each individual expert handles only a fraction of tokens, so aggressive quantization on them costs less in average quality than the same treatment of router, projections, or shared experts. Keeping the decision-making components at Q8_0 preserves model behavior; crushing the experts buys the size.

Usage

git clone https://github.com/antirez/ds4
cd ds4
./download_model.sh q2     # 128 GB RAM machines
./download_model.sh q4     # >= 256 GB RAM machines
./download_model.sh mtp    # optional MTP / speculative decoding
make

./ds4 -p "Explain Redis streams in one paragraph."
./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192

The download_model.sh script fetches from this repository, resumes partial downloads, and points ./ds4flash.gguf at the selected variant.

License

MIT. The base model copyright is held by DeepSeek; the GGUFs are redistributed under the base model's release terms.

Identity and Version

Repository
antirez/deepseek-v4-gguf
Publisher
Salvatore Sanfilippo
Task
Text generation
Modality
Text
Library
gguf
Parameters
Not stated by the source
Languages
en
Revision
f71f23d552d664e523b422157b2befbf74040380
First published
2026-04-26
Last updated
2026-08-31

Files and Weights

25 files, 3.8 TB in total. The weights are 22 files totalling 3.8 TB in gguf.

Weights22 files · 3.8 TB
Documentation1 file · 3.5 KB
Other1 file · 450.9 MB
Repository1 file · 4.2 KB
Every file
FileTypeSizeSHA-256
DeepSeek-V4-Flash-DSpark-support-0731.ggufWeights6.0 GB 7e319924541d
DeepSeek-V4-Flash-DSpark-support.ggufWeights6.0 GB 8b3adf5942be
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.ggufWeights86.7 GB ca22ae2f838e
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.ggufWeights86.7 GB efc7ed607ff2
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.ggufWeights86.7 GB 31598c67c8b8
DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed-0731.ggufWeights97.6 GB 659e22fbd01c
DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.ggufWeights97.6 GB edabc92af63a
DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.ggufWeights3.8 GB afd481ee689d
DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.ggufWeights156.0 GB 0e3a161b670f
DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.ggufWeights164.6 GB 6bb77b5ddcbc
DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix.ggufWeights164.6 GB a2a3b31eca06
DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2.ggufWeights164.6 GB 39e5de72ac54
DeepSeek-V4-Flash-Vision-Encoder.ggufWeights932.9 MB 00cd4d81a435
DeepSeek-V4-Flash-Vision-Exp-DSpark-support.ggufWeights6.0 GB 0807a67fd9ce
DeepSeek-V4-Flash-Vision-Exp-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8.ggufWeights86.7 GB 8f2d42c0071c
DeepSeek-V4-Flash-Vision-Exp-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8.ggufWeights97.6 GB cded4517bb9d
DeepSeek-V4-Flash-Vision-Exp-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out.ggufWeights156.0 GB fc1efb96fa26
DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix-0813.ggufWeights464.6 GB c4d997ab9894
DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.ggufWeights464.6 GB a0314d9c0e16
DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct.ggufWeights464.6 GB 0e481c300b52
DeepSeek-V4-Pro-Q4K-Layers-31-output.ggufWeights442.0 GB 41d14e4ccf9a
DeepSeek-V4-Pro-Q4K-Layers00-30.ggufWeights457.5 GB 3c4526735ce2
README.mdDocumentation3.5 KB
imatrix/DeepSeek-V4-Flash-chat-v2-routed-moe-ds4-1p5m.datOther450.9 MB 02a7c78c2987
.gitattributesRepository4.2 KB

License and Download

License
mit
Access
Open weights, no gate
Download size
3.8 TB
Download from Salvatore Sanfilippo

Released by Salvatore Sanfilippo through its official repository on Hugging Face. Read the license.

Built From

Memory Requirements

PrecisionWeights in memory
As published3.8 TB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About deepseek-v4-gguf

Can I use deepseek-v4-gguf commercially?

Yes. deepseek-v4-gguf is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.

Similar Models

Fine-tune Qwen3 (14B) for free using our Google Colab notebook! - Read our Blog about Qwen3 support: unsloth.ai/blog/qwen3 - View the rest of our notebooks in our docs here. Qwen3-Coder is available in multiple sizes. Today, we're excited to introduce Qwen3-Coder-30B-A3B-Instruct. This streamlined model maintains impressive performance and efficiency, featuring the following key enhancements: - Significant Performance among open models on Agentic Coding, Agentic Browser-Use, and other foundational coding tasks. - Long-context Capabilities with native support for 256K tokens, extendable up to 1M tokens using Yarn, optimized for repository-scale understanding. - Agentic Coding supporting for…

Open weights apache-2.0 transformers

Model · Text generation

opt-125m

AI at Meta

OPT was first introduced in Open Pre-trained Transformer Language Models and first released in metaseq's repository on May 3rd 2022 by Meta AI. Disclaimer: The team releasing OPT wrote an official model card, which is available in Appendix D of the paper. Content from this model card has been written by the Hugging Face team. To quote the first two paragraphs of the official paper OPT was predominantly pretrained with English text, but a small amount of non-English data is still present within the training corpus via CommonCrawl. The model was pretrained using a causal language modeling (CLM) objective. OPT belongs to the same family of decoder-only models like GPT-3. As such, it was…

Open weights other 2,048 tokens transformers

Model · Text generation

Ornith-1.5-9B-GGUF

Ornith

Chirp Chirp! We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement. Ornith-1.5 extends Ornith-1.0 (which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mid-training, and post-training) by expanding the self-improvement loop from scaffold and rollout optimization to jointly optimizing task generation, scaffold construction, and solution rollouts. Rather than relying on a fixed set of human-curated tasks and manually designed harnesses, Ornith-1.5 continuously generates new training tasks, discovers effective strategies for solving them, and improves the policy through reinforcement learning. For…

Open weights mit transformers

Model · Text generation

Ornith-1.5-35B-A3B-GGUF

Ornith

Chirp Chirp! We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement. Ornith-1.5 extends Ornith-1.0 (which was developed on top of Qwen3.5 and Gemma4 with additional continued pretraining, mid-training, and post-training) by expanding the self-improvement loop from scaffold and rollout optimization to jointly optimizing task generation, scaffold construction, and solution rollouts. Rather than relying on a fixed set of human-curated tasks and manually designed harnesses, Ornith-1.5 continuously generates new training tasks, discovers effective strategies for solving them, and improves the policy through reinforcement learning. For…

Open weights mit transformers

Model · Text generation

Ornith-1.0-9B-GGUF

Ornith

Aloha! Today, we are releasing Ornith-1.0, a self-improving family of open-source models for agentic coding. This model card documents Ornith-1.0-9B, the most lightweight member of the Ornith family, designed for efficient single-GPU deployment. Ornith-1.0-9B is a dense ~9B model (≈19 GB in bf16), so it serves comfortably on a single 80GB GPU. The recipes below stand up an OpenAI-compatible server; add --tensor-parallel-size / --tp if you want to shard across more GPUs. For a quick local test (or to script offline generation), load the model directly with Transformers. Make sure you have a recent release installed — see the Transformers installation guide; Ornith-1.0-9B requires…

Open weights mit transformers

Uncensored Qwen3.8-27B, published as GGUF quantizations with the multi token prediction (MTP) head retained and verified. Refusal behaviour has been substantially reduced, not eliminated. See Measured behaviour for the numbers. Capabilities, training data, and architecture are otherwise unchanged. - Refusal directions removed with Heretic, which co minimizes refusal count against KL divergence from the base model. No handwritten refusal removal code, no finetuning, no additional training data. - Abliteration runs at bf16 (no 4 bit quantization). the resulting LoRA is merged into the bf16 base, so the published weights are not a quantized round trip. - mtp. tensors are copied verbatim from…

Open weights apache-2.0 llama.cpp