A NInfer v3 artifact of ISTA-DASLab's (2.40 bits per weight over the transformer weights) for NInfer-all, the master branch of iamwavecut/ninfer-all. Qwen3.8-Flash-Next is a 125B-parameter mixture of experts: 48 layers of 512 routed experts, 10 of them active per token, about 6B active parameters. The GGUF assigns a ggml quantization type to every tensor; this artifact keeps each tensor's ggml blocks byte for byte, so nothing is requantized, and the engine multiplies the blocks in place. Q20 is the release built for speed: it avoids the lookup-table formats. Stock NInfer builds refuse these files: the model family and the gguf formats exist only in that line; build master from commit…
Open weights
other
ninfer
Quantized GGUF of Qwen/Qwen3.8-Flash-Next. This is not an official Qwen release. The weights remain under the Qwen Community License 1.0. The file is Qwen3.8-Flash-Next-HC-Q8.gguf. - 179.551 billion parameters - 92,347,778,272 bytes - 4.11 bits per weight on average - Architecture qwen4exp, context trained at 262,144 HC-Q8 starts from a Q40ROCMI4 quant. The 200 hyperconnection up and down matrices were restored from the original BF16 tensors in Qwen revision de4b8e4d and stored as ordinary Q80. The 98 injection tensors stayed BF16. Every other tensor kept the ROCmI4 bytes. Q40ROCMI4 and Q30ROCMFPX are ROCmFPX quant types. Stock llama.cpp does not load this file. Use the ROCmFPX v2 HIP…
Open weights
other
A sensitivity-guided, mixed-precision MLX conversion of Qwen/Qwen3.8-Flash-Next, rebuilt directly from the official BF16 checkpoint with the model's matching native MTP block preserved. oQ3 uses a 3-bit affine base and spends additional precision on sensitive modules. Layer sensitivity was measured with a validated quantized calibration proxy, while every released weight was quantized from the official BF16 checkpoint. The result is a compact model with 746 higher-precision module overrides rather than a uniform 3-bit layout. The upstream tokenizer, current chat template, vision processor, generation configuration, licence, and native MTP configuration are retained. In a compatible oMLX…
Open weights
other
180B parameters
262,144 tokens
mlx
An FP16 conversion of Vontra/Qwen3.8-Flash-Next-MLX-oQ3-MTP, reprocessed with omlx-fp16-clone. Weights removed after benchmarking showed no performance benefit on M1 Ultra. FP16 was not faster than BF16 on M1 Ultra. The results show a clear split: FP16 prefill is significantly faster, but FP16 decode is slower and memory usage is higher. The omlx-fp16-clone script must promote all vision/audio passthrough tensors from BF16 to FP32 (an oQ invariant — oMLX enforces this and it cannot be skipped). This produces a checkpoint with three simultaneous dtypes: integer (quantized weights), FP16 (scales and biases), and FP32 (vision/audio tensors). The mixed-dtype layout appears to prevent oMLX's…
Open weights
other
mlx
This package brings Qwen3.8-Flash-Next to MoEspresso. Qwen3.8-Flash-Next is a 125B Mixture-of-Experts model with 6B active parameters per token and a 51B-parameter PLE/n-gram memory. On a 2021 M1 Max with a 24-core GPU and 32 GB of unified memory, MoEspresso serves this package at roughly 12–15 decode tokens/s. The local bounded configuration also scored 84.3% on a frozen 48-question generated-answer comparison, ahead of Claude Opus 4.8 xhigh at 80.3% on the same questions. The complete download occupies 145.75 GB (135.74 GiB), including the original BF16 PLE tables. MoEspresso keeps only its active working set in unified memory. The package requires MoEspresso 3.0.0 on an arm64 Apple…
Open weights
other
262,144 tokens
mlx
English | 中文 Qwen/Qwen3.8-Flash-Next converted to the.ninfer v3 format for NInfer-Offload, which runs this 125B-total / 6B-active MoE model on a 16 GB consumer GPU by keeping the routed experts in system RAM. It includes the model's MTP predictor for speculative decoding. This file only works with NInfer-Offload. It is not a GGUF or safetensors checkpoint and cannot be loaded by llama.cpp, vLLM, transformers or upstream NInfer. The 110 GB artifact is stored as NInfer's native multi-file set (at most 50 GB per file). Keep all files in one directory and pass only the entry file qwen38flashnext.ninfer to NInfer; it reads the.part-000N continuation files itself. No reassembly is needed.…
Open weights
other
ninfer
Open weights
other
97.2B parameters
A lossless NVFP4-CSF container of local-inference-lab/Qwen3.8-Flash-Next-NVFP4 at revision 6909a5bed089a48fa07e956d3915af2537de9368, the Qwen3.8-Flash-Next QAD checkpoint (NVFP4 routed experts, MXFP8 attention and shared experts). CSF stores the routed-expert block-scale planes in a compressed form. Nothing else changes: FP4 weight nibbles, every other tensor, the tensor names, dtypes, shapes and the source metadata are all byte-identical. Decoding is integer arithmetic on bytes, with no requantization or fitting. The original safetensors shards can be restored bit-exactly. - Built with trellis-quant trellisquant.losslessscalecheckpoint (commit 60feca330087, family qwen38flashnextnvfp4).…
Open weights
other
Qwen3.8-Flash-Next in the light DS4-IQ2 packaging of an IQ2 main (IQ2XXS gate/up, Q2K down padded to 768, embedded MTP block) plus an external demand-paged PLE Q41 sidecar instead of a ~95 GiB resident BF16 n-gram. Runs resident and zero-swap on a 64 GiB Apple Silicon box (measured on M5 Pro), 8K→220K context. - MTP (--mtp) adds ~+17% single-stream over MTP-off; draft acceptance ~67%. - Adaptive draft depth (engine env DS4QWEN4MTPDEPTH, default auto): drafts a 2nd token on deterministic/structured/code continuations for a further +5–6%, and falls back on free-form prose so it never regresses. Output stays autoregressive-exact (verify only commits argmax-matching drafts → zero quality…
Open weights
apache-2.0
I wanted a smaller GGUF I could run, and I wanted to know more than just whether it loaded. This is where that experiment landed. How we got here has the story, including the first Q2 export, the H64-S4 experiment, the interrupted run, and what I could and couldn't verify. You need a runtime that supports the matching Hadamard transform. A runtime that ignores the rotation metadata can compute the wrong result. The tested llama.cpp changes, source patch, and build instructions are in runtime/. The export has nine GGUF shards, about 68.9 GB in total. It is based on Qwen/Qwen3.8-Flash-Next, revision de4b8e4d43b917e7706784d8bb445c9af86a3540. I used mixed precision to spend the bits where the…
Open weights
other
gguf
Model · Image and text to text
Baa.ai
Open weights
other
gguf
Excellent 50% prune by Nickyang utilizing new RAZOR methodology. cleyesode further reduced memory requirements via nvfp4 quantization for use on Blackwell sm120 gpus. Quantization via NVIDIA/Model-Optimizer custom qwen4exp workflow: Please see post-quantization verificationreport for details. PLE isolated, full model repack for reduced memory pressure. Calibrated on curated samples from Nickyang/RazorCal: This model demonstrates very impressive retention of pre-prune / pre-quantize abilities - benchmarks in-progress. MTP is present but untested, updates will be posted as they become available. vLLM compatible via the latest branch: Example launch on 4 5060Ti 16GB GPUs (tensor parallel 4…
Open weights
other
88.1B parameters
262,144 tokens
transformers
ROCmFPX quantized GGUF of Qwen/Qwen3.8-Flash-Next — the official base model, locally requantized to 7 ROCmFPX tiers (plus vision mmproj). ROCmFPX is a proprietary quantization format (a llama.cpp-rocm extension; this batch uses GGML file types 103/104/105/110/111/114/115) — upstream llama.cpp cannot load these files. Load them with llama.cpp-rocm, a llama.cpp fork that supports all upstream GGUF formats plus ROCmFPX natively. Run directly with llama-server (OpenAI-compatible API), or pool two Strix Halo machines with ggml-rpc-server + --rpc (see Usage below). Measured with the llama.cpp-rocm RPC build (build-rpc/bin/llama-bench, sha256…
Open weights
apache-2.0
gguf
Qwen/Qwen3.8-Flash-Next packed for sushi, a native inference engine for Apple Silicon. Target: a 64 GB Mac. On 96 GB and 128 GB Macs it runs the full 1M-token context. sushi only. This pack needs sushi v1.0.0 or later on Apple Silicon with macOS 26.2 or later. It does not load in transformers, vLLM, mlx-lm or exllamav3. - --kv-quant 8 keeps the KV cache at 8 bits, and --mtp-head-kv-quant does the same for the MTP head's own KV. - --prefill-chunk 2048 caps the prompt tokens forwarded per step, which keeps the prefill peak small. - --prefix-cache-disk 10GB keeps up to 10 GB of seen prompt prefixes on the SSD, so a repeated prompt skips its prefill, across restarts too. - MTP (the model's own…
Open weights
other
28.8B parameters
1,048,576 tokens
Qwen/Qwen3.8-Flash-Next packed for sushi, a native inference engine for Apple Silicon. Sushi-3bpw there. sushi only. This pack needs sushi v1.0.0 or later on Apple Silicon with macOS 26.2 or later. It does not load in transformers, vLLM, mlx-lm or exllamav3. - MTP (the model's own draft head) and the 8-bit KV cache are on by default; --mtp-head-kv-quant stores the MTP head's own KV at 8 bits too. - A browser download of the binary is quarantined by macOS: clear it with xattr -dr com.apple.quarantine sushi-macos-arm64. By default a draft token is kept only when it is exactly what the model would have sampled. Add --mtp-typical 0.2 to also keep drafts the model finds likely enough: sampled…
Open weights
other
A byte-identical mirror of orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 plus everything needed to reproduce a production deployment on two NVIDIA DGX Spark (GB10) nodes: pre-exported FP8 PLE tables, the vLLM patches, the launcher and the measured throughput below. At a glance How the nodes are wired and what each technique does: docs/. The authoritative, complete version is deploy/qwen38.env + deploy/build/run-node.sh. Lessons learned while getting here (YaRN override shape, memory traps, NCCL NCCLIBHCA setup, never compiling on a serving node, benchmark noise) are collected in docs/lessons-learned.md. The PLE n-gram table is the largest single tensor (~95 GiB BF16). Instead of pinning it…
Open weights
apache-2.0
180B parameters
262,144 tokens
transformers
This is a quantized build of Qwen3.8-Flash-Next-Uncensored (BF16). Only the routed MoE experts are quantized; every other tensor is kept exactly as in the BF16 source. The model card below this section is the original BF16 card. Total ~174 GiB (the BF16 source is ~336 GB). Format. compressed-tensors pack-quantized, the layout llm-compressor writes: per expert projection, weightpacked (int32, eight 4-bit values each), weightscale (BF16, one per 32 consecutive input-channel weights) and weightshape. quantizationconfig in config.json targets re:.mlp\.experts\.. and ignores re:.mtp., re:.visual. and lmhead. Method. Data-free round-to-nearest, no calibration data (no GPTQ / AWQ). Each weight is…
Open weights
apache-2.0
180B parameters
262,144 tokens
transformers
Open weights
262,144 tokens
This repository contains an Ascend ModelSlim W8A8DYNAMIC conversion of pinned at revision It targets two Atlas 300I Duo cards, exposed as four Ascend 310P3 devices, with tensor parallelism across all four devices. The checkpoint preserves the base model's Qwen4 experimental architecture: 48 decoder layers, 512 routed experts, hybrid Gated DeltaNet/Qwen Sparse Attention, PLE n-gram embeddings, a vision encoder, and MTP weights. The current Ascend serving path has only been exercised for text generation. Multimodal execution is not claimed as supported, and MTP remains provisional pending the quality comparison and final public runtime freeze. The conversion used Ascend ModelSlim's…
Open weights
other
262,144 tokens
vllm
The model weights in this repo are not mine. They are the GSQ-RCO IQ2XS quantization of Qwen3.8-Flash-Next made by the Deep Algorithms and Systems Lab at ISTA, copied byte for byte from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF at revision ed59f92082b1e93c0e96d60a8b11aab089b52f09. The SHA256 of both shards and of the vision projector match their files. See SHA256SUMS. If you use these weights, cite the two ISTA-DASLab papers. The BibTeX is at the bottom of this page. My part is one 480 KB file, Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf, and the measurements below. For any other size, or for the original model without the vector, go to the ISTA-DASLab repo. For the size that stays…
Open weights
other
gguf
The model weights in this repo are not mine. They are the GSQ-RCO IQ3S quantization of Qwen3.8-Flash-Next made by the Deep Algorithms and Systems Lab at ISTA, copied byte for byte from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF at revision ed59f92082b1e93c0e96d60a8b11aab089b52f09. The SHA256 of both shards and of the vision projector match their files. See SHA256SUMS. If you use these weights, cite the two ISTA-DASLab papers. The BibTeX is at the bottom of this page. My part is one 480 KB file, Huihui-Qwen3.8-Flash-Next-refusal-direction-r.gguf, and the measurements below. For any other size, or for the original model without the vector, go to the ISTA-DASLab repo. For the faster and…
Open weights
other
gguf
F
Model · Text generation
FL
Run Qwen 3.8 Flash Next heretic-2 locally on two RTX 3090 24 GB GPUs and 128 GB RAM. Start with the GitHub quickstart and pinned runtime; the weights require its custom vLLM overlay. This checkpoint is the heretic-2 reassembly of the (AutoRound W4A16) instead of Intel's stock AutoRound target. 1,860 tok/s prefill · 89.1 tok/s decode · 2× RTX 3090 + 128 GB RAM decode after 131,072 input tokens. Each generates 2,048 tokens. The latest experimental runtime reaches the first token in 139.8 seconds with 260,096 input tokens, then generates 2,048 tokens at 86.2 tok/s. That fills the native 262,144-token window. The full request takes 163.6 seconds. The wait fell from 214.5 to 139.8 seconds…
Open weights
other
262,144 tokens
HuggingFace's download widget does not recognize KT GGUFs. Download links are in the table below. Text-only GGUF conversion of Qwen3.8-Flash-Next, quantized with an importance matrix calibrated for agentic coding. This version uses IQ4KT for the 160-column PLE table and IQ3KT for the expert-down and shared-expert-down tensors. The previous release used IQ4NL for tensors whose row widths were not multiples of 256. The new file is 80.72 GiB, compared with 88.42 GiB for the previous IQ4KT. This file requires the IQ3KT/IQ4KT 32-element row-tail changes. Runtime validation used candidate 0b7b979. The source branch and PR are pending publication. This upload is the test artifact for that…
Open weights
other
Model · Image and text to text
Jun Kim
This model was quantized using oQ (oMLX v0.7.1) mixed-precision quantization with Improved oQe Quantization (jundot/omlx#4385). A mixed-precision MLX checkpoint of Qwen/Qwen3.8-Flash-Next, an experimental preview of the architecture behind Qwen4: a mixture-of-experts vision-language model with 125B parameters and 6B activated, plus a 51B n-gram embedding and a 4B MTP module, that pairs Gated DeltaNet with Qwen Sparse Attention (QSA) and uses Gated Residual streams. It includes the language model, the n-gram embedding, the vision encoder and the multi-token prediction (MTP) head. The effective average is 3.305 bits/weight excluding the N-gram embedding table, or 3.787 bits/weight for the…
Open weights
other
180B parameters
262,144 tokens
mlx
