Unsloth AI

Model · Text generation

Qwen3-Coder-30B-A3B-Instruct-GGUF

Unsloth AI

Fine-tune Qwen3 (14B) for free using our Google Colab notebook! - Read our Blog about Qwen3 support: unsloth.ai/blog/qwen3 - View the rest of our notebooks in our docs here. Qwen3-Coder is available in multiple sizes. Today, we're excited to introduce Qwen3-Coder-30B-A3B-Instruct. This streamlined model maintains impressive performance and efficiency, featuring the following key enhancements: - Significant Performance among open models on Agentic Coding, Agentic Browser-Use, and other foundational coding tasks. - Long-context Capabilities with native support for 256K tokens, extendable up to 1M tokens using Yarn, optimized for repository-scale understanding. - Agentic Coding supporting for…

Model

Qwen3.8-27B-GGUF

Unsloth AI

Analysis of best Qwen3.8 GGUF providers. Unsloth Dynamic v3.0 delivers >10% top-1% better accuracy at the same size compared to every other provider. Read more Following the widespread community adoption of the Qwen3.5 and Qwen3.6 series, we are pleased to introduce Qwen3.8, the most capable generation in the Qwen open-model family to date. Built on the architectural foundation of Qwen3.5, Qwen3.8 delivers substantial gains across coding, professional work, research, and long-horizon agentic tasks. Qwen3.8-27B brings these advances to a compact, deployment-friendly dense model: a native vision-language model that understands images and videos, with flexible thinking control, designed to…

Model

Qwen3.8-27B-NVFP4

Unsloth AI

Following the widespread community adoption of the Qwen3.5 and Qwen3.6 series, we are pleased to introduce Qwen3.8, the most capable generation in the Qwen open-model family to date. Built on the architectural foundation of Qwen3.5, Qwen3.8 delivers substantial gains across coding, professional work, research, and long-horizon agentic tasks. Qwen3.8-27B brings these advances to a compact, deployment-friendly dense model: a native vision-language model that understands images and videos, with flexible thinking control, designed to carry complex, multi-step tasks through to completion with greater reliability. To achieve optimal performance, we recommend the following settings: 1. Sampling…

Model · Image and text to text

Qwen3.6-27B-NVFP4

Unsloth AI

2.5x faster throughput than other NVFP4 quants. This is an Unsloth NVFP4 quantized checkpoint calibrated on a mixture of our Unsloth dataset + UltraChat dataset. Works on a 24GB VRAM GPU. Benchmarks on 1xB200 128 concurrency. For accuracy benchmarks, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all: Read all benchmarks in our NVFP4 blog To install vLLM in a separate venv: Then to serve the 27B NVFP4 quant: Also do NOT use the Marlin backend since it's 2x slower - use the native vLLM or cute-DSL / CUTLASS / flashinfertrtllm backends! You must use the below or you will get 2x slower inference! This checkpoint…

Model · Image and text to text

Qwen3.5-9B-GGUF

Unsloth AI

You can now also fine-tune the model locally with Unsloth. - Read our Qwen3.5 fine-tuning guide here. Over recent months, we have intensified our focus on developing foundation models that deliver exceptional utility and performance. Qwen3.5 represents a significant leap forward, integrating breakthroughs in multimodal learning, architectural efficiency, reinforcement learning scale, and global accessibility to empower developers and enterprises with unprecedented capability and efficiency. For more details, please refer to our blog post Qwen3.5. WMT24++: a harder subset of WMT24 after difficulty labeling and rebalancing; we report the averaged scores on 55 languages using XCOMET-XXL. Empty…

Model · Image and text to text

Qwen3.8-Flash-Next-GGUF

Unsloth AI

As the frontier of foundation models pushes toward ever-larger parameter counts and ever-longer context windows, the question is no longer just how much we can scale, but how efficiently we can do so. Sustainable progress toward artificial general intelligence (AGI) that benefits everyone demands architectural innovation. Today, we are sharing a concrete step in that direction: Qwen3.8-Flash-Next. This experimental preview of the architecture that will underpin Qwen4 is built around a fundamental rethinking of how the core components of modern large language models (LLMs) interact at scale. The first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces: For…

Model · Image and text to text

Inkling-Small-GGUF

Unsloth AI

Inkling-Small is a general-purpose multimodal model that accepts text, image and audio inputs and generates text outputs. It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and multimodal tasks. It is released with open weights to support research, fine-tuning and integration into third-party products by downstream developers. Languages: English, with…

Model · Image and text to text

Qwen3.6-35B-A3B-NVFP4

Unsloth AI

1.56x faster throughput than other NVFP4 quants. This is an Unsloth NVFP4 quantized checkpoint calibrated on a mixture of our Unsloth dataset + UltraChat dataset. Works on a 32GB VRAM GPU. Benchmarks on 1xB200 128 concurrency. Use the 35B NVFP4 Fast version for 1.79x faster at a little less accuracy For accuracy benchmarks, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all: Read all benchmarks in our NVFP4 blog To install vLLM in a separate venv: Then to serve the 35B variant: You must use the below or you will get 2x slower inference! Also do NOT use the Marlin backend since it's 2x slower - use the native…

Model · Image and text to text

Qwen3.6-35B-A3B-GGUF

Unsloth AI

Following the February release of the Qwen3.5 series, we're pleased to share the first open-weight variant of Qwen3.6. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, offering developers a more intuitive, responsive, and genuinely productive coding experience. This release delivers substantial upgrades, particularly in For more details, please refer to our blog post Qwen3.6-35B-A3B. Empty cells (--) indicate scores not available or not applicable. For streamlined integration, we recommend using Qwen3.6 via APIs. Below is a guide to use Qwen3.6 via OpenAI-compatible API. Qwen3.6 can be served via APIs with popular inference frameworks. In…

Model · Image and text to text

Qwen3.6-27B-MTP-GGUF

Unsloth AI

Set -DGGMLCUDA=OFF for CPU/Metal. -np > 1 and --mmproj are not yet supported with MTP. - Developer Role Support so Qwen3.6 can work in Codex, OpenCode and more! - Qwen3.6 can now be run and fine-tuned in Unsloth Studio. Read our guide. Following the February release of the Qwen3.5 series, we're pleased to share the first open-weight variant of Qwen3.6. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, offering developers a more intuitive, responsive, and genuinely productive coding experience. This release delivers substantial upgrades, particularly in For more details, please refer to our blog post Qwen3.6-27B. For streamlined integration…

Model · Any to any

gemma-4-12B-it-qat-GGUF

Unsloth AI

This model ships a Multi-Token Prediction drafter at the repo root (mtp-gemma-4-12B-it.gguf, a near-lossless smart Q40). A recent llama.cpp auto-discovers it from -hf, so you do not pass --model-draft: The drafter shares the target's KV cache and does not change the output (the target verifies every drafted token). See the MTP/ folder for the other precisions and explicit usage. Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context…

Model · Image and text to text

Qwen3.6-27B-GGUF

Unsloth AI

Following the February release of the Qwen3.5 series, we're pleased to share the first open-weight variant of Qwen3.6. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, offering developers a more intuitive, responsive, and genuinely productive coding experience. This release delivers substantial upgrades, particularly in For more details, please refer to our blog post Qwen3.6-27B. For streamlined integration, we recommend using Qwen3.6 via APIs. Below is a guide to use Qwen3.6 via OpenAI-compatible API. Qwen3.6 can be served via APIs with popular inference frameworks. In the following, we show example commands to launch OpenAI-Compatible API…

Model · Image and text to text

Qwen3.6-35B-A3B-MTP-GGUF

Unsloth AI

Set -DGGMLCUDA=OFF for CPU/Metal. -np > 1 and --mmproj are not yet supported with MTP. - Developer Role Support so Qwen3.6 can work in Codex, OpenCode and more! - Qwen3.6 can now be run and fine-tuned in Unsloth Studio. Read our guide. Following the February release of the Qwen3.5 series, we're pleased to share the first open-weight variant of Qwen3.6. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, offering developers a more intuitive, responsive, and genuinely productive coding experience. This release delivers substantial upgrades, particularly in For more details, please refer to our blog post Qwen3.6-35B-A3B. Empty cells (--) indicate…

Model · Image and text to text

Qwen3.5-4B-GGUF

Unsloth AI

You can now also fine-tune the model locally with Unsloth. - Read our Qwen3.5 fine-tuning guide here. Over recent months, we have intensified our focus on developing foundation models that deliver exceptional utility and performance. Qwen3.5 represents a significant leap forward, integrating breakthroughs in multimodal learning, architectural efficiency, reinforcement learning scale, and global accessibility to empower developers and enterprises with unprecedented capability and efficiency. For more details, please refer to our blog post Qwen3.5. WMT24++: a harder subset of WMT24 after difficulty labeling and rebalancing; we report the averaged scores on 55 languages using XCOMET-XXL. Empty…

Model · Image text to video

MiniMax-H3-GGUF

Unsloth AI

Instructions further below. GGUF for MiniMax-H3, compatible on most platforms including stablediffusion.cpp and Unsloth. You can run MiniMax-H3 via Unsloth: https://github.com/unslothai/unsloth/ GGUF quantizations of MiniMaxAI/MiniMax-H3 MiniMax H3 is an omni-modal generative system that produces video with native stereo audio, up to 15 seconds at 24 FPS with 32 kHz stereo audio. Both halves of the runtime are in this repo: the denoisers and the Qwen3-VL text encoder they need. H3 ships two denoisers, and which one you load decides what the model can be given: - fl2vapruned, the H3-Base first-and-last-frame variant. Text, plus zero, one or two frames. - ref2vapruned, the reference variant.…

Model · Image and text to text

gemma-4-12b-it-GGUF

Unsloth AI

Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context window of up to 256K tokens and maintains multilingual support in over 140 languages. Featuring both Dense and Mixture-of-Experts (MoE) architectures, Gemma 4 is well-suited for tasks like text generation, coding, and reasoning. The models are available in five distinct sizes: E2B, E4B, 12B, 26B A4B, and 31B. Their diverse sizes make them deployable in environments ranging from…

Model · Any to any

gemma-4-E4B-it-qat-GGUF

Unsloth AI

This model ships a Multi-Token Prediction drafter at the repo root (mtp-gemma-4-E4B-it.gguf, a near-lossless smart Q40). A recent llama.cpp auto-discovers it from -hf, so you do not pass --model-draft: The drafter shares the target's KV cache and does not change the output (the target verifies every drafted token). See the MTP/ folder for the other precisions and explicit usage. Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context…

Model · Any to any

gemma-4-E2B-it-qat-GGUF

Unsloth AI

This model ships a Multi-Token Prediction drafter at the repo root (mtp-gemma-4-E2B-it.gguf, a near-lossless smart Q40). A recent llama.cpp auto-discovers it from -hf, so you do not pass --model-draft: The drafter shares the target's KV cache and does not change the output (the target verifies every drafted token). See the MTP/ folder for the other precisions and explicit usage. Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context…

Model · Image to image

FLUX.2-klein-4B-GGUF

Unsloth AI

This is a GGUF quantized version of FLUX.2-klein-4B. unsloth/FLUX.2-klein-4B-GGUF uses Unsloth Dynamic 2.0 methodology for SOTA performance. - Important layers are upcasted to higher precision. - Uses tooling from ComfyUI-GGUF by city96. The FLUX.2 [klein] model family are our fastest image models to date. FLUX.2 [klein] unifies generation and editing in a single compact architecture, delivering state-of-the-art quality with end-to-end inference in as low as under a second. Built for applications that require real-time image generation without sacrificing quality, and runs on consumer hardware, with as little as 13GB VRAM. FLUX.2 [klein] 4B is a 4 billion parameter rectified flow…

Model · Text to image

Z-Image-Turbo-GGUF

Unsloth AI

Welcome to the official repository for the Z-Image（造相）project! Z-Image is a powerful and highly efficient image generation model with 6B parameters. Currently there are three variants: - Z-Image-Turbo – A distilled version of Z-Image that matches or exceeds leading competitors with only 8 NFEs (Number of Function Evaluations). It offers sub-second inference latency on enterprise-grade H800 GPUs and fits comfortably within 16G VRAM consumer devices. It excels in photorealistic image generation, bilingual text rendering (English & Chinese), and robust instruction adherence. - Z-Image-Base – The non-distilled foundation model. By releasing this checkpoint, we aim to unlock the full potential…

Model · Image to video

LTX-2.3-GGUF

Unsloth AI

This is a GGUF quantized version of LTX-2.3. unsloth/LTX-2.3-GGUF uses Unsloth Dynamic 2.0 methodology for SOTA performance. - Important layers are upcasted to higher precision. - Uses tooling from ComfyUI-GGUF by city96. There are two sets of GGUF's published. One for the dev model and one for the distilled. The distilled model is optimized for few step generation, think 4-8 steps. dev on the other hand needs more steps at least 20, but you get better outputs. The distilled variant is useful as a drafting model or a refining model. In fact the workflow published below, uses the distilled lora on top of the dev model to refine the intial output. Download the mp4 in the repo and open it with…

Model · Text to image

Qwen-Image-2512-GGUF

Unsloth AI

This is a GGUF quantized version of Qwen-Image-2512. unsloth/Qwen-Image-2512-GGUF uses Unsloth Dynamic 2.0 methodology for SOTA performance. - Important layers are upcasted to higher precision. - To use the model, read our guides for ComfyUI or stable-diffusion.cpp. - Uses tooling from ComfyUI-GGUF by city96. We are excited to introduce Qwen-Image-2512, the December update of Qwen-Image’s text-to-image foundational model. You are welcome to try the latest model at Qwen Chat. Compared to the base Qwen-Image model released in August, Qwen-Image-2512 features the following key improvements: Enhanced Huamn Realism Qwen-Image-2512 significantly reduces the “AI-generated” look and substantially…

Model · Text to video

Wan2.2-TI2V-5B-GGUF

Unsloth AI

Every GGUF quantisation of Wan2.2-TI2V-5B that QuantStack/Wan2.2-TI2V-5B-GGUF publishes, plus the companion VAE, mirrored here. Unsloth Studio offers this repo as the curated one-click GGUF pick for Wan2.2 TI2V 5B, so its availability is Studio's problem rather than the repacker's: a rename or a takedown turns the pick into a 404 no client can work around. All 13 quants are mirrored, not a chosen few, because the picker lets you choose the precision. The weights are unmodified: byte for byte the files of the same name in the source repo. TI2V-5B is a 720P-only checkpoint: the supported sizes are 1280x704 and 704x1280, and its VAE has temporal compression 4, so valid frame counts are 4k+1.…

Model · Text to speech

orpheus-3b-0.1-ft

Unsloth AI

03/18/2025 – We are releasing our 3B Orpheus TTS model with additional finetunes. Code is available on GitHub: CanopyAI/Orpheus-TTS Orpheus TTS is a state-of-the-art, Llama-based Speech-LLM designed for high-quality, empathetic text-to-speech generation. This model has been finetuned to deliver human-level speech synthesis, achieving exceptional clarity, expressiveness, and real-time streaming performances. Check out our Colab (link to Colab) or GitHub (link to GitHub) on how to run easy inference on our finetuned models. Do not use our models for impersonation without consent, misinformation or deception (including fake news or fraudulent calls), or any illegal or harmful activity. By…

Model · Any to any

gemma-4-E2B-it-qat-mobile-GGUF

Unsloth AI

This model ships a Multi-Token Prediction drafter at the repo root (mtp-gemma-4-E2B-it.gguf, a near-lossless smart Q40). A recent llama.cpp auto-discovers it from -hf, so you do not pass --model-draft: The drafter shares the target's KV cache and does not change the output (the target verifies every drafted token). See the MTP/ folder for the other precisions and explicit usage. Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned variants. Gemma 4 features a context…

Model · Image to text

GLM-OCR

Unsloth AI

Join our WeChat and Discord community Use GLM-OCR's API GLM-OCR is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization. The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance…