Sarvam-30B, 2-bit GGUF
An 11 GB GGUF of Sarvam-30B that runs on a CPU with about 12 GB of free RAM. It scores 81.7% on MMLU, against 85.1 published for the full model. Sarvam-30B is a mixture-of-experts model: 32 billion weights, about 2.4 billion of them active per token, which is why a CPU can run it.
Files
Use the IQ2_M file. The IQ1_M file is here for comparison: it saves 3 GB but loses much more quality.
For the larger Sarvam-105B, see Aaravomega/sarvam-105b-GGUF-IQ2_M-IQ1_M (2-bit and 1-bit) and Aaravomega/sarvam-105b-GGUF-Q8_0 (8-bit).
How to run
You need a recent llama.cpp build with Sarvam support (architecture bailingmoe2; tested with release b11731) and about 12 GB of free RAM.
hf download Aaravomega/sarvam-30b-GGUF-IQ2_M-IQ1_M sarvam-30b-IQ2_M-indic.gguf --local-dir .
llama-server -m sarvam-30b-IQ2_M-indic.gguf -c 16384 --jinja
Then open http://127.0.0.1:8080, or call the OpenAI-compatible API at /v1/chat/completions. Sarvam-30B is a reasoning model: it thinks before answering, typically for 1,000 to 2,000 tokens, and in our tests about one answer in four in Indian languages ran past 4,096. Allow 8,192 output tokens or more.
Quality
The reference is the 8-bit Q8_0 file these were made from, which is nearly identical to the original model. KL divergence measures how far a file's next-token probabilities drift from the 8-bit model's (lower is better), over 40 chunks of 512 tokens each of English (wikitext-2) and Hindi (Hindi Wikipedia).
| Test |
IQ2_M |
IQ1_M |
| KL divergence, English / Hindi |
0.320 / 0.231 |
0.935 / 0.747 |
| Perplexity increase vs 8-bit, English / Hindi |
+15.5% / +7.1% |
+92.5% / +58.2% |
| Same top token as 8-bit, English / Hindi |
76.0% / 77.5% |
59.9% / 61.0% |
Quiz accuracy on the first 400 questions of each, scored by likelihood with llama-perplexity (random guessing scores 25%):
| Quiz |
8-bit |
IQ2_M |
IQ1_M |
| HellaSwag (English) |
69.25% |
65.25% |
55.75% |
| ARC-Challenge, Hindi (sarvamai/arc-challenge-indic) |
36.75% |
36.50% |
33.25% |
| ARC-Challenge, English |
47.25% |
43.75% |
35.50% |
Hindi holds up better than English in both files, which fits the Indian-language importance matrix used to make them.
Against Sarvam's published score
Sarvam reports 85.1 on MMLU for the full model, run in thinking mode at temperature 1.0 with up to 65,536 tokens per answer. We asked a random sample of MMLU test questions the same way, with lower caps on answer length:
| File |
Questions |
Answer cap |
Score |
95% range |
No answer |
Score on answered questions |
| IQ2_M |
300 |
16,384 tokens |
81.7% |
77–86% |
3 |
82.5% (245 of 297) |
| IQ2_M |
100 |
4,096 tokens |
79.0% |
70–86% |
8 |
85.9% (79 of 92) |
| IQ1_M |
50 |
4,096 tokens |
42.0% |
29–56% |
25 |
84.0% (21 of 25) |
With room to finish, IQ2_M answers 99% of the questions and scores 81.7%, 3.4 points under the full model's published 85.1, which lies inside this sample's 95% range. Half its answers take under 1,107 tokens, but one in eight (12%) needs more than 4,096. IQ1_M is usually right when it answers, but half the time its thinking never ends.
Sarvam reports 80.0 on MMLU-Pro, a harder version with 10 choices per question, with the same settings. We asked a random sample of 200 of its test questions the same way:
| File |
Questions |
Answer cap |
Score |
95% range |
No answer |
Score on answered questions |
| IQ2_M |
200 |
16,384 tokens |
66.5% |
60–73% |
14 |
71.5% (133 of 186) |
Here the gap to the full model is larger: 13.5 points, well outside the sampling error. MMLU-Pro makes the model think longer (median 2,381 tokens against 1,107 on MMLU), and one answer in ten hit the 16,384-token cap. The weakest subjects were law (5 of 15) and philosophy (4 of 12); maths (27 of 32) and economics (18 of 19) held up best.
Sarvam reports 76.8 on MILU, a knowledge benchmark in 11 Indian languages, without giving its settings. We asked 165 MILU test questions (15 per language) the same way as MMLU:
| File |
Questions |
Score |
95% range |
No answer within 4,096 tokens |
Score on answered questions |
| IQ2_M |
165 |
52.1% |
45–60% |
44 |
71.1% (86 of 121) |
On MILU the model thinks much longer than on MMLU (median 1,918 tokens against 1,095), so 27% of answers ran out of budget. The score on answered questions flatters it slightly, because the hardest questions are the likeliest to run long, so we can't yet say how close IQ2_M is to the full model on MILU.
How these were made
- Source: the 8-bit
sarvam-30B-Q8_0.gguf from Sumitc13/sarvam-30b-GGUF.
- Importance matrix: from deepak-p-yadav/sarvam-30b-IQ2_M-indic, computed on 128 chunks of text in 22 Indian languages. It tells the quantizer which weights matter most for that text.
- Tool:
llama-quantize from llama.cpp b11731. No training or fine-tuning.
Compared with deepak-p-yadav's IQ2_M, which uses the same importance matrix and the same bit layout but was made from Sarvam's official 4-bit Q4_K_M file: this file stays closer to the 8-bit model (KL divergence 0.320 vs 0.349 in English, 0.231 vs 0.266 in Hindi; Hindi perplexity +7.1% vs +14.1%). Quiz scores are a tie.
Rare characters display correctly
Sarvam's tokenizer stores characters it has no token for as raw bytes, using tokens <0x00> to <0xFF>. llama.cpp's converter currently labels those tokens as normal text, so Sarvam GGUFs print things like ₹<0xE2><0x80><0xAF>84 instead of ₹ 84. In these files the 256 byte tokens are labelled correctly, so emoji, thin spaces and rare symbols come out right; tokenization is unchanged. To repair another Sarvam GGUF, see this discussion.
Limitations
- Low-bit files lose quality, more in English than in Hindi on our tests; check them on your own task.
- The samples are small: about ±4.5 points for MMLU (300 questions), ±6.5 for MMLU-Pro (200) and ±8 for MILU (165). Answers were capped at 16,384 tokens for MMLU and MMLU-Pro and 4,096 for MILU, below Sarvam's 65,536; MILU and MATH-500 runs at 16,384 tokens are in progress.
- The quizzes use likelihood scoring, which is harsh on chat models; even the 8-bit model scores only about 37% on Hindi ARC.
Credits and license
Sarvam-30B is by Sarvam AI, released under Apache 2.0, which also covers these files. Thanks to Sumitc13 for the 8-bit conversion and to deepak-p-yadav for the Indian-language importance matrix.
