Merged checkpoint produced by the family-aware Delta-P2S experiment package.
Open-weight model · Text generation
Vigyan-3B-4x-MoE
by Shreyansh singh shreyansh12183/Vigyan-3B-4x-MoE
Vigyan-3B-4x-MoE is an open-weight model for text generation from Shreyansh singh, released under Creative Commons Attribution-NonCommercial 4.0. It has 12.8B parameters and a 32,768-token context. At 16-bit it needs about 30.8 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index. It draws 2 downloads a month.
Vigyan-3B 4× MoE is an experimental Sparse Mixture of Experts model upcycled from 4 specialized domain LoRA adapters (Science, Technology, Engineering, Mathematics).
Parameters12.8B
Context32,768
Weights25.7 GB
Licensecc-by-nc-4.0
AccessOpen weights
Monthly Downloads2
Runs On
What it takes to serve Vigyan-3B-4x-MoE (12.8B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 25.7 GB | 30.8 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 12.8 GB | 15.4 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 6.4 GB | 7.7 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 7, 2026.
Vigyan-3B-4x-MoE on every accelerator the SAVRN Index prices, at every precision
Model Card
Vigyan-3B 4× MoE is an experimental Sparse Mixture of Experts model upcycled from 4 specialized domain LoRA adapters (Science, Technology, Engineering, Mathematics). It was assembled to evaluate whether post-hoc expert stitching on compact language models ($\le 3\text{B}$) can deliver multi-domain specialization at single-expert inference latency. - shreyansh12183/vigyan-3b-adapter-science - shreyansh12183/vigyan-3b-adapter-technology - shreyansh12183/vigyan-3b-adapter-engineering - shreyansh12183/vigyan-3b-adapter-mathematics 1. Delimiter Blindness: Without general conversational anchor replay during adapter training, the model degrades into token echo loops when receiving general chat…
Excerpt from the card by Shreyansh singh, licensed cc-by-nc-4.0.
Configuration
- Architecture
- Qwen2MoeForCausalLM
- Context length (tokens)
- 32,768
- Layers
- 36
- Hidden size
- 2,048
- Feed-forward size
- 11,008
- Attention heads
- 16
- Key/value heads
- 2
- Vocabulary size
- 151,936
- Experts
- 4
- Experts active per token
- 1
- Model type
- qwen2_moe
Identity and Version
- Repository
- shreyansh12183/Vigyan-3B-4x-MoE
- Publisher
- Shreyansh singh
- Task
- Text generation
- Modality
- Text
- Library
- Not stated by the source
- Parameters
- 12.8B parameters
- Languages
- moe
- Revision
- 4b58eaaea515b46e321b1c4d1e4891efeac1d19d
- First published
- 2026-10-05
- Last updated
- 2026-10-06
Files and Weights
16 files, 25.7 GB in total. The weights are 6 files totalling 25.7 GB in safetensors.
Weights6 files · 25.7 GB
Configuration4 files · 85.9 KB
Tokenizer2 files · 11.4 MB
Documentation2 files · 4.1 KB
Other1 file · 2.5 KB
Repository1 file · 1.6 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model-00001-of-00006.safetensors | Weights | 5.0 GB | 286d2e2de4a5 |
| model-00002-of-00006.safetensors | Weights | 5.0 GB | 7e4d85a14afb |
| model-00003-of-00006.safetensors | Weights | 5.0 GB | 8cdeb7185ac5 |
| model-00004-of-00006.safetensors | Weights | 5.0 GB | 4c1a218a46c1 |
| model-00005-of-00006.safetensors | Weights | 5.0 GB | c339ad6f3dee |
| model-00006-of-00006.safetensors | Weights | 850.1 MB | b08eeae4c87c |
| config.json | Configuration | 1.8 KB | — |
| mergekit_moe_config.yml | Configuration | 1.5 KB | — |
| model.safetensors.index.json | Configuration | 81.5 KB | — |
| moe_head_to_head_100_battle_test.json | Configuration | 1.1 KB | — |
| HEAD_TO_HEAD_BATTLE_TEST.md | Documentation | 1.3 KB | — |
| README.md | Documentation | 2.8 KB | — |
| chat_template.jinja | Other | 2.5 KB | — |
| .gitattributes | Repository | 1.6 KB | — |
| tokenizer.json | Tokenizer | 11.4 MB | e98bea228083 |
| tokenizer_config.json | Tokenizer | 691 B | — |
License and Download
- License
- cc-by-nc-4.0
- Access
- Open weights, no gate
- Download size
- 25.7 GB
Download from Shreyansh singh
Released by Shreyansh singh through its official repository on Hugging Face. Read the license.
Built From
- Derived from Qwen/Qwen2.5-3B-Instruct
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 25.7 GB |
| 16-bit | 25.7 GB |
| 8-bit | 12.8 GB |
| 4-bit | 6.4 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About Vigyan-3B-4x-MoE
How much GPU memory does Vigyan-3B-4x-MoE need?
About 30.8 GB at 16-bit and 7.7 GB at 4-bit: the weights (12.8B parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run Vigyan-3B-4x-MoE on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use Vigyan-3B-4x-MoE commercially?
Not without separate permission. Vigyan-3B-4x-MoE is released under Creative Commons Attribution-NonCommercial 4.0. CC BY-NC 4.0 permits sharing and adapting with credit for non-commercial purposes only. Commercial use needs separate permission from the rights holder.
What is Vigyan-3B-4x-MoE's context length?
32,768 tokens, from the maximum position embeddings in its published configuration.
Similar Models
This is a modified version of google/translategemma-12b-it optimized for deployment with vLLM. No retraining was performed. Only configuration files and the chat template were modified. Model weights are identical to the original. As of 2025-01-29, vLLM does not natively support TranslateGemma's custom structured input format. See vllm-project/vllm#32446 for the upstream tracking issue. Until that is merged, this repo provides a workaround by modifying configuration files to make TranslateGemma compatible with vLLM's standard chat API. This conversion is based entirely on the work done by Infomaniak-AI/vllm-translategemma-4b-it. The same conversion approach was applied to the 12B model.…
"Experimental Mistral Nemo Mix" using... "Model Stock merge with Mergekit". This is a mix of some known "RP Uncensored" models. I'm not responsible of the use of this. From the "errors" without head, to "the simpler excellence". This model is "quality". "Version V6" leaked stability, "Version V9" leaked creativity. This model has "pruned weights" full of "pure stable creativity", from original "base" models "tunned for dark RP". No more "mixes, from mixed mixes", without the "fixmistralregex" patch problem. All done with patience in mind... Have fun! TheDrummer/UnslopNemo-12B-v4.1 as base. The Tokenizer & Embedding Sizes (131072) are clean + It's "supposed to" support now a full "128K…
"Experimental Mistral Nemo Mix" using... "Model Stock merges with Mergekit". This is a mix of some known "RP Uncensored" models. I'm not responsible of the use of this. After noticing some "errors" on my previous versions including the "RemiX" one. I decided to go "simplier". "Model Stock" cleanly mixes and filters out the noise from each model. natong19/Mistral-Nemo-Instruct-2407-abliterated # Base! natong19/Mistral-Nemo-Instruct-2407-abliterated # Base! natong19/Mistral-Nemo-Instruct-2407-abliterated # Base! natong19/Mistral-Nemo-Instruct-2407-abliterated # Base! natong19/Mistral-Nemo-Instruct-2407-abliterated # Base!.\mergekitconfig1 # Intelligence.\mergekitconfig2 # RP &…
A 12B model that rewrites AI-written drafts (emails, essays, reports, forum posts; English and Chinese) so they read like a person wrote them. It is trained to keep every number, unit, date, name and quote, and to add nothing. It runs locally. No AI detector was used anywhere in training. App: download the.dmg (Mac with Apple silicon) or the Windows installer from Releases (current: app 0.3.1). On first run it suggests one of five sizes for your memory (Q80, Q6K, Q4KM, Q3 or 2-bit, down to 8 GB machines; app 0.3.0 and later) and downloads it once; after that it works offline. Sizes and quality notes: Quantized versions. Command line, for long documents and agents: pipx install…
Qwen1.5-MoE is a transformer-based MoE decoder-only language model pretrained on a large amount of data. For more details, please refer to our blog post and GitHub repo. Qwen1.5-MoE employs Mixture of Experts (MoE) architecture, where the models are upcycled from dense language models. For instance, Qwen1.5-MoE-A2.7B is upcycled from Qwen-1.8B. It has 14.3B parameters in total and 2.7B activated parameters during runtime, while achieving comparable performance to Qwen1.5-7B, it only requires 25% of the training resources. We also observed that the inference speed is 1.74 times that of Qwen1.5-7B. The code of Qwen1.5-MoE has been in the latest Hugging face transformers and we advise you to…