# PowerMoE-3b by IBM Research: Open-Weight Model Source: https://savrn.com/models/powermoe-3b Markdown alternate of the page above; the site index is https://savrn.com/llms.txt --- ## Runs On What it takes to serve PowerMoE-3b (3.4B parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them. | Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits | | --- | --- | --- | --- | --- | --- | | 16-bit | 6.7 GB | 8.1 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 8-bit | 3.4 GB | 4.0 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | | 4-bit | 1.7 GB | 2.0 GB | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) (192 GB) Vultr | $1.85 | [1x H100](https://savrn.com/ai-index/pricing/gpus/h100) $1.99 · [1x MI325X](https://savrn.com/ai-index/pricing/gpus/mi325x) $2.00 | Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the [SAVRN Index](https://savrn.com/ai-index/pricing/gpus), read Oct 25, 2026. [PowerMoE-3b on every accelerator the SAVRN Index prices, at every precision](https://savrn.com/models/powermoe-3b/gpus) ## SAVRN's Notes on PowerMoE-3b Forty experts, eight awake for each token: that is how IBM Research's PowerMoE-3b holds 3.4B parameters while activating 800M per token. Memory does not care which experts are asleep, though. All 40 must be resident, so 16-bit needs 8.1 GB, 8-bit 4.0 GB and 4-bit 2.0 GB. Each lands on one MI300X with 192 GB at $1.85 an hour on demand and leaves the card mostly empty, so share it. The download is 13.5 GB, twice the 16-bit footprint. Apache 2.0 makes commercial use, modification and redistribution straightforward; keep the notices and record your changes. Two checks. The context window is 4,096 tokens, so long documents will not fit in one pass; this model is for short prompts, code and math. And the training mix includes proprietary datasets, with every published number reported by the publisher itself, so evaluate on your own workload first. SAVRN Research, 2026-09-18 ## Model Card By IBM Research, published under apache-2.0, revision 13fcb5a98001. ### Model Summary PowerMoE-3B is a 3B sparse Mixture-of-Experts (sMoE) language model trained with the Power learning rate scheduler. It sparsely activates 800M parameters for each token. It is trained on a mix of open-source and proprietary datasets. PowerMoE-3B has shown promising results compared to other dense models with 2x activate parameters across various benchmarks, including natural language multi-choices, code generation, and math reasoning. Paper: https://arxiv.org/abs/2408.13359 ### Usage Note: Requires installing HF transformers from source. #### Generation This is a simple example of how to use PowerMoE-3b model. [Read the full model card (196 words)](https://savrn.com/models/powermoe-3b/card) ## Configuration Architecture GraniteMoeForCausalLM Context length (tokens) 4,096 Layers 32 Hidden size 1,536 Feed-forward size 512 Attention heads 24 Key/value heads 8 Vocabulary size 49,152 Experts 40 Experts active per token 8 RoPE base 10,000 Stored precision float32 Model type granitemoe ## Identity and Version Repository ibm-research/PowerMoE-3b Publisher IBM Research Task Text generation Modality Text Library transformers Parameters 3.4B parameters Languages Not stated by the source Revision 13fcb5a98001438bed01cf1ac4b423751dc4c2ea First published 2024-08-14 Last updated 2024-09-24 ## Files and Weights 11 files, 13.5 GB in total. The weights are 3 files totalling 13.5 GB in safetensors. Weights3 files · 13.5 GB Configuration4 files · 27.6 KB Tokenizer2 files · 2.1 MB Documentation1 file · 3.8 KB Repository1 file · 1.5 KB Every file | File | Type | Size | SHA-256 | | --- | --- | --- | --- | | model-00001-of-00003.safetensors | Weights | 4.8 GB | 410fe4e59854 | | model-00002-of-00003.safetensors | Weights | 4.8 GB | 2c318c4e0cb9 | | model-00003-of-00003.safetensors | Weights | 3.9 GB | 948cbf3057a5 | | config.json | Configuration | 928 B | — | | generation_config.json | Configuration | 137 B | — | | model.safetensors.index.json | Configuration | 25.6 KB | — | | special_tokens_map.json | Configuration | 906 B | — | | README.md | Documentation | 3.8 KB | — | | .gitattributes | Repository | 1.5 KB | — | | tokenizer.json | Tokenizer | 2.1 MB | — | | tokenizer_config.json | Tokenizer | 4.1 KB | — | ## License and Download License apache-2.0 Access Open weights, no gate Download size 13.5 GB [Download from IBM Research](https://huggingface.co/ibm-research/PowerMoE-3b) Released by IBM Research through its official repository on Hugging Face. [Read the license](https://www.apache.org/licenses/LICENSE-2.0). ## Built From - Described by arXiv:2408.13359 ## Evaluations Each result is shown as reported, with the conditions its reporter stated. None is a SAVRN measurement. A comparison lines two results up only when their configuration, unit and setup are all stated and identical. | Benchmark | Conditions | Result | Reported by | Revision | Date | | --- | --- | --- | --- | --- | --- | | ARC | Task text-generationMetric accuracy-normComparison conditions not established | 58.1 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | BoolQ | Task text-generationMetric accuracyComparison conditions not established | 65 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | GSM8k (5 shot) | Task text-generationMetric accuracyComparison conditions not established | 25.9 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | Hellaswag | Task text-generationMetric accuracy-normComparison conditions not established | 71.5 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | MBPP | Task text-generationMetric pass@1Comparison conditions not established | 32.4 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | MMLU (5 shot) | Task text-generationMetric accuracyComparison conditions not established | 42.8 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | OpenBookQA | Task text-generationMetric accuracy-normComparison conditions not established | 41 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | PIQA | Task text-generationMetric accuracy-normComparison conditions not established | 79.1 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | Winogrande | Task text-generationMetric accuracy-normComparison conditions not established | 65 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | humaneval | Task text-generationMetric pass@1Comparison conditions not established | 20.1 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | | math (4 shot) | Task text-generationMetric accuracyComparison conditions not established | 14.8 | [ibm-research](https://huggingface.co/ibm-research/PowerMoE-3b) Publisher reported | Evaluated revision not stated | — | ## Memory Requirements | Precision | Weights in memory | | --- | --- | | As published | 13.5 GB | | 16-bit | 6.7 GB | | 8-bit | 3.4 GB | | 4-bit | 1.7 GB | Weights only, from the published parameter count; the key-value cache and runtime add to this. ## Compare PowerMoE-3b - [PowerMoE-3b vs Qwen2.5-3B-Instruct](https://savrn.com/models/compare/powermoe-3b-vs-qwen2-5-3b-instruct) - [PowerMoE-3b vs Qwen3-4B](https://savrn.com/models/compare/powermoe-3b-vs-qwen3-4b) ## Questions About PowerMoE-3b ### How much GPU memory does PowerMoE-3b need? About 8.1 GB at 16-bit and 2 GB at 4-bit: the weights (3.4B parameters) plus a working margin. A long context needs more. ### What is the cheapest GPU to run PowerMoE-3b on? At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists. ### Can I use PowerMoE-3b commercially? Yes. PowerMoE-3b is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors. ### What is PowerMoE-3b's context length? 4,096 tokens, from the maximum position embeddings in its published configuration. ## Similar Models Model · Text generation ### [MarinSkyRL-Open-MOPD-SmolLM3-3B-step-32](https://savrn.com/models/marinskyrl-open-mopd-smollm3-3b-step-32) [Open Athena](https://savrn.com/model-publishers/open-athena) This is a MarinSkyRL-native Open-MOPD student after 32 optimizer steps. It starts from the authors' mixed-domain SFT checkpoint. Student responses were scored by the authors' math, code, and instruction-following RL teachers, routed by domain. The objective uses the student's selected top-16 token IDs and a clipped policy surrogate. This is an early checkpoint, not the authors' step-200 final model. The checkpoint is an unquantized, six-file Hugging Face export of the durable MarinSkyRL globalstep32 FSDP2 checkpoint. The policy export was used for the independent step-32 evaluation. The export's model.safetensors SHA-256 is bb7326640142069bc2e1fba5f54f15e0cccb1ff861f34f318b372eaab7abaf4b.… Open weights apache-2.0 3.3B parameters 65,536 tokens transformers [View model](https://savrn.com/models/marinskyrl-open-mopd-smollm3-3b-step-32) Model · Text generation ### [prism-roleplay-1.5-small-fast](https://savrn.com/models/prism-roleplay-1-5-small-fast) [Vertex AGI](https://savrn.com/model-publishers/vertexagi) The Prism Roleplay 1.5 Small recipe on PrismML's 1-bit Bonsai-8B: a standalone model, faster and smaller than the 4-bit original, with a small quality gap A standalone model, not an adapter: load it directly with mlx-lm. It is Prism Roleplay 1.5 Small's training recipe (same data, hyperparameters and system prompt) applied to prism-ml/Bonsai-8B-mlx-1bit, a 1-bit (g128) build of Qwen3-8B. How the LoRA was merged into a 1-bit model. A plain merge re-quantizes every layer to 1 bit, and the small LoRA update disappears in the rounding (we measured it: merged held-out loss 3.415 = untuned base 3.405). So the 16 layers the LoRA touched (all their attention and MLP projections) are merged at fp16… Open weights apache-2.0 3.3B parameters 65,536 tokens mlx [View model](https://savrn.com/models/prism-roleplay-1-5-small-fast) Model · Text generation ### [amharic-bell-tts-4bit](https://savrn.com/models/amharic-bell-tts-4bit) [BmanCman](https://savrn.com/model-publishers/bmancman) This is the model card of a transformers model that has been pushed on the Hub. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Use the code below to get started with the model. Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). Open weights 3.3B parameters 131,072 tokens transformers [View model](https://savrn.com/models/amharic-bell-tts-4bit) Model · Text generation ### [gaggle-nanochat-pretrained-climbmix-20260924](https://savrn.com/models/gaggle-nanochat-pretrained-climbmix-20260924) [Castorini](https://savrn.com/model-publishers/castorini) A 3.3B-parameter decoder-only base language model, pre-trained from scratch with nanochat on one pass over ClimbMix (287B tokens). This is the causal model described in the pre-training section of (PDF). It has had no post-training (no instruction tuning or RLHF), so it continues text rather than following instructions. The same weights load three ways: with transformers (trustremotecode=True), with vLLM (through a small plugin), and with nanochat's own code from the original checkpoint files, which are unchanged. one pass of 287B tokens. The last of its 6,543 shards is held out for validation. Documents are packed into 2,048-token sequences with nanochat's BOS-aligned best-fit packing.… Open weights mit 3.3B parameters transformers [View model](https://savrn.com/models/gaggle-nanochat-pretrained-climbmix-20260924) Model · Text generation ### [Ternary-Bonsai-2-27B-vllm](https://savrn.com/models/ternary-bonsai-2-27b-vllm) [Fraser Price](https://savrn.com/model-publishers/fraserprice) Bonsai 2 27B by Prism ML, repacked for vLLM. Unofficial; not affiliated with Prism ML. It needs the prismternary vLLM plugin from fraserprice/bonsai-vllm, which has the run command, kernels and throughput numbers: Built and tested for the RTX PRO 6000 Blackwell only; other NVIDIA GPUs are untested. Problems: open an issue. - The ternary weights of the MLX pack, bit for bit: the same 2-bit codes and FP16 group scales (g128), in the same Hadamard-rotated basis. MLX's redundant per-group biases (-scale) are dropped, and the rotation signs move into config.json. - The embedding table is dequantized to BF16. - Norms and the linear-attention state path are Prism ML's own tensors from the MLX… Open weights apache-2.0 3.5B parameters 262,144 tokens vllm [View model](https://savrn.com/models/ternary-bonsai-2-27b-vllm) Model · Text generation ### [Llama-3.2-3B-Instruct](https://savrn.com/models/llama-3-2-3b-instruct) [Meta Llama](https://savrn.com/model-publishers/meta-llama) The Llama 3.2 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction-tuned generative models in 1B and 3B sizes (text in/text out). The Llama 3.2 instruction-tuned text only models are optimized for multilingual dialogue use cases, including agentic retrieval and summarization tasks. They outperform many of the available open source and closed chat models on common industry benchmarks. Model Architecture: Llama 3.2 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for… Access requested at publisher llama3.2 3.2B parameters transformers [View model](https://savrn.com/models/llama-3-2-3b-instruct) ## IBM Research [All models and datasets](https://savrn.com/model-publishers/ibm-research) ## Versions - [13fcb5a98001](https://savrn.com/models/powermoe-3b/versions/13fcb5a98001) · current 2026-09-22 ## Explore More - [All text generation models](https://savrn.com/models/tasks/text-generation) - [All models under apache-2.0](https://savrn.com/models/licenses/apache-2-0) - [Model comparisons](https://savrn.com/models/comparisons) - [The model directory](https://savrn.com/models) - [Open model prices by host](https://savrn.com/ai-index/pricing/open-models) ## Source - Repository metadata, read 2026-10-19. - [Hugging Face record](https://huggingface.co/ibm-research/PowerMoE-3b) - [How the hub is built](https://savrn.com/model-hub/methodology)