Zenyx-V3-Base

Zenyx V3 Base (1.5B Mixture-of-Experts)

Zenyx V3 is an efficient 1.5B-parameter Mixture-of-Experts (MoE) foundation model built for low-latency inference and high throughput. It is written from scratch in JAX/Flax and trained on TPU v5e-8.

[!IMPORTANT] This is a BASE model — it is not instruction-tuned. It completes text; it does not follow instructions or hold a conversation. Prompt it with a prefix to continue ("The capital of France is"), not with a request ("Explain gravity"). Pretraining is still in progress; SFT/chat variants will follow.

Current checkpoint: step 86,400 · 59.1B tokens seen

Model Architecture

• Sparse Mixture-of-Experts: 12 routed experts + 1 shared expert, exactly 2 active per token, with a Sinkhorn transport-based gate.
• Attention: Multi-Query Attention (MQA) with 12 query heads and a single shared key/value head, plus low-rank query and output projections.
• Hyper-Connections: Sinkhorn-normalised residual routing for gradient stability at scale.
• Multi-Token Prediction (MTP): one auxiliary prediction head during training.
• Context: pretrained at up to 4,096 tokens (progressive 2,048 → 4,096). YaRN and RoPE scaling factors are precomputed so context can be extended at inference time beyond the trained length.

Benchmarks — checkpoint step 86,400 (59.1B tokens)

All tasks are evaluated with the standard base-model protocol: the model scores the log-likelihood of every candidate continuation and the highest-scoring one is taken as the answer. Nothing is generated and no output parsing is involved, so the numbers do not depend on instruction-following ability. 0-shot, full evaluation sets, no subsampling.

acc_norm normalises each continuation's log-likelihood by its length in characters, which removes the bias toward short answers; it is the headline metric wherever the task has candidates of differing lengths.

Bold marks the metric that is conventional for that task — acc_norm for HellaSwag, ARC, PIQA and OpenBookQA; acc for WinoGrande, BoolQ, SciQ and LAMBADA. The convention is applied per task, not chosen per result: it lowers the reported figure for ARC-Easy (44.53 rather than 49.54) and PIQA (59.79 rather than 60.83). Δ compares the bolded metric to the random baseline.

Both metrics

Language modelling

WikiText-2 is scored with a rolling 1024-token window at stride 512, so every counted token is predicted with at least 512 tokens of left context and each token is counted exactly once. (Scoring disjoint windows instead inflates these figures by ~15% because the leading tokens of each window are predicted from nothing.)

Trajectory across all benchmarked checkpoints

Tokens seen: 34.8B | 36.5B | 39.4B | 42.4B | 45.3B | 49.1B | 54.9B | 59.1B. The pretraining data mixture was changed partway through this sequence (code weight raised, several synthetic sources cut), so these columns do not represent a tokens-only progression.

Accuracy benchmarks (higher is better)

Language modelling (LOWER is better)

The Pile, by content type (bits/byte, LOWER is better)

Does few-shot prompting help? (MMLU by shot count)

No. More demonstrations do not help and the 5-shot result is the best of the three at both checkpoints, with 10-shot dropping to the 25% chance line (-1.47 points vs 5-shot at step 86,400, ~2.8 sigma). The same ordering appears independently at both checkpoints, so it is not a fluke of one run.

This is what a model without in-context learning looks like: using examples to infer a task is an ability that emerges later in training, and before it does, extra shots are just tokens competing for attention with the actual question. Practical consequence: prompt this model with a short direct prefix, not a long few-shot preamble.

Arithmetic

Exact-match on the answer, greedy decoding, GPT-3 prompt format (Question: What is 47 plus 21? / Answer: 68).

The model essentially cannot do arithmetic — but two-digit subtraction moved from 0.65% to 3.30% between these two checkpoints (5.1x, ~6 sigma on identical problems), which is the signature of a capability just beginning to emerge. Note that 15.5% of the pretraining mix is mathematics, yet that has bought fluency in mathematical language rather than the ability to compute.

Items are generated in-harness from a fixed seed using this prompt format, because EleutherAI/arithmetic is a loading script with no parquet branch and cannot be fetched under datasets>=3. Both checkpoints see byte-identical problems, so the comparison is exact — but these numbers are not interchangeable with published EleutherAI/arithmetic results.

Language modelling by genre (The Pile)

Bits-per-byte on each Pile domain, lower is better, scored with the same rolling 1024-token window as WikiText-2 so the numbers are directly comparable to it. This is the clearest picture of what the model is actually good at, because it measures raw prediction rather than multiple-choice ability.

The ordering here is a direct readout of the pretraining mix: code, papers and mathematics sit at the top because they are what the model has been fed most of.

Progress since the previous checkpoint

Same suite, same code, same full evaluation sets — only the checkpoint differs. Step 82,400 → 86,400 is +4.19B tokens.

Δ is on the conventional metric for each task. Bold marks a change larger than two standard errors of the difference; anything unbolded is inside the noise floor and should not be read as movement. The quoted error treats the two runs as independent, which is conservative here — they score identical items, so the true paired error is smaller.

What actually changed. A quiet interval on the accuracy benchmarks -- 8 of 15 improved (sign test p = 0.50, a coin flip, no signal either way) -- which is the expected shape between adjacent checkpoints at this token scale. Nothing here contradicts the previous interval's real gains; it simply did not add more of the same size.

• WikiText-2 bits/byte held flat (1.0267 -> 1.0279, +0.0011). LAMBADA perplexity kept improving (32.23 -> 30.35) even though its accuracy ticked down slightly (-0.25 sigma, noise).
• Arithmetic stayed at its post-correction floor: 0.150% -> 0.093%, with strict and lenient identical at every step size again, confirming this is a genuine capability gap rather than a formatting artifact -- consistent with the correction below.
• The Pile (1.5M-char cap, trajectory-comparable): 21 of 22 domains improved, 1 regressed (OpenSubtitles, +0.0032, small) -- token-weighted 1.0400 -> 1.0304 (-0.0096), the strongest single-interval improvement since step 76,800 and consistent with that checkpoint's pattern: the Pile remains the most reliable signal of real progress at this scale. A second Pile measurement, at a 400,000-char cap, backs the head-to-head model comparisons below (Github 0.5797, ArXiv 0.8738) -- do not compare those numbers to the ones above, the sampled text differs.

Second model comparison added below: XHToken/Spark-X2.5-1.7B-Base. Unlike gemma-4-E2B, where this model won on code and mathematics, Spark wins every single one of the 22 Pile domains measured and leads decisively on ARC-Challenge and LAMBADA. Spark is close in size (1.71B vs 1.58B) and vocab (131,072 vs 129,280), so this is a much more size-matched comparison than Gemma -- and the honest read is that Spark is a substantially stronger model at a similar parameter count. See the comparison table for the full breakdown.

Head-to-head vs google/gemma-4-E2B

Zenyx V3 at step 82,400 (1.58B total, MoE, ~55B tokens) against Google's 5.10B-parameter Gemma 4 E2B, which was trained on a corpus larger by roughly two orders of magnitude.

Both models were scored by one copy of the task code on the same source text, with bits/byte normalised by that text's UTF-8 byte length -- the only metric here that is legitimate across a 129,280-entry vocab and a 262,144-entry one. Perplexity is per-token and is deliberately not reported.

The Pile, all 22 domains (bits/byte, LOWER is better)

Zenyx wins 7 of 22 domains. Byte-weighted over all of them, Zenyx 1.0848 vs Gemma 1.0586 (+0.0262) -- Gemma ahead overall.

Rows marked * rest on under 250,000 bytes and carry correspondingly more noise. Excluding them, the weighted result reverses (15 domains: Zenyx 1.0341 vs Gemma 1.0430, -0.0090). That subset is not the headline: dropping the thin domains removes three of Zenyx's four worst losses while removing only two of its wins, so it flatters this model.

Science accuracy (higher is better)

What it says. Zenyx wins decisively on English technical and mathematical text -- DM Mathematics by 0.4196, its largest margin anywhere, plus ArXiv, GitHub and HackerNews. It loses heavily on multilingual and conversational text (YoutubeSubtitles, EuroParl, Ubuntu IRC), which is a category it was never trained for: the corpus is English and excludes Chinese outright, while Gemma is built multilingual. On the science tasks Zenyx is at chance on ARC-Challenge -- no measurable ability there yet -- and well behind on SciQ, though far above chance.

Those results belong together. A suite that flattered this model would not report chance-level performance where that is the truth, which is what makes the code and mathematics wins credible.

Head-to-head vs XHToken/Spark-X2.5-1.7B-Base

A second, more size-matched comparison: Spark is 1.71B params against this model's 1.58B, and its vocab (131,072) is close to this model's (129,280) -- so, unlike the Gemma comparison, tokenizer size is not a meaningful confound here either.

Same methodology as the Gemma comparison: one copy of the task code, same source text, bits/byte normalised by shared UTF-8 bytes.

The Pile, all 22 domains (bits/byte, LOWER is better)

Spark wins 0 of 22 domains -- every one of them. Byte-weighted, Zenyx 1.0798 vs Spark 0.7806 (+0.2992). Rows marked * rest on under 250,000 bytes.

ARC-Challenge and LAMBADA (higher acc is better)

What it says. Unlike Gemma, where this model won on code and mathematics, Spark leads on every single Pile domain measured, with the smallest margins on exactly the domains this model has invested in most -- GitHub (+0.1955) and DM Mathematics (+0.2055) -- and the largest on multilingual text neither model's English-only corpus should be expected to win (EuroParl, YoutubeSubtitles). Spark also shows real multi-step reasoning (26.5% vs 45.3% on ARC-Challenge, chance is 25%) and a much lower LAMBADA perplexity, both markers this model does not yet show at this token count. The honest read: at a closely matched size and vocabulary, Spark X2.5 is a substantially stronger base model. LAMBADA perplexity: Zenyx 30.35 vs Spark 4.24.

Reading these numbers. This is a partially-trained 1.5B base model, so knowledge-heavy multiple-choice tasks sit close to their random baselines — that is expected at this scale and token count. The signal to watch is the language-modelling side: LAMBADA accuracy and WikiText perplexity measure whether the model has actually learned to predict text, and those improve steadily long before multiple-choice benchmarks move. Note also that BoolQ's majority-class baseline is 62.2%, so a score near that is not evidence of comprehension.

Hardware Serving Benchmarks (NVIDIA L4, 24 GB)

Measured with the JAX/Flax serving loop: static shape pre-allocation, bucketed prefill and GPU-native sampling.

Cold shapes pay a one-off JIT compile (tens of seconds) the first time a new (prompt length, max tokens) pair is seen; warm requests are the numbers above.

Inference Example

from zenyx_v3_inference import ZenyxGenerator

generator = ZenyxGenerator(step=86400)

# Base model: give it a prefix to CONTINUE, not an instruction to follow.
print(generator.generate(
    "The capital of France is",
    max_new_tokens=80,
    temperature=0.7,
    repetition_penalty=1.15,
))

Evaluation Reproducibility

Benchmarks were produced by modal_base_evals.py on a single NVIDIA L4, scoring continuations in batches with length-bucketed padding. Task formats follow the lm-evaluation-harness conventions (prompt templates, acc / acc_norm definitions and answer-key handling), so the numbers are broadly comparable to published base-model results, though this is an independent implementation rather than a harness run.

Limitations

• Pretraining is incomplete — the model will change substantially with more tokens.
• Not instruction-tuned, not RLHF'd, and not safety-filtered. Outputs may be factually wrong, biased, or nonsensical.
• Trained predominantly on English text, code, mathematics and synthetic reasoning data; other languages are not supported.