Zenyx-MCEN-Delta-Mamba

Zenyx-MCEN-Delta-Mamba (565M) — work in progress

A ~565M-parameter language model that interleaves Mamba-3 (MIMO) and Gated DeltaNet-2 (GDN-2) sequence mixers with two MCEN stages (a reversible fixed-point "reasoning" core with a delta-rule memory), trained from scratch on the Zenyx-V3 pretraining mixture with the DeepSeek tokenizer. This repository holds training checkpoints, logs and diagnostics of an ongoing run; it is not a finished release.

[!IMPORTANT] Snapshot. Every number on this card was measured at step 4,100 (0.54B of a planned 1.5B training tokens, 36% of 11,444 steps), on 2026-10-14. Training continues, so later checkpoints in checkpoints/main565m/ are newer than this card. This is a base model: it completes text and does not follow instructions.

Architecture (preset z565m)

Training

Data mixture — 121 sources in the frozen manifest (data/manifest_v2.json, created 2026-10-10 06:04:26, hash 9454e840c2b9f032…); grouped shares:

Benchmarks (step 4,100)

The comparison models are the same author's: Zenyx-v2-base (85M unique parameters, 32 effective layers, 32k vocabulary) and Zenyx-V3-Base (1.58B-parameter MoE, ~81B training tokens). They are far larger or far longer-trained than this snapshot, so read the comparisons as "where does this early checkpoint stand", not as a like-for-like contest. MCEN and V3 numbers are on a Kaggle T4 (fp16); V2 in fp32.

The Pile, by domain (bits per byte, LOWER is better)

Text: NeelNanda/pile-10k (pinned revision 127bfedcd5), documents grouped by pile_set_name, capped at 1.5M characters per domain, scored with a rolling 1024-token window at stride 512 so every token is scored exactly once with ≥ 512 tokens of left context; bits/byte is normalised by the UTF-8 bytes of the source text, so it is comparable across tokenizers.

MCEN is lower (better) than V2 in 19 of 22 domains; it is higher (worse) in: PhilPapers (+0.050), EuroParl (+0.344), YoutubeSubtitles (+0.381). The sample sizes of EuroParl, Ubuntu IRC, YoutubeSubtitles, Enron, PhilPapers, NIH ExPorter and HackerNews are small (each under ~250 KB), so those rows are noisy.

SciQ

acc 55.30% / acc_norm 49.80% (n = 1000, chance 25%); standard error about 1.6 points. For reference, Zenyx-v2-base scores 52.10% / 47.20% with the same code and Zenyx-V3-Base 79.90% / 72.70%.

Zenyx-Fact1000 (internal)

A frozen, 1000-question, four-way factual / common-sense completion benchmark (chance 25%; the model must rank the correct completion above three same-type distractors by log-likelihood; sha256 843416524fb39c5f…; not yet published). A "prior-only" control scores the same candidates after the neutral context "The answer is" — a score near it means the model is merely preferring common words.

Task suite (the 19 tasks + extras used for Zenyx-V3-Base)

Being run on this checkpoint (step 4,100) at the time of writing; the results will be added to this card when it finishes.

Generation

Greedy decoding frequently falls into repetition loops; a mild repetition penalty (1.1) reduces them, but the model still invents facts and drifts off topic on open-ended prompts, as expected this early in training. Treat it as a research checkpoint, not an assistant.

What is and is not established about MCEN itself

The MCEN stages are the novel component, and this run does not yet show that they help. At step 1,600 their output was measured at 0.33% (first stage) and 0.03% (second stage) of the magnitude of the residual stream they write into, and switching either stage off changed the held-out loss by about 1e-4 nats or less. As of 2026-10-14 a matched ablation (the same small backbone with and without MCEN stages, two seeds each) is still running; until it reports, the benchmark numbers above should be attributed to the whole model (Mamba-3 + GDN-2 backbone, tokenizer and data), not to MCEN.

Repository contents

• checkpoints/main565m/step_* — checkpoints (LATEST points to the newest; only the last two are kept) · data/manifest_v2.json — frozen data manifest · logs/, evals/, forensics/ — training logs and diagnostics.
• The training and inference code is not in this repository.

Limitations

• Early, still-training base model: knowledge is shallow (see Fact1000) and generation is unreliable.
• English / code / mathematics oriented; weak on multilingual and conversational text (see EuroParl, YoutubeSubtitles).
• Not instruction-tuned, not safety-tuned.
• Benchmarks are the author's own implementations, not a harness run; Fact1000 is internal and unpublished.