AGILLM-GB10-1PF

Current production state — RPV16-affine

• Parameters: 1,990,038,784 conventional parameters (~1.990B)
• Training target: 600 billion tokens
• Objective: autoregressive next-token prediction only
• Position scheme: NoPE, with causal/window structure only
• Production geometry: batch 24 × sequence 2048 = 49,152 tokens/microstep
• Expert routing: exact balanced top-1, 6 experts/stage, 8,192 rows/expert
• Sparse core: paired-4:8 packed NVFP4, dominant 1280↔5120 expert matrices
• Vocabulary: 129,280-token inherited AGILLM BPE
• Output distribution: affine rank-16 normalized product vocabulary over 505×256 token factors
• Update schedule: stages 15→0 one at a time, then one head/interface phase; no full-model gradient anchor
• Optimizer: static bank of 16 stage-local PagedAdamW8bit optimizers plus one head optimizer
• Measured full-cycle training rate: 42,018.5 tok/s after decoder warmup, versus 29,060.5 tok/s for the previous flat-head local-tied architecture (+44.6%)
• Observed sparse-MMA microbenchmark peak on this Vast GB10: 992.018 TFLOP/s dense-equivalent, 99.20% of the 1 PFLOP/s target
• Verified HF recovery checkpoint: RPV16-affine step 1856 / 91,226,112 seen tokens; HF path recovery/rpv16-affine-step00001856.ckpt
• Current local shared-quant recovery: step 2112 / 103,809,024 seen tokens

AGILLM-GB10-1PF

A ~2B-parameter language-model architecture designed around NVIDIA GB10 / SM121 sparse NVFP4 from the start, rather than retrofitting a generic Transformer.

Locked v2 architecture

• 1,990,038,784 conventional parameters (0.498% below the 2B design target)
• 600,000,000,000 training-token target (~302 tokens/parameter)
• 16 sequential stages
• 6 top-1 experts per stage, 96 experts total
• model width 1280
• every expert uses 1280 -> 5120 -> 1280 SwiGLU
• expert matrices are paired 4:8 structured sparse NVFP4
• native packed mxf4nvf4 SM121 sparse MMA is the target hot path
• BF16 residual stream and FlashAttention-2 GQA
• 20 query heads / 5 KV heads / 64 head dimension
• local windows 256 -> 512 -> 1024, then four global stages
• exact AGILLM 129,280-ID BPE tokenizer bundle
• 256-dimensional token interface, projected to the 1280 compute core
• rotating stage-local updates plus one product-vocabulary head/interface phase; no full-model gradient anchor

The architecture is machine-readably frozen in ARCHITECTURE_LOCK.json.

Why the model looks strange

GB10 has an unusually large 128 GB GPU-addressable unified-memory pool but far less memory bandwidth than its sparse FP4 tensor-core issue capability would suggest. The model therefore tries to turn language-model training into large, repeatedly reusable sparse NVFP4 GEMMs instead of making the GPU shuffle small matrices and optimizer state all day.

The key expert shape, 1280 x 5120, comes from direct measurements on this GB10 and matches the existing paired-4:8 CUTLASS sparse runtime. It is not a copied Llama ratio.

1 PFLOP target

The repository reports these separately:

1. register-resident sparse NVFP4 issue rate
2. real sparse GEMM TFLOP/s
3. whole-step useful TFLOP/s
4. training tokens/s
5. seen training tokens and language-model loss

No synthetic filler FLOPs are counted as useful model work.

Verified historical starting evidence on this GB10 was 969.794609 TFLOP/s best. After stopping all unrelated GPU compute and autotuning launch geometry/register pressure, the current checked peak is 992.017989 TFLOP/s, or 99.20% of 1 PFLOP/s, at 34 accumulators, 20 blocks/SM, 128 threads and 1200 iterations. The Vast container is denied permission to raise the GPU locked clock, so the remaining gap is presently host-clock limited. Search receipts are in evidence/1pf_search/.

NVIDIA's ~1 PFLOP figure is a sparse FP4 hardware ceiling, not a claim that an end-to-end training step can execute at 1 PFLOP/s. A 2 PFLOP number, if ever reported here, must be explicitly labelled algorithmic/effective useful compute and never confused with actual single-GB10 tensor issue throughput.

Real training canary

The model has already completed real training steps on HuggingFaceFW/fineweb-edu using the bundled tokenizer, causal cross-entropy, sparse forward kernels, backward propagation and AdamW.

Initial full-gradient canary:

• 1,986,888,960 exact parameters instantiated
• 512 real text tokens
• finite CE + auxiliary loss
• finite gradient norm
• all 288 sparse expert linears executed
• optimizer step completed

A two-microbatch accumulation test reused packed weights: the second microbatch added 288 sparse kernel calls but only 9 new packs, demonstrating the intended packed-weight-residency behavior. The subsequent stage-local schedule reduced live CUDA allocation substantially while retaining a real final-CE gradient for the selected stage.

Receipts are under evidence/canary/.

Data

agillm_gb10_1pf.py contains a streaming Hugging Face dataset loader. The intended 600B run uses a weighted mixture including FineWeb-Edu, FineWeb, Wikipedia, C4, Cosmopedia, Proof-Pile-2, Dolma and code sources. Source failures are allowed to fail over rather than stall training.

Run

python3 agillm_gb10_1pf.py profile
python3 agillm_gb10_1pf.py dataset-probe --dataset fineweb-edu
python3 agillm_gb10_1pf.py train --dataset fineweb-edu --batch 1 --seq 512 --steps 2 --grad-accum 2

The repository contains the exact tokenizer bundle and the current GB10 sparse runtime binaries plus CUDA source used to rebuild/audit them.

Repository policy

Source, target contracts, tokenizer, kernel sources, reproducibility receipts and promoted checkpoints belong here. Transient profiler dumps and every throwaway checkpoint do not. Hugging Face is a model repository, not a landfill with Git LFS.

Attribution

The upstream peak-MMA reproducer under third_party/gb10_peak_mma/ retains its BSD-3-Clause license and commit metadata. AGILLM-GB10-1PF source outside third-party material is Apache-2.0 unless a file says otherwise.

Production update — 2026-09-16

The active GB10 training architecture now uses NoPE, exact 8,192 routed rows per expert for the 24×2048 production geometry, one bounded native sparse context per layer, and checkpointed exact tied-vocabulary CE to avoid materialising the full 49,152×129,280 logit matrix. This reduced CUDA reserved memory from roughly 95.8 GB to roughly 24–36 GB during early stage rotation while improving warm microstep throughput into the ~19–23k tok/s range. Sparse packs remain stable between stage changes rather than growing every microstep.

Training uses a 17-slot hardware-native update cycle: stages 15→0 one at a time, followed by one exact tied token-interface anchor. No update makes all ~1.987B parameters trainable simultaneously. The target remains 600B training tokens.

RPV16-affine promotion — 2026-09-16

The flat 129,280-way tied-softmax head was replaced by a rank-16 affine product vocabulary. Since 129,280 = 505 × 256, each mixture component predicts a normalized 505-way factor and 256-way factor; each rank uses a different bijective affine permutation of token IDs before factorization. The mixture therefore remains a proper normalized distribution over all 129,280 token IDs while reducing the output-head compute width by roughly 10.5× versus the flat classifier.

The decoder was migrated with seven head-only optimizer updates from step 896 to step 952 before local stage training resumed. Loss moved from roughly 11.787 to 11.683 during that migration. A complete post-warm 17-phase cycle sustained 42,018.5 training tok/s end-to-end. Batch-size tests placed the full-graph throughput knee at batch 24 / 8,192 routed rows per expert: batch 12 (M=4096) and batch 48 (M=16384) were both slightly slower per token.

Benchmark and migration receipts are under evidence/rpv16/.

Shared activation quantization promotion — 2026-09-16

Production now quantizes each expert input once for gate + up instead of twice. Gate and up still use separate packed sparse weights, independent FP4 weight scales, and their ordinary native sparse GEMMs; only the identical activation quantization is shared. An 8,192-row expert A/B was bitwise identical for output, dX, and all gate/up/down weight gradients. Frozen-expert forward improved from 5.634 ms to 5.216 ms (+8.0%). A matched full stage-4 replay improved median throughput from 54,956 to 56,218 tok/s (+2.30%).

After live promotion from checkpoint 2048, the same stage-14 eight-microstep window improved from 30,188.7 to 31,515.2 tok/s (+4.39%), with exact 8,192-row routing and a successful optimizer update. The first shared-quant recovery checkpoint was written at step 2112 / 103,809,024 tokens. The previously published 42,018.5 tok/s full-cycle figure predates shared-quant; a new complete-cycle figure is not claimed until measured. Receipts are under evidence/sharedquant/.