TensorBoard scalar logs and per-loop activation diagnostics for seven ablations of a 329M looped MoE transformer. Each run changes exactly one switch of the base recipe. - 10 layers, hidden 512, 8 heads / 4 KV heads - Global batch 1024 sequences x 1024 tokens, micro-batch 16, AdamW (0.9, 0.95), weight decay 0.1, gradient clip 1.0, bf16 - Loop residual scale gamma = lambda / (H sqrt(L)) and embedding re-injection g = lambda / (t sqrt(L)), with lambda = 0.5 frozen bpsegmentlen=0 runs all H loops as one graph and backpropagates once per step. It also removes the intermediate-loop losses and the extra optimizer steps of the segmented scheme. With H=3 the segmented scheme has a single segment…
Independent publisher
Pengxiang Li
pengxiang
Video generation, Image editing, AD
Models in Library1
Datasets in Library0
Models on Hugging Face20
Followers9
Compare the published evidence
This profile connects the 1 model and dataset record shown below to their upstream cards and sources. The library counts describe SAVRN's coverage, rather than this publisher's entire output.
Reported licenses: apache-2.0. Open a record to review its specific license and conditions.
Compare model specifications, source dates, and hardware estimates on the individual records. Publisher identity verification is separate from a model's quality, deployment readiness, or suitability for your work.