semsimula-ladder-owt-d384-l2-attention

Fock-PARFLM, exchange field on (arm 'attention')

One arm of the SemSimula mechanism ladder — OpenWebText, d=384, L=2: five models that differ by exactly one mechanism each, so that every gap in the ordering prices one component by construction rather than by attribution.

This arm removes: the transformer itself.

The ladder

Italic entries are pre-registered predictions, recorded before the run. The ordering is itself the prediction; any inversion is a result.

Every arm above sees exactly 532,480,000 training tokens — 32,500 steps × 32 × 512 = 532,480,000 tokens — on the same corpus, tokenizer and validation batches. No arm is trained longer than another, so a gap in the table is a gap between mechanisms and never a difference in budget.

What each gap prices

What the ladder is measuring

The models below integrate a damped mechanical system in semantic space. With the register mechanism on, the equation of motion carries a non-conservative term and is Lagrangian only in the Lagrange–d'Alembert sense:

$$m \ddot{h} = -\nabla V_\theta(h) - \gamma m \dot{h} + F_{\mathrm{rc}}(h, r) + F_\phi(h)$$

\(V_\theta\) is the pointwise scalar potential, \(V_\phi\) the pairwise one, and \(F_{\mathrm{rc}}\) the reverse channel through which the register bank acts on the token state. \(F_{\mathrm{rc}}\) is not the gradient of anything in \(h\), so it is the term that makes the trajectory non-geodesic; the damping \(-\gamma m \dot{h}\) is parallel to the velocity and changes only the speed along the path, never the path itself.

Setting \(F_{\mathrm{rc}} = 0\) (the none-norc arm) leaves every force a gradient, so \(V = V_\theta + V_\phi\) admits a Jacobi metric and the step is its damped geodesic step, up to the LayerNorm projection. That arm is the only fully conservative model in the collection, and the ladder prices what its geodesics cost.

This model

Protocol

• Corpus OpenWebText (GPT-2 BPE, 50,257); 32,500 steps × 32 × 512 = 532,480,000 tokens.
• Settled PPL is the mean of the last three 500-step evals at block 512 — not the best checkpoint.
• Shared across all arms: tokenizer, corpus, validation batches, width d=384, block 512, effective batch 32, token budget (532,480,000 tokens, identical on every arm), WSD schedule (warmup 5%, stable to 65%, decay to 100%, floor 6e-05).
• Not shared: depth (GPT-2 L=8 vs Fock L=2) and the learning rate of the GPT-2 arm (6e-04, separately tuned).
• Full protocol, pre-registered bands and run health: Depth_Ladder_and_Matched_Baseline_Protocol.md §5.7.

Reproduction

Set in Cell 0 of colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb and run Cell 0 → 1 → 1b → 1c → 1d → 2 → 3 → 4 → 5 → 5b → 6:

LADDER_MECHANISM='attention', REVERSE_CHANNEL=True, LADDER_L=2, LADDER_LR=1.2e-3

Training artefacts were written to semsimula_fock_cfc_baoab_owt_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_attn/.

Files

The code, and why this subset

code/ mirrors the layout of the companion repository, so the notebook's own sys.path lines resolve unchanged.

The closure was computed by walking the imports, not chosen by hand, so the set is reproducible and minimal: drop any file and an import fails.

Caveats

• L=2 only. Every trajectory-level claim is scoped to two layers; the L=4 rung is queued.
• OpenWebText only. How the work divides between the conservative potential and the register mechanism may be a property of the corpus; a TinyStories replication is designed but not run.
• Depth is not matched between the GPT-2 arm (L=8) and the Fock arms (L=2). The ratio column is reported for orientation, not as a like-for-like efficiency claim.
• Ablations overstate. Removing this mechanism from a trained model at inference indicated 3.75× to 3.91×; training without it gives 1.31×. Inference-time ablation overstated the mechanism by about 2.9× in PPL ratio and 4.9× in nats. Only trained-without numbers appear here.

Citation

@techreport{Gueorguiev2026SemanticSimulation,
  author = {Gueorguiev, Dimitar P.},
  title  = {Semantic Simulation: A Prescriptive Lagrangian Framework for Semantic Inference},
  note   = {Conservative Potentials, Non-Conservative Memory, and the Shared-Potential Separator},
  year   = {2026}
}

Card generated from _ladder/ladder.json on 2026-09-27.