semsimula-ladder-live-owt-d384-l2-attention-potential-rfqk

Fock-PARFLM with a hardened conservative exchange field (QK-normalised routing), L=2, everything live (Gen 3)

Generation 3: current. Every gradient path that applies is live. This model is the L=2 attention_potential model with its exchange field's routing hardened: the routing scores are cosine similarities times a clamped, learned per-head logit scale, the scheme the creation gate already uses, and the field's parameters get their own gradient clip at 0.3. Nothing else differs. Its routing projections had grown 5–6× unchecked. The Gen 3 collection holds the live-gradient ladder.

Result

• Settled is the mean of the last three evals (52.83, 53.05, 52.81); best 50.97 at step 31,000, final 52.81. Consecutive evals near the end differ by about ±2 PPL.
• The hardening recovers what the unhardened field lost: 2.4% better than the same model without it.
• The hardened field still adds nothing measurable: 0.4% better than the model without the field, which is within eval noise. The three L=2 models with the register path settle within 2.5% of one another, whatever else is switched on.
• Without the registers the same field is worth much more: the conservative-only model with this field and no register read settles at 54.27, 6.0% below the conservative-only model. The field and the register path largely substitute for each other (see Two attention-like routes compared below).
• At step 3,000: 113.52, against the unhardened model's 118.14 (−3.9%) and the L=2 no-exchange live model's 123.59 at the same step.
• Pre-registration: two predictions, frozen before the run: settled 52.1 or lower (called 50%), actual 52.90, a miss; the exchange field's gradient flat within ±25% of its step-5,000 value through the stable phase (called 70%), a miss: it rose about 90% (0.43 → 0.82) through the stable phase, driven by the value projection W_v, which QK-normalisation does not touch.
• Run health: 0 watchdog, 0 spikes, 1.4% clip-hit (9 of 650 logged steps, max norm 1.08, all between steps 19,700 and 26,000); the global gradient norm rose from 0.44 to 0.87 through the constant-learning-rate phase and fell to 0.71 in the decay; one Colab session, fresh run, 1.67 s/step.

This model sees exactly 532,480,000 training tokens — 32,500 steps × 32 × 512 = 532,480,000 tokens — the same budget, corpus, tokenizer and validation batches as every other model in the table. No model is trained longer than another.

What changed from the unhardened model

Two switches, both off by default and verified bit-identical to the committed code when off (logits, loss and every gradient):

• QK-normalised routing (RELAX_ATTN_QK_NORM, tag rfqk). Queries and keys are L2-normalised, so each score is a cosine times a per-head scale σ = min(exp λ, 100), initialised at 1/0.07. Eight new parameters, one per head.
• Its own gradient clip (RELAX_FIELD_CLIP = 0.3, tag rfclip0p3), as the other gates have.

Causality

SCAF leak monitor CLEAN at all seven audits (5k to 30k, 32.5k; future perturbation exactly 0, leak tax ≤ 1.9e-4 nats). The step-32,500 audit printed its honest and standard perplexities as nan; its future-perturbation test read exactly 0. Independent check on the final weights: future perturbation at five cut points exactly 0, batch independence exactly 0, prefix-only leak tax −1.4e-4 nats. The prefix-only logits differ by up to 0.05 from the full-sequence ones: shortening the input changes the reduction order. It carries no information, since any dependence on future content would have broken the bit-exact future-perturbation test. Output: causality_check_output.txt.

The hardened field

• The routing logits are bounded, as designed. The query and key weights still grew, but under normalisation the scores no longer depend on their size.
• The field's gradient still grows, 6.9× from step 500 to the best checkpoint (unhardened: 12×). Almost all of it is the value projection, which the hardening does not touch, so the pre-registered flat gradient missed.
• Switching the field off costs +51% (51.34 → 77.31) at inference, yet the model trained without it is as good: the field substitutes for other work rather than adding to it.

Two attention-like routes compared (XR)

Two mechanisms in this family move content from earlier tokens to the current one. The register path writes the tokens into a small pool of register particles and reads them back through the reverse channel. The exchange field pushes each token with an attention-like force built from earlier tokens' states. Both are measurement instruments for studying attention-like mechanisms in conservative models, not proposed architectures. Four models have or lack each route (settled perplexity, L = 2, the same 532.5M tokens; the change is against F3.1, which has neither):

Each route was then switched off in the trained model, and the loss increase measured by kind of position (100 validation sequences of 512 tokens). Terms:

• Induction position: the next token is the one that followed an earlier occurrence of the current token, so copying from earlier in the text would predict it (8% of positions).
• Novel position: the next token has not occurred earlier in the text (51%).
• Kernel: how much weight a route gives each earlier token when it updates the current one. Its effective rank counts how many independent patterns of weights it uses.

• Beside each other, the routes divide the work. The register path carries loss at every kind of position. The field serves copying: its cost is eight times larger at induction positions than at novel ones, the job of a transformer's induction heads.
• Alone, either route carries loss everywhere. The field alone has almost the same profile as the register path alone, and its kernel widens to match (effective rank 25 at the last layer, against 7.4 beside the registers and 29 for the register path alone). It still puts twice the register path's weight on the copy target. The division of labour arises from training the two routes together; it is not built into the field.
• A removal cost measures reliance, not worth. A trained model leans on whatever route it has: switching XRB-F's field off costs 1.32 nats per token, yet training with the field lowered the loss by only 0.06 nats (−6.0% in perplexity). Worth is read from the four trained models: the field alone gives three quarters of what the register path alone gives, and beside the register path it adds 0.4%. The two routes largely substitute for each other.

Full account and figure: Fock_Mechanism_v2.1_Principle_of_Operation.md, §7.4; pre-registrations, scoring and the per-route kernels and capacities: Depth_Ladder_and_Matched_Baseline_Protocol.md, §5.25.

Geometry: the most refinement-ready model with the Fock mechanism

Terms. The three gates of the flow-or-maps probe (6b-7, CG2) ask whether the layer stack behaves like one continuous flow, or like a sequence of separate maps that only work at the step size they were trained at.

• Gate 1, inertia. The velocity entering each layer is reset to zero. The rise in perplexity is what the momentum is worth to the trained model.
• Gate 2, extension. More layers at the trained step Δt, so the trajectory is integrated past the trained time T = L·Δt.
• Gate 3, refinement. The same total time T, cut into more and smaller steps (Δt = T/N, N > L). A flow reads about the same (FLOW); separate maps degrade (MAPS). Refinement invariance is the precondition for reading the trajectory as a geodesic of the learned metric, rather than of the step map at one step size.

Defined and analysed in the book, Semantic Simulation (Zenodo, latest version): §8.9 Settling, refinement and depth extension, with Proposition 44 (phase-sampled dissipation: the θ/sin θ amplification, θ = ω·Δt) and Proposition 45 (the exact damped-mode flow is refinement-invariant); Remark 62 Refinement invariance is a testable precondition; and §27.15, the replay instrument and the CG experiments.

• Refinement. At 1.5× the steps over the same integration time, this model loses +115%, against +1,274–1,342% for its L=2 siblings, and less than the L=4 model (+216%) or the conservative-only model (+143%). The hardening is the only difference from the unhardened model, so within this pair it is the cause. One seed.
• Among all models, no longer the best. The conservative-only model with the exact damped stiff-mode flow, SR2, loses 0.405 nats (+50%) on the same test against this model's 0.765 (+115%), and its per-layer flow converges as the steps are refined. This model remains the most refinement-ready with the Fock mechanism on.
• A different solution. Freezing the register bank costs +2,280%, against +17–24% for its siblings: this model relies on accumulated register content far more.
• What it rules out. Its output-layer stiff modes still cross π under refinement (θ ≈ 3.40 there), and its first-layer register increment is 7.8× the state, as large as its siblings'. Neither the π crossing nor the size of the register push explains why the L=2 Fock models fail refinement.
• Extension is worse, and every token is still strongly forced: the trajectory is forced, not geodesic.

Refinement readiness: what we are studying

A model is refinement-ready if cutting the same integration time into more, smaller steps leaves its output about the same (Gate 3 above). That is the precondition for reading its layer stack as a continuous flow rather than a fixed sequence of maps. This pair is the sharpest contrast in the programme. The unhardened model and this model differ only in the exchange field's routing hardening, yet at 1.5× the steps:

Ruled out so far, each by a pre-registered test: - the stiff modes crossing π under refinement (the hardened model's still cross); - the size of the register push (the hardened model's is 7.8× the state, as large as its siblings'); - the exchange field itself (the no-exchange model fails just as badly); - the repeated register bookkeeping alone (holding it back on the no-exchange model recovers only about 13% of the penalty).

The hypothesis under test. A Fock model is refinement-ready when its register state is accumulated: a memory built up over the layers, so more steps add more of the same and the update behaves like a flow. It fails when its register state is reset and rewritten at each layer: refinement then inserts rewrite events the model never trained with, and the update behaves like a sequence of separate maps. The conservative-only model, which has no registers, refines well (+143%).

Evidence so far (2026-10-06):

• The no-exchange and unhardened models empty nearly every register at the first layer and refill it from the current input. The hardened model empties about half and keeps the rest, retaining about 90× more of its initial register state.
• Holding the per-step register updates back under refinement helps the no-exchange model slightly but breaks the hardened one. In the hardened model those updates are part of why it refines well: its register state integrates smoothly with more steps.
• Not yet established: every model here is one seed. The decisive test, a second seed of the hardened and of the no-exchange model, is queued. Until it runs, the routing hardening is the cause only within this pair.

Pre-registration and results: Depth_Ladder_and_Matched_Baseline_Protocol.md, §5.18.

Reproduction

Set in Cell 0 of colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb and run Cell 0 → 1 → 1b → 1c → 1d → 2 → 3 → 4 → 5 → 5b → 6:

LADDER_MECHANISM='attention_potential', REVERSE_CHANNEL=True, LADDER_L=2, LADDER_LR=1.2e-3,
RELAX_GRAD_PATH='live', VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
RELAX_ATTN_QK_NORM=True, RELAX_FIELD_CLIP=0.3   # tag 'rglive', 'vplive', 'xilive', 'rfqk', 'rfclip0p3'; dt = T/L = 4

One Colab session, a fresh run of 32,500 steps (PROBE_MAX_STEPS = None). Cell 5b prints a HARDENED EXCHANGE FIELD banner (logit scale init 14.29, ceiling 100, 8 heads; the 0.3 clip group) and the gradient-path banners.

Training artefacts were written to semsimula_fock_cfc_baoab_owt_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_rglive_vplive_xilive_rfqk_rfclip0p3_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_attnpot/.

Glossary

Synchronization and binding (SY programme)

Do the couplings of these models synchronize tokens, and does that group related words? The question was put to every Gen 3 model and to the matched GPT-2 with criteria fixed before any measurement. Full results and figures: Spontaneous_Synchronization_and_Semantic_Binding_Test_Programme.md, Part II; pre-registrations and scoring: Depth_Ladder_and_Matched_Baseline_Protocol.md, §5.22.

Terms.

• Coupling: any term through which another token's state moves this token: the pair potential \(V_{\phi}\), the Poisson-mode wells, the Fock register read, the exchange field, and the context field \(\xi\), which couples tokens because it is built from the earlier ones.
• Synchronization, contraction \(S\): how much closer the token states end up because of one coupling, measured against a replay in which that coupling pushes every token alike. \(S \gt 0\): drawn together; \(S \lt 0\): spread apart.
• Binding contrast: how much more alike the states of related words are than those of unrelated words at the same distance in the text, in cosine units. An effect counts only if it is at least 0.02 and its 95% interval excludes zero.
• Noun chunk: the words of one simple noun phrase, such as the old wooden table.
• Head–dependent pair: a word and the word it grammatically depends on, such as keys and are in the keys to the cabinet are.
• Reference, coreference, antecedent: what an expression stands for. Two expressions are coreferent when they pick out the same thing, as Mary and she do in Mary said she would come; the earlier one, Mary, is the antecedent. Binding reference means linking a pronoun to the expression it stands for.
• Prediction control: what remains of a grouping after removing everything that the similarity of the words' predicted next words explains. A grouping that survives it is not just a side effect of predicting the next word.
• Coreference probe: a small attention-shaped reader trained to pick each pronoun's antecedent from the states. It is compared with always picking the nearest earlier mention, which is right 54.1% of the time (chance: 13.6%).
• Extra depth, over-synchronization: the trained model run for more steps than it was trained for, at the trained step size. Over-synchronization is the tokens drawing together (\(S \geq 0.05\)) or losing dimensions (effective rank down by 20% or more) as it fails.

Findings across the models (each call was pre-registered):

• The couplings do draw tokens together (SY-L0: hit), the pair potential modestly and the Poisson wells strongly, though toward their fixed centres rather than toward each other. None groups related words on the routes the test was designed around (SY-L1: miss).
• The context field \(\xi\) groups the words of noun chunks in every model, and the grouping survives the prediction control. It spreads the tokens apart overall while drawing each chunk's words together: clustered synchronization, as in oscillator networks that split into synchronized groups.
• The matched GPT-2 forms the same grouping, built by attention in its middle layers. The grouping is generic to language models at this budget; how it is made is not.
• No model binds reference. No probe beats the nearest-mention rule by a margin the test can resolve, in these models or in GPT-2.
• Extra depth collapses the tokens at the catastrophic stage of its failure (SY-G2: hit), not at its onset.

Measured on this model (G3′, hardened exchange field):

Theoretical foundations of the ladder

The design and theory behind every arm of the ladder, in Gen 2 and Gen 3 alike: the integrator they share, the potential they share, and the mechanisms the ladder compares. The same list appears on every ladder card.

Files

Companion notes

Caveats

• One seed. Neither the 0.4% gap to the model without the field nor the refinement result has seed variance behind it.
• No Gen 2 twin. The hardening was introduced under live gradients only.
• Parameters are not matched to the GPT-2 baseline (77.4M against 33.7M), the baseline is untuned, and its depth is four times this model's.
• OpenWebText only.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Non-Conservative Memory, and the Shared-Potential Separator},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v22 (Oct 5, 2026).
               Companion code repository (DOI 10.5281/zenodo.23152203):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-10-17.