semsimula-ladder-live-owt-d384-l2-attention-potential-rfqk-norc

Conservative-only plus the hardened exchange field, no register read (XRB-F): one attention-like route on its own, L=2, everything live (Gen 3)

Generation 3: current. Every gradient path that applies is live. This model is the conservative-only model (F3.1) with the hardened exchange field of G3′ added, and nothing else: no register read. It completes a 2 × 2 comparison of the two attention-like routes in this family, the exchange field and the register path. Like them, it is an exploratory measuring instrument for studying attention-like mechanisms in conservative models, not a proposed architecture. The Gen 3 collection holds the live-gradient ladder.

Result

• Settled is the mean of the last three evals (54.20, 54.39, 54.22); best 52.50 at step 31,000, final 54.22. Consecutive evals near the end differ by about ±1 PPL.
• The field alone is a real but weaker route. It lowers the conservative-only model's perplexity by 6.0%; the register path alone lowers it by 8.0%.
• Beside the registers it adds almost nothing: G3′ is 0.4% better than G2. The two routes largely substitute for each other.
• At step 3,000: 114.76, against 113.52 for the same field with the register path (G3′) and 127.73 for the conservative-only model without it (F3.1).
• Pre-registration: six calls, fixed before the first training step (protocol §5.25, XR part B), all hit: the field alone helps (at most 56.60; actual 54.27), it does not match the register path (above 53.12), it is worth more alone than beside the registers (6.0% against 0.4%), without registers it carries loss at novel positions (removal cost 1.16 nats, threshold 0.50) and still serves copying (2.61 nats at induction positions, threshold 1.74), and it refines better than F3.1 (+134% against +143%, narrowly).
• Run health: 0 watchdog triggers, 0 spikes; 11.1% clip-hit (72 of 650 logged steps, max norm 1.24, all between steps 11,600 and 27,950); one Colab session, fresh run, 1.60 s/step (14.5 hours).

This model sees exactly 532,480,000 training tokens — 32,500 steps × 32 × 512 = 532,480,000 tokens — the same budget, corpus, tokenizer and validation batches as every other model in the table. No model is trained longer than another.

What this model is

• The base is F3.1: the learned potential Vθ, the pair potential Vφ and the context channels ξ, integrated by the CfC/BAOAB layer step. Every force is a gradient.
• The exchange field adds, at each layer, a force on every token from the earlier tokens, built like attention: per-head query, key and value projections of the token states (8 heads of width 48), routing weights from the cosine of query and key times a learned per-head scale, and a readout matrix back to the state. The routing weights are held fixed with respect to the state being moved, so the force is the exact gradient of a potential and the model stays conservative. The routing is hardened as in G3′: normalised queries and keys, the scale capped at 100, and a gradient clip of 0.3. Trained per-head scales: 11.7 to 21.4, none near the cap.
• No register read. As in F3.1, the 32 registers are still built and written, and a small repulsion term between them (coefficient 0.05) acts during training, but nothing reads them into the tokens, so they cannot affect a prediction. This keeps the model different from F3.1 only by the field.

Causality

SCAF leak monitor CLEAN at all seven audits (5k to 30k, 32.5k; future perturbation exactly 0, leak tax ≤ 8.4e-5 nats). The step-32,500 audit printed its leak tax as nan, as every arm's final audit does; its future-perturbation test read exactly 0. Independent check on the final weights: future perturbation at five cut points exactly 0, batch independence exactly 0, prefix-only leak tax −6.8e-5 nats. The prefix-only logits differ by up to 0.04 from the full-sequence ones: shortening the input changes the reduction order. It carries no information, since any dependence on future content would have broken the bit-exact future-perturbation test. Output: causality_check_output.txt.

Two attention-like routes compared (XR)

Two mechanisms in this family move content from earlier tokens to the current one. The register path writes the tokens into a small pool of register particles and reads them back through the reverse channel. The exchange field pushes each token with an attention-like force built from earlier tokens' states. Both are measurement instruments for studying attention-like mechanisms in conservative models, not proposed architectures. Four models have or lack each route (settled perplexity, L = 2, the same 532.5M tokens; the change is against F3.1, which has neither):

Each route was then switched off in the trained model, and the loss increase measured by kind of position (100 validation sequences of 512 tokens). Terms:

• Induction position: the next token is the one that followed an earlier occurrence of the current token, so copying from earlier in the text would predict it (8% of positions).
• Novel position: the next token has not occurred earlier in the text (51%).
• Kernel: how much weight a route gives each earlier token when it updates the current one. Its effective rank counts how many independent patterns of weights it uses.

• Beside each other, the routes divide the work. The register path carries loss at every kind of position. The field serves copying: its cost is eight times larger at induction positions than at novel ones, the job of a transformer's induction heads.
• Alone, either route carries loss everywhere. The field alone has almost the same profile as the register path alone, and its kernel widens to match (effective rank 25 at the last layer, against 7.4 beside the registers and 29 for the register path alone). It still puts twice the register path's weight on the copy target. The division of labour arises from training the two routes together; it is not built into the field.
• A removal cost measures reliance, not worth. A trained model leans on whatever route it has: switching XRB-F's field off costs 1.32 nats per token, yet training with the field lowered the loss by only 0.06 nats (−6.0% in perplexity). Worth is read from the four trained models: the field alone gives three quarters of what the register path alone gives, and beside the register path it adds 0.4%. The two routes largely substitute for each other.

Full account and figure: Fock_Mechanism_v2.1_Principle_of_Operation.md, §7.4; pre-registrations, scoring and the per-route kernels and capacities: Depth_Ladder_and_Matched_Baseline_Protocol.md, §5.25.

Geometry

Terms. The three gates of the flow-or-maps probe (6b-7, CG2) ask whether the layer stack behaves like one continuous flow, or like a sequence of separate maps that only work at the step size they were trained at.

• Gate 1, inertia. The velocity entering each layer is reset to zero. The rise in perplexity is what the momentum is worth to the trained model.
• Gate 2, extension. More layers at the trained step Δt, so the trajectory is integrated past the trained time T = L·Δt.
• Gate 3, refinement. The same total time T, cut into more and smaller steps (Δt = T/N, N > L). A flow reads about the same (FLOW); separate maps degrade (MAPS). Refinement invariance is the precondition for reading the trajectory as a geodesic of the learned metric, rather than of the step map at one step size.

Defined and analysed in the book, Semantic Simulation (Zenodo, latest version): §8.9 Settling, refinement and depth extension, with Proposition 44 (phase-sampled dissipation: the θ/sin θ amplification, θ = ω·Δt) and Proposition 45 (the exact damped-mode flow is refinement-invariant); Remark 62 Refinement invariance is a testable precondition; and §27.15, the replay instrument and the CG experiments.

• Refinement. At 1.5× the steps over the same integration time this model loses +134%, between G3′ and F3.1 and far from G2 (+1,274%). The exchange field is not what makes the register-path models fail refinement. Gate 3 still grows away from the trained step count at every N, so the stack is still a set of maps, not a flow.
• It leans on momentum. Resetting the velocity entering each layer costs +113%, three times what it costs every other two-layer model, including G3′, which has the same field. This was not predicted and is not yet explained.
• 6b-9. Adding Vφ and the field to the Vθ-only step closes 0.55 of the gap to the real step (in units of the step's length), and LayerNorm 0.32. These are displacements, not reliance. Two lines of the cell's printed text are stale: it calls the exchange field "non-conservative" (this attention_potential field is a gradient at fixed routing, as the cell's own header says), and it quotes a reverse-channel figure (+31.3%) from a Gen 2 model; this model has no reverse channel. Neither affects a number.

Reproduction

Set in Cell 0 of colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb and run Cell 0 → 1 → 1b → 1c → 1d → 2 → 3 → 4 → 5 → 5b → 6:

LADDER_MECHANISM='attention_potential', REVERSE_CHANNEL=False, LADDER_L=2, LADDER_LR=1.2e-3,
RELAX_GRAD_PATH='live', VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
RELAX_ATTN_QK_NORM=True, RELAX_FIELD_CLIP=0.3   # tag 'norc', 'rglive', 'vplive', 'xilive', 'rfqk', 'rfclip0p3'; dt = T/L = 4

One Colab session, a fresh run of 32,500 steps (PROBE_MAX_STEPS = None). The run log reports reverse channel: OFF in its header.

Training artefacts were written to semsimula_fock_cfc_baoab_owt_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_rglive_vplive_xilive_rfqk_rfclip0p3_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_attnpot/.

Glossary

Synchronization and binding (SY programme)

Do the couplings of these models synchronize tokens, and does that group related words? The question was put to every Gen 3 model and to the matched GPT-2 with criteria fixed before any measurement. Full results and figures: Spontaneous_Synchronization_and_Semantic_Binding_Test_Programme.md, Part II; pre-registrations and scoring: Depth_Ladder_and_Matched_Baseline_Protocol.md, §5.22.

Terms.

• Coupling: any term through which another token's state moves this token: the pair potential \(V_{\phi}\), the Poisson-mode wells, the Fock register read, the exchange field, and the context field \(\xi\), which couples tokens because it is built from the earlier ones.
• Synchronization, contraction \(S\): how much closer the token states end up because of one coupling, measured against a replay in which that coupling pushes every token alike. \(S \gt 0\): drawn together; \(S \lt 0\): spread apart.
• Binding contrast: how much more alike the states of related words are than those of unrelated words at the same distance in the text, in cosine units. An effect counts only if it is at least 0.02 and its 95% interval excludes zero.
• Noun chunk: the words of one simple noun phrase, such as the old wooden table.
• Head–dependent pair: a word and the word it grammatically depends on, such as keys and are in the keys to the cabinet are.
• Reference, coreference, antecedent: what an expression stands for. Two expressions are coreferent when they pick out the same thing, as Mary and she do in Mary said she would come; the earlier one, Mary, is the antecedent. Binding reference means linking a pronoun to the expression it stands for.
• Prediction control: what remains of a grouping after removing everything that the similarity of the words' predicted next words explains. A grouping that survives it is not just a side effect of predicting the next word.
• Coreference probe: a small attention-shaped reader trained to pick each pronoun's antecedent from the states. It is compared with always picking the nearest earlier mention, which is right 54.1% of the time (chance: 13.6%).
• Extra depth, over-synchronization: the trained model run for more steps than it was trained for, at the trained step size. Over-synchronization is the tokens drawing together (\(S \geq 0.05\)) or losing dimensions (effective rank down by 20% or more) as it fails.

Findings across the models (each call was pre-registered):

• The couplings do draw tokens together (SY-L0: hit), the pair potential modestly and the Poisson wells strongly, though toward their fixed centres rather than toward each other. None groups related words on the routes the test was designed around (SY-L1: miss).
• The context field \(\xi\) groups the words of noun chunks in every model, and the grouping survives the prediction control. It spreads the tokens apart overall while drawing each chunk's words together: clustered synchronization, as in oscillator networks that split into synchronized groups.
• The matched GPT-2 forms the same grouping, built by attention in its middle layers. The grouping is generic to language models at this budget; how it is made is not.
• No model binds reference. No probe beats the nearest-mention rule by a margin the test can resolve, in these models or in GPT-2.
• Extra depth collapses the tokens at the catastrophic stage of its failure (SY-G2: hit), not at its onset.

Measured on this model (XRB-F, the exchange field without the register path; descriptive, no SY call was fixed for it):

Theoretical foundations of the ladder

The design and theory behind every arm of the ladder, in Gen 2 and Gen 3 alike: the integrator they share, the potential they share, and the mechanisms the ladder compares. The same list appears on every ladder card.

Files

Companion notes

Caveats

• One seed. None of the gaps between the four models has seed variance behind it; settled values vary by about ±1 PPL between evaluations.
• Only the hardened field was tested alone. A run without the hardening would separate its effect.
• Parameters are not matched to the GPT-2 baseline (77.2M against 33.7M), the baseline is untuned, and its depth is four times this model's.
• OpenWebText only.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Non-Conservative Memory, and the Shared-Potential Separator},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v22 (Oct 5, 2026).
               Companion code repository (DOI 10.5281/zenodo.23152203):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-10-17.