Matched GPT-2 baseline
One arm of the SemSimula mechanism ladder — OpenWebText, d=384, L=2: five models that differ by exactly one
mechanism each, so that every gap in the ordering prices one component by
construction rather than by attribution.
This arm removes: the reference architecture.
The ladder
Italic entries are pre-registered predictions, recorded before the run.
The ordering is itself the prediction; any inversion is a result.
Every arm above sees exactly 532,480,000 training tokens — 32,500 steps × 32 × 512 = 532,480,000 tokens — on the same corpus, tokenizer and validation batches. No arm is trained longer than another, so a gap in the table is a gap between mechanisms and never a difference in budget.
What each gap prices
| gap |
prices |
value |
| Matched GPT-2 baseline → Fock-PARFLM, exchange field on |
what the transformer has that this architecture does not |
13.70 PPL (+27.5%), measured |
| Fock-PARFLM, exchange field on → Fock-PARFLM, exchange field as a potential |
the price of conservativity |
17.39 PPL (+27.4%), measured |
| Fock-PARFLM, exchange field as a potential → Fock-PARFLM, no exchange field |
the exchange field |
-13.92 PPL (+-17.2%), measured |
| Fock-PARFLM, no exchange field → Conservative-only control — the Fock mechanism off |
the Fock mechanism (the register-to-token path) |
20.95 PPL (+31.3%), measured |
| Fock-PARFLM, no exchange field → Fock-SPLM — no Vφ, Fock mechanism on |
Vφ / PARF, with the register mechanism present |
pending |
| Conservative-only control — the Fock mechanism off → Multi-ξ SPLM — no Vφ, no Fock |
Vφ / PARF, without it — the 2×2's second reading |
pending |
| Fock-SPLM — no Vφ, Fock mechanism on → Multi-ξ SPLM — no Vφ, no Fock |
the Fock mechanism again, on a Vφ-free base (replication) |
pending |
What the ladder is measuring
The models below integrate a damped mechanical system in semantic space.
With the register mechanism on, the equation of motion carries a
non-conservative term and is Lagrangian only in the
Lagrange–d'Alembert sense:
$$m \ddot{h} = -\nabla V_\theta(h) - \gamma m \dot{h} + F_{\mathrm{rc}}(h, r) + F_\phi(h)$$
\(V_\theta\) is the pointwise scalar potential, \(V_\phi\) the pairwise one,
and \(F_{\mathrm{rc}}\) the reverse channel through which the register bank
acts on the token state. \(F_{\mathrm{rc}}\) is not the gradient of anything
in \(h\), so it is the term that makes the trajectory non-geodesic; the
damping \(-\gamma m \dot{h}\) is parallel to the velocity and changes only
the speed along the path, never the path itself.
Setting \(F_{\mathrm{rc}} = 0\) (the none-norc arm) leaves every force a
gradient, so \(V = V_\theta + V_\phi\) admits a Jacobi metric and the step is
its damped geodesic step, up to the LayerNorm projection. That arm is
the only fully conservative model in the collection, and the ladder prices
what its geodesics cost.
This model
|
|
| architecture |
GPT-2 decoder, d=384, L=8, 6 heads, d_ff=1536, tied embeddings |
| parameters |
33,691,776 |
| training tokens |
532,480,000 — the same on every arm |
| learning rate |
0.0006 |
| settled PPL |
49.81 |
| best PPL |
49.76 (step 32,500) |
| final PPL |
49.76 |
| ratio to matched GPT-2 |
1.000× |
| pre-registration |
below 54.59, point 49.5, band 49.0-50.0 — HIT, error +0.26 |
| run health |
reference arm |
Protocol
- Corpus OpenWebText (GPT-2 BPE, 50,257); 32,500 steps × 32 × 512 = 532,480,000 tokens.
- Settled PPL is the mean of the last three 500-step evals at block 512 — not the best checkpoint.
- Shared across all arms: tokenizer, corpus, validation batches, width d=384, block 512, effective batch 32, token budget (532,480,000 tokens, identical on every arm), WSD schedule (warmup 5%, stable to 65%, decay to 100%, floor 6e-05).
- Not shared: depth (GPT-2 L=8 vs Fock L=2) and the learning rate of the GPT-2 arm (6e-04, separately tuned).
- Full protocol, pre-registered bands and run health:
Depth_Ladder_and_Matched_Baseline_Protocol.md §5.4.
Reproduction
Set in Cell 0 of colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb
and run Cell 0 → 1 → 1b → 1c → 1d → 2 → 3 → 4 → 5 → 5b → 6:
separate notebook: colab_matched_gpt2_baseline_openwebtext.ipynb
Training artefacts were written to semsimula_matched_gpt2_baseline_owt/.
Files
| path |
what |
README.md |
this card |
config.json |
architecture, training and result, machine-readable |
checkpoints/*_best.pt |
weights at the best eval |
checkpoints/*_step25000.pt |
an intermediate checkpoint; see the note below |
results/training_log.jsonl |
per-eval record; the settled PPL above is the mean of its last three rows |
results/*_training_output.txt |
the full run log, including per-step diagnostics |
results/gpt2_baseline_summary.json |
end-of-run summary; read NOTE_summary_json_fock_fields.md beside it first |
code/ |
everything needed to reproduce the run, below |
The code, and why this subset
code/ mirrors the layout of the companion repository, so the notebook's
own sys.path lines resolve unchanged.
| path |
what |
code/colab_matched_gpt2_baseline_openwebtext.ipynb |
the whole run. Self-contained: the model is defined in Cell 3 and nothing is imported from the companion repository |
Caveats
- This is a reference, not a tuned transformer. It exists to give the
ladder a scale, and it is the only arm whose learning rate was tuned
separately (6e-04). Read it as "what a conventional architecture reaches
on this budget", not as a claim about GPT-2.
- Depth is not matched. L=8 here against L=2 on every Fock arm. The
ratio column elsewhere in the collection is for orientation, not a
like-for-like efficiency claim.
- The second checkpoint is not the same kind as the other arms'. The
Fock arms each ship step 500, essentially untrained. This arm has no
step-500 checkpoint; its second file is step 25,000 at PPL 52.87,
already within 6% of the endpoint. Do not read a cross-arm comparison
of "first" checkpoints as like-for-like.
gpt2_baseline_summary.json carries a stale cross-reference. Its
fock_* fields and ratio_fock_to_gpt2_settled are not a ladder
arm: the file was written before any L=2 arm existed and those numbers
come from a pre-ladder L=8 run read mid-schedule. The GPT-2 fields
in it are correct. NOTE_summary_json_fock_fields.md sits beside it.- OpenWebText only.
Citation
@techreport{Gueorguiev2026SemanticSimulation,
author = {Gueorguiev, Dimitar P.},
title = {Semantic Simulation: A Prescriptive Lagrangian Framework for Semantic Inference},
note = {Conservative Potentials, Non-Conservative Memory, and the Shared-Potential Separator},
year = {2026}
}
Card generated from _ladder/ladder.json on 2026-09-27.
