qwen2.5-0.5b-topo-governed-cbp-fineweb: Model Card

article: https://medium.com/ai-simplified-in-plain-english/the-architecture-of-permanence-how-topo-cbp-tamed-the-uncharted-stream-a17c1d8425af

Qwen2.5-0.5B-Topo-Governed-CBP-FineWeb

An open-weights causal transformer checkpoint evaluating the synthesis of Continual Backpropagation (CBP) and the Topological Governor (TOPO-2026) under non-stationary online web streaming.

This model validates that an autoregressive foundation model can continuously adapt to streaming data without experiencing representational drift or catastrophic forgetting on silicon.

• Base Model: Qwen/Qwen2.5-0.5B
• Governing Framework: TOPO-2026 Topological Governor
• Training Protocol: Continual Backpropagation with dual-phase manifold projection
• Streaming Dataset: HuggingFaceFW/fineweb-edu (sample-10BT)
• Optimization Horizon: 5,000 continuous gradient steps
• Execution Seed: 123 (deterministic across Python, NumPy, PyTorch, and CUDA)
• Companion Code: GitHub: CDL_CBP_TOPO.ipynb

---

Model Summary

In Loss of plasticity in deep continual learning (Nature, 2024), Dohare, Hernandez-Garcia, Lan, Rahman, Mahmood, and Sutton demonstrated that standard gradient-based optimization systematically loses plasticity, causing representational rank collapse. While their Continual Backpropagation (CBP) algorithm preserved feature diversity by recycling low-utility units, the authors noted that CBP alone does not prevent catastrophic forgetting. Furthermore, unconstrained continual updates cause networks to drift uncontrollably—what Richard Sutton described as streaming agents "losing their minds."

qwen2.5-0.5b-topo-governed-cbp-fineweb provides an empirical resolution to this stability-plasticity impasse. By decomposing the representation manifold into an invariant prime-anchored coordinate ring and an attenuated plastic complementary subspace, the network sustains active loss descent while holding its geometric reference frame to exact zero drift.

---

Mathematical Specification

The architecture implements a dual-phase number-theoretic boundary condition on the token embedding manifold ($\mathbf{W}_{\text{embed}} \in \mathbb{R}^{V \times D}$):

1. Invariant Prime Coordinate Ring

$$\mathcal{P} = {p \in \mathbb{P} \mid p \le 13} = {2, 3, 5, 7, 11, 13}$$

2. Dynamically Derived Euler Attenuation

The gradient energy entering the complementary plastic subspace is modulated by the truncated Euler product over $\mathcal{P}$: $$\Lambda = 1 - \prod_{p \in \mathcal{P}} \left(1 - \frac{1}{\sqrt{p}}\right) = 0.9785142874$$

3. Dual-Phase Manifold Execution

• Pre-Step Gradient Orthogonalization & Attenuation: $$\nabla_{\mathbf{W}i} \mathcal{L} \leftarrow \begin{cases} \mathbf{0}, & \text{if } i \in \mathcal{P} \ \Lambda \cdot \nabla{\mathbf{W}_i} \mathcal{L}, & \text{if } i \notin \mathcal{P} \end{cases}$$
• Post-Step Exact Projection: Second-order optimizer drift caused by decoupled weight decay ($\lambda_{\text{wd}} \mathbf{W}$) and running momentum buffers is arrested by projecting the reference coordinates back onto the active device: $$\mathbf{W}_i \leftarrow \begin{cases} \mathbf{W}_i^{(0)}, & \text{if } i \in \mathcal{P} \ \mathbf{W}_i, & \text{if } i \notin \mathcal{P} \end{cases}$$

---

Empirical Verification: Governed vs. Unconstrained Baseline

Both the governed model and an unconstrained baseline were evaluated side by side over 5,000 streaming steps on FineWeb-Edu under identical execution seed 123 and AdamW hyperparameters ($\text{lr} = 2 \times 10^{-5}$, $\lambda_{\text{wd}} = 1 \times 10^{-4}$):

$$\Delta_{\text{drift}} = \max_{p \in \mathcal{P}} \Vert{}\mathbf{W}p^{(t)} - \mathbf{W}_p^{(0)}\Vert{}\infty$$

Key Findings

1. Immediate Baseline Erosion: Without topological governance, unconstrained AdamW immediately suffers coordinate displacement on Step 1 ($3.05 \times 10^{-5}$), shifting into the $10^{-3}$ distortion regime by Step 250.
2. Absolute Topological Invariance: The governed model sustains 0.0000000000 drift across all 5,000 steps, completely arresting weight decay and momentum leakage.
3. Zero Plasticity Penalty: The governed model converges synchronously with the baseline ($2.8799$ vs. $2.8798$ final loss), proving that topological permanence does not impede active gradient descent on live web data.

---

Usage

This model checkpoint uses standard SafeTensors format and is directly loadable into Hugging Face Transformers:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "frankmorales2020/qwen2.5-0.5b-topo-governed-cbp-fineweb"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32,
    device_map="auto"
)

prompt = "The stability-plasticity dilemma in continual deep learning can be resolved by"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.7,
        top_p=0.9,
        do_sample=True
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))


```text
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Loading weights: 100% 290/290 [00:00<00:00, 982.29it/s]The stability-plasticity dilemma in continual deep learning can be resolved by using a combination of deep and shallow learning. In this paper, we use the example of a large scale recurrent neural network (RNN) to show the advantages of using a combination of deep and shallow learning. The RNN is used to solve the stability-plasticity dilemma in a large scale continual deep learning. The RNN is used to represent a large scale data structure by learning the local patterns of the data structure. The RNN is trained by using the RNN as a deep neural network