Sparse-AST-BWM — latest checkpoint, step 93,213
This repository contains one current checkpoint of the serial old/new Sparse-AST family for byte-level Blender Python completion. It was recovered from results (4).zip, the newest relevant training-results archive found in the local Downloads search, and its saved training state confirms step 93,213. It continues from step 90,213 through 3,000 successful updates, with no recorded non-finite skips.
Updated 16 October 2026. The current release is under latest/ and uses SafeTensors for model weights and training-state tensors. That release contains no .pt checkpoints. Existing remote variant folders are retained as historical artifacts; they are not the latest checkpoint. The pre-update legacy code remains available at revision 7c1001debae58e03d2a3e7073819de4c50ca0cfe.
Files in latest/
| File |
Purpose |
model.safetensors |
Exact FP32 model weights for step 93,213 |
config.json |
Actual tensor-derived architecture and training-context information |
model.py, generate.py |
SafeTensors loaders and API/command-line greedy byte completion |
family_oldnew_architecture.py, test_models.py |
Required custom PyTorch architecture |
training_state.safetensors, training_state.json |
Optimizer, scaler, RNG states, row order/cursor and training progress for this same checkpoint |
training_result.json, training_config.json |
Original recorded training results and settings |
data_audit.json |
Dataset split/provenance audit from the training archive |
validation.json |
Independent conversion checks, inference parity and SHA-256 hashes |
training_state.safetensors is resumable state for the same model version, not a second model checkpoint. Inference needs only the model weights, config and Python architecture files.
Architecture and context
- 18 serial stages, each retaining its own legacy/new branch pair.
- Width 704; feed-forward width 1,408; eight attention heads.
- 512-entry output vocabulary; inputs and generated text use UTF-8 bytes.
- Stored positional embedding capacity: 512 positions.
- This continuation trained with 128-byte contexts, not a verified 512-token or 4K context window.
- 201,380,162 model parameters (about 201.38M), stored in FP32.
Branches are mixed locally at each serial stage. This is not a top-k router over interchangeable stages. Legacy branches remain frozen during training. The actual saved input embedding and output projection differ and are preserved as separate matrices; the loader does not retie or change them.
Load and generate
Install PyTorch 2.4 or newer, safetensors and huggingface_hub. This is a custom PyTorch model; use the bundled loader.
import sys
from pathlib import Path
from huggingface_hub import snapshot_download
revision = "main" # Pin a commit hash for reproducible deployments.
folder = snapshot_download(
"Chaman1234/Sparse-AST-BWM",
revision=revision,
allow_patterns=["latest/*.py", "latest/config.json", "latest/model.safetensors"],
)
folder = str(Path(folder) / "latest")
sys.path.insert(0, folder)
from model import load_model, generate
model = load_model(folder, device="cpu")
print(generate(model, "import bpy\nbpy.ops.mesh.", max_new_bytes=80))
Generation defaults to the 128-byte training context and greedy decoding over byte values 0–255. The longer positional table does not establish reliable performance at longer contexts. Generated code should be inspected and executed in an appropriate Blender environment before use.
Resume state without pickle
from model import load_training_state
state = load_training_state("Chaman1234/Sparse-AST-BWM", revision=revision)
assert state["step"] == 93213
optimizer.load_state_dict(state["optimizer"])
scaler.load_state_dict(state["scaler"])
Construct the model's optimizer using model.trainable_parameter_groups() before restoring its state. Restore python_rng, torch_rng, and, on compatible CUDA hardware, cuda_rng; restore the recorded order, cursor and progress counters in the training loop. The loader reconstructs original dictionary key types and tuples from JSON and SafeTensors. No pickle is used to load the published release.
Evaluation and limits
The archived run reports the following teacher-forced byte-completion results on 256 examples per split:
| Split |
Cross-entropy |
Top-1 byte accuracy |
Top-5 byte accuracy |
| Validation |
0.015052 |
99.5342% |
100.0000% |
| Test |
0.029494 |
99.3290% |
99.8614% |
These are archived training-run metrics, not a newly reproduced benchmark, full-script success rate, Blender execution result or visual-quality assessment. The data audit reports family-disjoint train/validation/test splits within this sampled dataset; historical overlap with every earlier training corpus is not established. Source quality checks cover JSON/AST syntax and bpy imports, not runtime or visual correctness.
The release conversion independently verifies all model tensors exactly, all non-model training state exactly, finite parameters, and forward logits within FP32 tolerance (absolute 0.0001, relative 0.00001) with identical predicted tokens on two Blender-related prompts. See validation.json for measured differences and hashes. This establishes conversion fidelity, not general coding reliability.
Provenance
- Archive:
results (4).zip.
- Member:
sparse_ast_bigtrain_3k_output/checkpoint_latest.pt.
- Recorded final step: 93,213; parent step: 90,213.
- Recorded successful continuation updates: 3,000; attempts: 3,000; non-finite skips: zero.
- Training source: compact Blender task/code pairs described in
data_audit.json.
- Conversion used PyTorch's restricted
weights_only=True loader, followed by lossless SafeTensors serialization and parity checks.
Format references: SafeTensors and shared tensors, Hugging Face upload documentation.
Hybrid associative-memory / GRU pilot (evaluation-only)
A separate 74.13M-parameter byte model is available under hybrid-3memory-3gru-1attention/. Its FP32 inference checkpoint is stored as SafeTensors (model.safetensors) with matching PyTorch code and a technical PDF containing VRAM/RAM estimates and formulas. This model is distinct from the step-93,213 model in latest/.
The report explicitly labels the work evaluation-only. It has no matched vanilla-attention baseline, no tested source-checkpoint baseline, one training seed, and no measured end-to-end compute savings. The SafeTensors file contains model weights only; optimizer state, teacher logits, training data, and source weights are not included.
