ldt-10m

A 10,284,480-parameter LLaMA-style text model, trained from scratch. This is a verified first checkpoint for the LDT-10M request (model-requests #12, DedeProGames) — real weights, real training, but undertrained (see the honest status below). It is not a quality release yet; the card states that plainly.

Architecture

Standard LLaMA block, no sliding window, no GQA:

The parameter count is the learnable total: the raw safetensors sum is 14,216,640, which double-counts the tied embedding (tok.weight 12288×320 = 3,932,160) that head.weight aliases. Tied, the true count is 10,284,480.

Training

• Data: FineWeb-Edu (~30.1M tokens) + DCLM-baseline (~21.4M tokens) = ~50.5M tokens.
• Steps: 4000 (the requested 2.6B-token budget is not met — see below).
• Final val loss: 4.6020 (ppl 99.68) on the held-out split.
• Trained from scratch (no base model).

Honest status: undertrained, not broken

The model learned real context — val loss 4.602 is well below the 7.38 unigram floor, so it is not flatlined and not degenerate. But it has seen only ~50.5M tokens ≈ 4.9 tokens/param, far below the ~10–20+ tok/param small models typically need for coherent prose. The result is grammatical but semantically thin: wordy, repetitive, and it drifts off-topic mid-sentence.

Real samples (best.pt, step 4000, temp 0.8, top-k 40) — no broken tokens, no token-loops:

"The sun is" → "The sun is assembled by a new study because he is an associate of the study of the disease in the early years. I've been interested in having a very different study…"

"Once upon a time" → "Once upon a time, she is an attack. But he is not a good idea. But it is something that does not have a moment of his own life…"

"The cat sat on the" → "The cat sat on the ground. The sunp is a piece of light and is not a good deal of tear. The new story of the MD's Ford…"

"def hello():" → "def hello(): I have a lot of the best. I'm not sure what happened to me. I'll be able to do anything, but I think it's been just a lot to say…"

What this is: a verified, non-degenerate 10M checkpoint that learned grammar and surface fluency. What it is not: a coherent prose model. It is a first milestone on the path to the requested 2.6B-token model, not the destination.

Loading

Self-contained model.py (no transformers dependency required):

import torch
from model import LDT

m = LDT.from_pretrained("model.safetensors", device="cpu")
ids = torch.tensor([[1, 2, 3]])  # token ids
out = m.generate(ids, max_new_tokens=64, temperature=0.8, top_k=40, seed=0)

The tokenizer is a byte-level BPE (tokenizer.json, vocab 12288).

Roadmap

Continuing training toward the 2.6B-token budget is the next step; this repo will be updated (or a v2 published) once the model is coherent. This checkpoint is kept public as an honest intermediate, not a finished model.