VinaySLM is a small language model (SLM) built from scratch in PyTorch
to learn how decoder-only Transformers work end to end -- tokenization,
embeddings, causal self-attention, training, and generation -- without
importing a pretrained architecture. This is not a production-quality
language model.
What this is
- A tiny, hand-written, decoder-only (GPT-style) Transformer.
- 466,176 parameters.
- Trained on a ~600-token toy corpus of AI/programming sentences.
- Packaged here as a custom Hugging Face model (
trust_remote_code=True)
so it can be loaded and inspected through the transformers ecosystem,
while the actual Transformer implementation (modeling_vinayslm.py)
is a thin wrapper around this project's own, unmodified model code.
Architecture
|
|
| Type |
Decoder-only Transformer (GPT-style, pre-LayerNorm) |
| Parameters |
466,176 |
| Vocabulary size |
255 |
| Context length |
32 |
| Embedding dimension |
128 |
| Transformer layers |
2 |
| Attention heads |
4 |
| Feed-forward dimension |
512 |
| Dropout |
0.0 |
| Positional encoding |
Learned absolute positional embeddings |
| LM head |
Untied from the token embedding |
| Decoding |
Greedy (argmax) only -- no sampling yet |
Tokenizer
A minimal, from-scratch word-level tokenizer (VinaySLMTokenizer,
wrapping this project's WordTokenizer): text is lowercased and split into
words and punctuation with a regex; the vocabulary was built directly from
the training corpus. <pad>, <unk>, <bos>, <eos> are the only
special tokens. This is not a BPE or pretrained tokenizer.
Training data
A tiny, hand-written corpus of AI/programming sentences (see this
project's data/raw/toy_corpus.txt) -- a few hundred tokens total.
Important limitation
The model heavily overfits this tiny toy corpus. It was trained to
memorize a few dozen short sentences, not to generalize. Prompts close to
a training sentence will often reproduce that sentence verbatim; novel
prompts will produce low-quality or repetitive continuations. This model
exists to demonstrate a correct, from-scratch Transformer pipeline, not
to produce useful or general text.
Example
Prompt: "Machine learning"
Output: "machine learning allows computers to learn patterns from data."
How to load
This repository ships its own model/config/tokenizer code (see
configuration_vinayslm.py, modeling_vinayslm.py,
tokenization_vinayslm.py), so loading it requires trust_remote_code=True
and the vinayslm Python package installed (the custom code imports
this project's own TinyGPT/WordTokenizer classes rather than
duplicating them -- see "Limitations" in this project's README):
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("vinaypoduri/vinayslm-tiny", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("vinaypoduri/vinayslm-tiny", trust_remote_code=True)
Or, equivalently, import the classes directly:
from vinayslm.hf import VinaySLMForCausalLM, VinaySLMTokenizer, generate_greedy
model = VinaySLMForCausalLM.from_pretrained("vinaypoduri/vinayslm-tiny")
tokenizer = VinaySLMTokenizer.from_pretrained("vinaypoduri/vinayslm-tiny")
print(generate_greedy(model, tokenizer, "Machine learning", max_new_tokens=15))
How to reproduce this export
From the VinaySLM project root, with its .venv active:
python scripts/prepare_data.py
python scripts/train.py
python scripts/export_hf.py
scripts/export_hf.py loads the trained artifact from checkpoints/tiny/,
wraps it in the Hugging Face-compatible classes, verifies it produces
identical greedy output to the original local model, and writes this
directory. Nothing is uploaded to the Hugging Face Hub by this script.
