A 13.1M-parameter text-to-image model for pixel art, flat icons and stickers. Small enough for laptops and, next, the browser.
Part of Darkroom, a project to build tiny image generators that run where you are.
|
|
| Parameters |
13.1M (plus CLIP ViT-B/32 text encoder, TAESD decoder) |
| Weights |
26 MB (fp16) |
| Output |
256x256 RGB |
| Speed, T4 GPU |
187 ms per image (20 steps, guidance 4) |
| Speed, CPU (Kaggle) |
1,844 ms per image |
| CLIP score, held-out prompts |
30.28 (teacher PixArt-LCM: 32.64) |
Samples
The same held-out prompts, never seen in training, from the 0.6B teacher and from each Pinhole size:
Run it in your browser
No install needed: open the live demo. The model runs on your device with
ONNX Runtime Web (WebAssembly or WebGPU), so prompts and images never leave the browser.
| Browser file |
Size |
onnx/dit.onnx (Pinhole-S) |
53 MB |
onnx/text_encoder.onnx (CLIP ViT-B/32, fp32) |
253 MB |
onnx/taesd_decoder.onnx |
4.9 MB |
| Total first download (cached afterwards) |
311 MB |
Parity: the ONNX-only pipeline matches PyTorch to within a mean of 0.00/255 per pixel (same noise, 20 steps).
The browser's tokenizer is checked against Python by the demo's Self-test button.
import * as ort from "https://cdn.jsdelivr.net/npm/[email protected]/dist/ort.all.min.mjs";
import { AutoTokenizer } from "https://cdn.jsdelivr.net/npm/@huggingface/[email protected]";
import { PinholeWeb } from "./pinhole-web.js"; // web/pinhole-web.js in this repo
const pinhole = await PinholeWeb.load("nuhmanpk/pinhole-s", { ort, AutoTokenizer });
const { rgba, width, height } = await pinhole.generate("sticker of a happy red fox", { steps: 20, cfg: 4, seed: 7 });
canvas.getContext("2d").putImageData(new ImageData(rgba, width, height), 0, 0);
Quick start
pip install torch diffusers transformers safetensors huggingface_hub
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("nuhmanpk/pinhole-s"); sys.path.insert(0, path)
from pinhole import Pinhole
pipe = Pinhole.from_pretrained(path) # GPU if available, else CPU
images = pipe("sticker of a happy red fox", n=4, steps=20, cfg=4.0, seed=0)
images[0].save("fox.png")
Prompting
Start with one of the three styles, then describe a simple subject: <style> of a <adjective> <colour> <subject>.
| Style |
Example |
pixel art |
pixel art of a retro blue rocket |
flat icon |
flat icon of a cozy green house |
sticker |
sticker of a smiling pink cupcake |
Tips:
- Single, simple subjects work best: animals, food, objects, plants, vehicles and fantasy items.
- Adjust guidance with cfg: 3–6 is the useful range; higher means stronger prompt following and less variety.
- Adjust steps with steps: 12 is faster; 20 is the default.
The Pinhole family
| Model |
Params |
Weights |
CLIP score |
T4 GPU |
CPU |
pinhole-s |
13.1M |
26 MB |
30.28 |
187 ms |
1,844 ms |
pinhole-m |
34.4M |
69 MB |
30.36 |
225 ms |
3,661 ms |
pinhole-l |
70.8M |
142 MB |
30.57 |
258 ms |
6,764 ms |
| PixArt-LCM teacher (reference) |
600M + 4.7B text encoder |
~11 GB |
32.64 |
How it works
- Generator: a small diffusion transformer (adaLN-Zero blocks with cross-attention to text) predicts 4x32x32 latents with patch size 2 (256 tokens).
- Text: frozen CLIP ViT-B/32 text features (77 tokens plus the pooled embedding).
- Pixels: the TAESD decoder turns latents into 256x256 RGB.
- Training objective: rectified-flow matching (
x_t = (1 - t) x0 + t * noise, predict noise - x0) with logit-normal timesteps and 10% caption dropout for classifier-free guidance.
- Weights: an EMA of the training weights is shipped.
Training
- Data: nuhmanpk/darkroom-pinhole-18k, 18,000 images made by PixArt-alpha/PixArt-LCM-XL-2-1024-MS (4 steps) for short captions in 3 styles.
- Schedule: 30,000 steps, global batch 128, AdamW (lr 0.0004), warmup plus cosine, fp16, EMA 0.999, horizontal-flip augmentation.
- Hardware: Kaggle 2x T4, 74 minutes. Final training loss 0.3589.
Limitations
- Low resolution: 256x256, best for icons, sprites and stickers, not photographs.
- Simple subjects: it learned single, centered subjects on plain backgrounds. Complex scenes, counting, text in images and people are not supported.
- Limited by the teacher and a small dataset. Expect some off-prompt or blobby results, especially from
pinhole-s. The CLIP-score gap to the teacher is shown above.
- English captions only.
License
CreativeML Open RAIL++-M, inherited from the PixArt-LCM teacher, including its use-based restrictions (see LICENSE.md).
pinhole.py is Apache-2.0.
Credits
Built by Nuhman PK. Teacher: PixArt-α / PixArt-LCM. Text encoder: OpenAI CLIP ViT-B/32. Decoder: TAESD by Ollin Boer Bohan. Training ideas from DiT and rectified flow.
