The instruction-tuned release of Quartz Micro Preview
Base — the
same ~1B-parameter mixture-of-experts model, trained completely from
scratch on a single consumer GPU, now given a full-parameter supervised
fine-tune so it follows instructions instead of just completing text.
This is a preview, not a finished product. It is small on every axis
by design, and its evaluation below is reported honestly, including where
it still falls short — that's this project's standing practice, not a
disclaimer added after the fact.
What changed vs. the base checkpoint
- Full-parameter SFT, not LoRA — the base model has zero
instruction-following behavior, so a light adapter touch isn't the
right tool; every parameter was updated.
- Dataset: databricks-dolly-15k,
human-written (not distilled from another company's model), formatted as
### Instruction:\n...\n\n### Response:\n (with an added ### Context:\n
block for context-grounded examples), tokenized with the pretrain's own
custom 32K byte-level BPE tokenizer. 14,051 of 15,011 examples kept after
filtering to the model's 2,048-token context.
- Only the response tokens are supervised (prompt tokens masked to
label=-100); optimizer is 8-bit AdamW (bitsandbytes) with full gradient
checkpointing, matching the pretrain's own training stack.
- This is the second SFT attempt (v2): the first pass underperformed,
so this run used a higher learning rate and a fresh optimizer state
rather than continuing from the first run's optimizer. It completed
1,317 steps on a single GTX 1660 Ti (6GB VRAM).
Architecture
Unchanged from the base checkpoint — DeepSeek-style fine-grained
mixture-of-experts:
|
|
| Total parameters |
1,031.0M (~1.03B) |
| Active parameters / token |
394.0M |
| Hidden size |
1,024 |
| Layers |
20 (first 2 dense, rest MoE) |
| Attention |
16 query heads / 4 KV heads (GQA), head dim 64 |
| Context length |
2,048 tokens |
| Routed experts |
24 (6 active per token) |
| Shared experts |
2 (always active) |
| Expert FFN size |
640 (fine-grained segmentation) |
| Router |
top-6 of 24, 0.01-weighted load-balancing loss |
| Vocabulary |
32,000 tokens |
| Tied embeddings |
yes |
Held-out evaluation: base vs. this SFT checkpoint
Ten hand-authored prompts, not sampled from dolly-15k or any training
data, spanning open QA, closed/context QA, summarization, brainstorming,
classification, extraction, general QA, instruction-following, and
creative writing. Full transcript below is unedited.
Honest summary: a modest, mixed improvement — not a clean win.
The clearest gain is grounded, context-based QA. The weakest spot is
still strict instruction-following (format, length). Both checkpoints
remain largely incoherent past the first sentence or two, which tracks
with the model's size — that's not something this SFT pass was expected
to fix.
Win — closed QA (uses given context correctly):
Prompt: Context: The Golden Gate Bridge is 2,737 meters long and was
completed in 1937.\nQuestion: How long is the Golden Gate Bridge?
Base: ignores the context entirely, wanders into an unrelated story about
"the Banana" process and the Coast Guard.
SFT: "The Golden Gate Bridge is 2,737 meters long and has an area of
approximately 2.4 million square miles." — correctly extracts and
restates the fact (the second clause is invented, but the requested
answer is right).
Partial win — brainstorming (attempts the requested list format):
Prompt: List three ideas for reducing food waste in a household kitchen.
Base: no list structure, generic unrelated advice about water and diet.
SFT: "1. Make sure you have all food waste in your kitchen. \n2.f leaf
food and water should be discharged from local resources. \n3. Focus on
local food waste..." — correctly produces a numbered list (the content
is still weak/garbled).
Partial win — topical relevance:
Prompt: Why do cats purr?
Base: drifts immediately into an unrelated riff on "the human body,"
never mentions cats again.
SFT: "Finding the right balance between breast and chest health is
hard... So my recommendation is to keep your cat's skin healthy and
strong..." — stays anchored to "cat" throughout, base does not.
No win — strict instruction-following:
Prompt: Write a short two-sentence greeting for a birthday card.
Base: an unrelated bulleted list of vague social advice.
SFT: "The free meeting is open for all people... The national party has
the following 82 seats: The Democratic Party..." — completely
unrelated to the prompt, and no better than base here.
No win — classification:
Prompt: Classify the following as either a fruit or a vegetable:
broccoli
Base: never answers the classification, rambles about diet generally.
SFT: "broccoli, broccoli, cabbage,iant cucumber, cucumber, ormanus...
This fruit is most closely associated with pears and cucumber." — also
fails to answer, with garbled sub-word artifacts not seen in the base
output.
No GGUF build
Every other release on the Vertex AGI Hugging Face
org ships GGUF alongside the native
weights. This one doesn't, and that's deliberate rather than an oversight.
llama.cpp's closest matching architecture (deepseek, the original
DeepSeekMoE design) shares this model's leading-dense-layers + fine-grained
routed-experts + always-on shared-experts shape almost exactly — except it
skips renormalizing the selected top-6 expert weights after routing, while
this model's trained behavior does renormalize them. Measured on real
prompts, the top-6 experts this model picks only carry ~61% of the total
softmax mass on average (26–99% depending on the token) before that
renormalization — so a GGUF built on that architecture would run the
routed-expert pathway at a token-varying fraction of its trained strength,
not the same computation this checkpoint actually does. Rather than ship a
file that loads and runs but isn't faithful to the model, we're leaving
GGUF out until llama.cpp has a matching architecture (or we write one).
Files
model.safetensors — model weights, fp32 (optimizer state dropped)configuration_quartz.py, modeling_quartz.py — transformers-compatible
PretrainedConfig/PreTrainedModel wrapper (QuartzMoEConfig,
QuartzForCausalLM), wired up via auto_map in config.jsonmodel.py, config.py — the original plain-PyTorch model class and
architecture config, used by load_model.pyconfig.json — architecture config in HF's expected formattokenizer.json, tokenizer_config.json, special_tokens_map.json — a
standard transformers fast tokenizertokenizer/vocab.json, tokenizer/merges.txt — the custom tokenizer's
raw vocab/merges, used by load_model.py's plain-PyTorch path. Only
compatible with this model's weights.load_model.py — minimal working example using the plain-PyTorch path
Usage
Via transformers (recommended):
pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"VertexAGI/quartz-micro-preview", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained("VertexAGI/quartz-micro-preview")
prompt = "### Instruction:\nWhy do cats purr?\n\n### Response:\n"
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=True))
trust_remote_code=True is required — this is a bespoke architecture
(DeepSeek-style fine-grained MoE), not one of transformers' built-in
model types. No KV-cache support yet, so generate() recomputes attention
over the full sequence each step.
Plain PyTorch (no transformers dependency):
pip install torch safetensors tokenizers
python load_model.py
Limitations
Same size-driven limitations as the base checkpoint (factual
unreliability, weak general knowledge, no safety fine-tuning), plus
SFT-specific ones documented honestly above: strict format/length
instructions (e.g. "write exactly two sentences") are not reliably
followed, and generation occasionally produces garbled sub-word artifacts
the base checkpoint doesn't. Treat this as a proof-of-concept instruction
tune on a genuinely tiny model, not a general-purpose assistant.
Built by Vertex AGI. Every model we ship —
weights, not just claims.
