tiny-agent-gguf

Quantized language models for Tiny Agent — a Windows application that gives you Word, Excel and PowerPoint with an AI assistant that runs on your own CPU, with no GPU, no cloud API and no account.

The app: download releases/TinyAgentSetup-1.0.0.exe from hkblockchaintechnology/tiny-agent, double-click, and it installs like any other Windows program. This repository supplies the model that installer uses — or that you can point any local llama-server at.

These files are quantizations of existing Apache-2.0 models, not new models. Nothing here is fine-tuned: we convert and quantize, so the knowledge, biases and limitations are inherited unchanged from the base model.

TL;DR

Files in this repository

Sizes are the real blob sizes, not estimates. All three are Apache-2.0 and safe to redistribute inside a product.

What it can do

Measured by actually running it — these are observed behaviours, not claims:

Why this model, and not something bigger

Tested rather than assumed. Six CPU-sized models were run through the real pipeline against identical frozen research notes, and scored on how faithfully they filled the JSON slots:

*tiny-sample artefact; it returned the wrong shape.

Qwen3-1.7B grounds retrieved facts best while staying fastest of the faithful options. On this hardware the bottleneck is tokens per second, not parameters of world knowledge, so a larger model buys nothing measurable and costs multiples of the time per document. Full method and raw data: docs/MODEL-BENCH.md.

How Tiny Agent uses it

The model is never asked to write a document, and it never touches a file. It is asked one small question at a time, and deterministic Python assembles the result:

 1. route    which KINDS of change could this instruction need?
             (replacements | newHeadings | newParagraphs | rows | totals
              | slides | charts | theme)                    <- deterministic
 2. plan     the model fills ONLY those slots, JSON enforced by a llama.cpp
             grammar, so a malformed plan is structurally impossible
 3. apply    python-docx / openpyxl / python-pptx write a NEW file  <- deterministic
 4. verify   fingerprint the file before and after, and report anything that
             changed WITHOUT being asked                      <- deterministic

Step 1 is what makes a model this size dependable: rather than a flat form with eight optional sections to wander around, it is handed only the two or three the instruction actually needs. Steps 3 and 4 mean its output can be checked rather than trusted, which is the difference between a demo and a tool you would point at real work.

Measured on the reference hardware, 12 of 12 plans produced a usable document (3 runs each of a Word edit, a spreadsheet edit, a deck edit and a new Word table), with a warm plan taking 2.5–5 s typical and the slowest measured plan 27 s.

One measured failure worth publishing, because it is the kind of thing a card usually leaves out: offered a single optional group, this model answered with prose in summary and left the group empty — twice in a row, on every run. Naming the group in the retry prompt did not fix it; making that one group required in the retry grammar did, immediately. That is the whole design thesis in one incident: the fix belongs in the shape of the output, not in the politeness of the request.

Each step is a JSON-schema-constrained call. For example, a plan's contract is handed to llama.cpp as a grammar, so the shape is enforced by the decoder rather than requested in the prompt:

{
  "type": "object",
  "properties": {
    "newParagraphs": {
      "type": "array",
      "minItems": 1,
      "items": {"type": "string"}
    }
  },
  "required": ["newParagraphs"]
}

That is why a 1.7B model is usable for a real task: structure, formatting, file writing and verification are handled by code, and the model only supplies short strings where a small model is genuinely competent.

Hardware and measured speed

Reference machine — deliberately weak, and the actual development target:

The generation figures were measured with the exact CPU-only llama.cpp build that ships in the product, 4 threads, a 4096-token context and a 220-token budget. The n-gram row is the mean of repeat runs on the same request.

What that means in practice

Measured end to end on the shipped runtime, web research enabled, 11 sources retrieved:

The effective rate is well below the raw tokens/second because prompt processing over injected research context dominates, and every job makes several calls in sequence. That is why the interface streams progress: two to four minutes is perfectly acceptable for a "generate document" button, but only if the user can see which stage is running.

Speculative decoding is not optional on a CPU

Speculative decoding drafts several tokens cheaply and verifies them in one forward pass of the real model. On a CPU — where a forward pass is expensive and a batch is comparatively cheap — this is the single largest speedup available, and it costs nothing in output quality.

llama-server -m qwen3-1.7b-Q4_K_M.gguf --spec-type ngram-mod needs no extra download and measured 6.50 → 11.46 tok/s. It mines repeated token patterns from the context, so it is a warm-session win: the first request of a session gains almost nothing (7.33 tok/s) and the third reaches 11.46 tok/s as the shared hash pool fills. That suits a document generator exactly, because a job is several calls in a row sharing a long system prompt and emitting repetitive JSON.

EAGLE3 is the theoretically stronger option: a one-layer draft that reads the target model's hidden states and should therefore achieve a much higher acceptance rate. This repo ships qwen3-1.7b-eagle3.gguf and it is used like this:

llama-server -m qwen3-1.7b-Q4_K_M.gguf -md qwen3-1.7b-eagle3.gguf \
    --spec-type draft-eagle3 --spec-draft-n-max 8

It measured slower on a CPU, and this card will not pretend otherwise.

Acceptance was only 8.9%, and the draft shares the same 4 physical cores as the target, so it steals more throughput than it saves. EAGLE3 is a large win on a GPU — where the tiny draft runs nearly free beside the target — and a regression on a 4-thread CPU.

This was re-checked on a much faster machine (a 20-core server CPU running CPU-only, 4 threads) using our own converted GGUF: baseline 21.90 tok/s, n-gram 21.60 tok/s on a cold single run, EAGLE3 16.30 tok/s (0.74x). So EAGLE3 is slower regardless of target conversion, and n-gram's benefit genuinely depends on a warm, repetitive session — which is the normal case for this tool.

Prefer n-gram speculation on CPU; treat the EAGLE3 file as an experiment.

Practical consequences:

• Cold load costs ~13 s on a spinning disk. Keep the model resident between requests rather than reloading per call.
• Use one thread per physical core. More threads than physical cores is slower.
• Qwen3 is a reasoning model. Left alone it spends most of its budget on hidden thinking tokens, so disable thinking (think=false, or chat_template_kwargs={"enable_thinking": false}).
• A grounded 3-section report takes ~140 s end to end. That is acceptable for a "generate document" button, but only with progress reporting.

Quick start

llama.cpp server (OpenAI-compatible, supports JSON-schema constrained output):

llama-server -m qwen3-1.7b-Q4_K_M.gguf -c 4096 -t 4 --port 8080

llama.cpp CLI:

llama-cli -m qwen3-1.7b-Q4_K_M.gguf -c 4096 -t 4 \
  -p "Return only JSON: {\"heading\": str, \"paragraphs\": [str]} for a section about heat pumps."

Ollama (optional — the product does not need it):

ollama pull hf.co/hkblockchaintechnology/tiny-agent-gguf

The official CPU-only llama.cpp Windows release is an 18.8 MB zip (a 46 MB folder unpacked) and needs no GPU, CUDA toolkit or Python environment — that is the intended way to ship this.

Prompting notes for a model this size

1. One question per call. Asking for several things at once is where small models fall apart.
2. Constrain the output shape. Use a JSON schema or GBNF grammar. It removes an entire class of parsing failures.
3. Disable thinking. Qwen3 will otherwise spend its whole budget reasoning.
4. Give it the facts. Put retrieved text in the prompt; it cannot supply accurate specifics from memory.
5. Ask for less text. 2–3 short paragraphs per section works; "write a detailed 2000-word report" does not.
6. Expect and clean up artefacts. It adds list numbering to headings and sometimes leaks markdown into prose fields. Strip these deterministically during assembly rather than fighting them with prompt wording.

Limitations

• Hallucination. Without grounding, specifics (figures, dates, product names) are unreliable and often invented. Tiny Agent mitigates this by injecting retrieved web text and attaching a source list, but it does not eliminate it.
• Inherited knowledge cutoff and biases. Because this is a quantization, the base model's training data, cutoff and biases apply unchanged.
• Language. The base model is multilingual, but the pipeline prompts and all validation were done in English.
• Repetition. Sections written in separate calls sometimes restate the same point. Human editing is expected.
• Not a general chat model. It is tuned here for structured slot-filling, not for open-ended conversation.
• Quantization degrades quality slightly versus the full-precision base model. 4-bit is the point where quality and speed balance best on this class of hardware.

Training data

No training or fine-tuning was performed. This repository contains quantized conversions of:

• Qwen/Qwen3-1.7B
• Qwen/Qwen3-0.6B

and a converted speculative draft:

• AngelSlim/Qwen3-1.7B_eagle3 (Apache-2.0, Tencent AngelSlim) — converted to GGUF with convert_hf_to_gguf.py --target-model-dir Qwen/Qwen3-1.7B. It is a draft model: it proposes tokens that the 1.7B model verifies, and it never changes the output, only the speed. It cannot be used on its own.

Refer to the base model cards for training data, evaluation results and limitations.

Why these models: both are Apache-2.0, so they can be redistributed with a commercial product without additional conditions. Llama, Gemma and several other otherwise-capable small models are deliberately not bundled, because their licences carry redistribution terms that are unsuitable for shipping to end users.

License and attribution

Apache-2.0, inherited from the base models. Copyright in the base weights belongs to the Qwen team; these are derivative quantizations redistributed under the same licence.

Related

• Project source, benchmarks and generated documents: hkblockchaintechnology/tiny-agent