amethyst-2-mini-failed

Amethyst 2 Mini (Failed)

A general-purpose chat model on Qwen3-4B that knows when to call a web-search tool, and when not to.

Part of the Amethyst family. Amethyst 2 Mini (Failed) keeps a direct, conversational voice and adds reliable web-search tool calling: search when the answer depends on current information, answer from knowledge when it does not, decompose multi-part questions into several queries, and summarize retrieved snippets into a cited answer.

Important: use the default chat template

Do not disable thinking (enable_thinking=False, --reasoning off, or a hand-built prompt that ends in an empty <think></think> block). The training data taught tool calls in turns that have no empty think block, so with one present the model tends to answer in prose instead of calling the tool (15/28 correct tool decisions instead of 28/28 in our test, sometimes inventing citations). Use the tokenizer's default apply_chat_template(..., add_generation_prompt=True) and the default llama.cpp --jinja template.

Training

• Base: Qwen3-4B, trained via LoRA on mlx-community/Qwen3-4B-4bit
• Data: 10,000 synthetic conversations distilled from nvidia/nemotron-3-super-120b-a12b and nvidia/nemotron-3-ultra-550b-a55b through NVIDIA NIM, filtered to 9,156 train / 796 validation. Mix: general chat 36%, search-positive 48%, no-search (answerable from knowledge) 8%, follow-up turns 8%.
• Method: LoRA rank 8, scale 20, 16 layers, lr 1e-5, batch 2, sequence length 2,048, 13,800 iterations (run in four resumed sessions).
• Tool format: the model emits <tool_call>{"name": "web_search", "arguments": {"queries": ["..."]}}</tool_call> and receives <tool_result>...</tool_result>, then answers with [1]-style citations. The tool only exists if you describe it in the system prompt, exactly as in training (see below).

Evaluation

48 hand-authored held-out prompts (zero overlap with the training templates): 28 tool-decision prompts (14 should search, 14 should not) and 20 general-chat prompts.

The base was run with enable_thinking=False (its normal non-thinking mode). Chat quality was judged on the LoRA-adapter form by nvidia/nemotron-3-super-120b-a12b; the MLX build below matches the adapter's held-out loss.

System prompts (use these exactly)

For plain chat (no tool):

You are Amethyst, a helpful, direct conversational assistant. Answer the actual question asked. Match your length to the question -- a one-line question gets a one-line answer, a substantive question gets real depth. Use plain language, skip filler openers, and don't restate the question back before answering. Format with markdown only when structure genuinely helps (lists, tables, code); prose questions get prose answers.

To enable web search, use the chat prompt followed by a blank line and the tool description (this is what the model was trained with):

You are Amethyst, a helpful, direct conversational assistant. Answer the actual question asked. Match your length to the question -- a one-line question gets a one-line answer, a substantive question gets real depth. Use plain language, skip filler openers, and don't restate the question back before answering. Format with markdown only when structure genuinely helps (lists, tables, code); prose questions get prose answers.

You have access to one tool:

web_search(queries: list[str]) -- search the live web. Pass 1-3 short keyword queries. Use it when the answer depends on current, local, or frequently changing information (news, prices, scores, releases, schedules, weather, "latest"/"current"/"today"). Do NOT use it for stable knowledge you already have -- arithmetic, definitions, settled history, language questions, or code.

To call it, emit exactly this and nothing else:
<tool_call>
{"name": "web_search", "arguments": {"queries": ["query one", "query two"]}}
</tool_call>

Results come back as:
<tool_result>
[1] Title -- snippet text (source.com)
[2] Title -- snippet text (source.com)
</tool_result>

After results arrive, answer in prose and cite the snippets you used as [1], [2], etc. If the results don't actually answer the question, say so plainly rather than guessing.

Formats in this repo

Why the MLX build is mixed precision: merging a LoRA into a 4-bit model re-quantizes it, and a plain 4-bit merge measurably damaged tool use (21/28 decisions, held-out loss 0.731). So the 16 layers the LoRA changed are merged at 8-bit and the remaining layers keep their original 4-bit weights. Q4_K_M loses some tool accuracy for the same reason; prefer Q8_0 when tool reliability matters.

MLX

from mlx_lm import load, generate

model, tokenizer = load("VertexAGI/amethyst-2-mini-failed")
system = """<paste the tool system prompt from above>"""
messages = [{"role": "system", "content": system},
            {"role": "user", "content": "What's the current price of gold per ounce?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=200))

GGUF (llama.cpp)

llama-server -m amethyst-2-mini-Q8_0.gguf --jinja -c 4096   # send the system prompt above in your chat request

Limitations

A 4B model: it can still miss a search on unusual phrasing, and the evaluation set is small (48 prompts) and hand-written. The tool must be wired up by you (the model only emits the call). Q4_K_M is measurably less reliable at tool decisions than Q8_0 and the MLX build.

License

Apache 2.0, inherited from Qwen3.