AliceAI-T5-35B-A0.6B-chat-lora-v6

AliceAI-T5-35B-A0.6B · chat + tool-call LoRA, v6

Status: v6, experimental. Read the tables below; they are generated from the run.

A LoRA adapter (rank 16, alpha 16) for yandex/AliceAI-T5-35B-A0.6B, a 35B-total / 0.6B-active sparse-MoE encoder-decoder base model with no chat format. It teaches the model to chat, to call tools when a task needs them, to answer in text when it does not, and to say nothing when there is nothing to say. Every answer it gives thinks first: each non-silent training target starts with a <think>…</think> block.

How we got here

v1 ran inside an agent harness and misbehaved: a bare "hi" produced 17 to 22 parallel tool calls, "don't read any skills" was ignored, replies were empty or endless. Cause, measured: its chat format used the vocabulary's [INST_START], [REPLY_END], [SYSTEM_*], [TOOL_RESULT_*] tokens, whose embeddings in the base model are untrained placeholders, identical to the unused <SPAN#5000>. The model could not tell roles apart or emit [REPLY_END], and LoRA on attention cannot teach embeddings. placeholder_ids.json lists all 5,832 placeholder tokens, and a unit test fails if the chat format ever uses one.

v2 switched to plain-text role markers and </s> as the end of a turn, and mixed plain chat into the data. It passed 61% of its behaviour probes (base model: 11%) but still, under a coding-agent prompt, invented a task for a bare "hi" and called a tool, repeated a tool call after its result instead of answering, over-thought simple questions, and answered "thanks" with reasoning instead of silence.

v3 changed only the data: more greetings, small talk and "no tools" instructions under varied agent prompts, real "final answer right after a tool result" turns, reasoning capped at 600 tokens and optional, thank-you silence under harness prompts, and no training target that repeats a recent call. It passed 88% of the (rebuilt) probes, with greetings at 24/24, but under pi's prompt it still repeated the call it had just made instead of answering once a tool result arrived (after-tool probes 4/10). The cause was visible in the data: pi's prompt only ever appeared with plain replies, never next to a tool result.

v4 changed only the data again, to put exactly that into training: real General-Agent trajectory prefixes retargeted onto pi's tools and prompt (about 85% of its calls map onto read/bash/edit/write, but no whole trajectory does, so only the part before the first unmappable call is used), plus templated pi flows with deterministic results and answers. It fixed the bug: after-tool probes went from 8/23 (v3) to 22/23. But it regressed elsewhere: a plain question under pi's prompt got a pointless file read (2/4 greedy), "don't use tools" was ignored in 3 of 8 probes, and on held-out real turns the right-kind-of-reply rate fell from 91% to 83% (it under-called tools). Its sampled score was 10 points below its greedy score, so many decisions were close calls.

v5 changed only the data once more (plain questions under pi's prompt with no instruction, "don't use tools", rebalancing towards tool calls, richer flows). It was the best version so far: 96% right-kind-of-reply on held-out real turns, and greetings, plain questions, "don't use tools" and silence all passed in both greedy and sampled mode. Tried in pi it behaved on the prompts that had broken v1, with one problem: on a story request it looped one dialogue for 13,221 tokens. It also still called a tool after "don't use tools" when the task needed one (2/8), and some answers after tool results were empty (the answer trapped inside <think>) or dropped the key fact.

v6 changes only the data, aimed at those, plus one new rule:

• The rule: every answer thinks first. Every non-silent target starts with <think>…</think>; the data builder rejects any target that does not, and re-checks the finished file. Reasoning is real where a source has it (the agent data: about 100% of action and final turns in Code, General and Search; UltraData's IF-think split) and exact where it is templated (flows, greetings, negations: the logic is known). A real turn with no reasoning is dropped, never given invented reasoning. UltraData's Knowledge traces were unusable (median 5,241 tokens; one in 400 fits a 600-token budget) and smoltalk has none, so neither is used. Silent rows stay empty: there is no answer to think before.
• negtask, about three times more: a task that would need a tool plus "don't use tools", with broader phrasings (count lines, find a file, make a directory, run a script). The right reply is an honest text answer, not a guess and not a call.
• longform: real long answers that end (UltraData IF-think), so long text learns to stop. Probes check for loops and endings.
• 133 probes (up from 129) with long-form probes, a loop check, and a think-rate metric on probes and held-out turns.

Chat template

The base repo ships no chat template; chat_template.jinja defines one inside the model's UL2 frame:

encoder:  [_S_]<|system|>…\n<|user|>…\n<|assistant|>…\n<|tool|>…\n<|assistant|><SPAN#0>
decoder:  <s><SPAN#0>  →  <think>reasoning</think>text\n<tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call></s>

Tools go in the system message as text in the GLM format (# Tools … <tools>{json per line}</tools> …), produced by glm_tools.py. Reasoning always comes first (capped at 600 tokens in training). An empty reply is just </s>. Pass alice_frame=False to apply_chat_template when a runtime adds [_S_] and <SPAN#0> itself (vLLM plugin below).

Use

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

hf_hub_download("kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v6", "alice_chat.py", local_dir=".")
from alice_chat import clean_decode, parse_output

BASE, ADAPTER = "yandex/AliceAI-T5-35B-A0.6B", "kaivoss/AliceAI-T5-35B-A0.6B-chat-lora-v6"
tok = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)          # carries the chat template
model = AutoModelForSeq2SeqLM.from_pretrained(BASE, trust_remote_code=True, dtype=torch.bfloat16,
                                              attn_implementation="eager", device_map={"": "cuda:0"})
model = PeftModel.from_pretrained(model, ADAPTER).eval()     # or load the merged repo directly

messages = [{"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Give me one fact about Zürich."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
src = tok(prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(model.device)
span = tok.convert_tokens_to_ids("<SPAN#0>")
dec = torch.tensor([[model.config.decoder.bos_token_id, span]], device=model.device)
out = model.generate(input_ids=src, attention_mask=torch.ones_like(src), decoder_input_ids=dec,
                     do_sample=True, temperature=0.6, top_p=0.95, max_new_tokens=512)
gen = out[0, 2:].tolist()
gen = gen[: gen.index(tok.eos_token_id)] if tok.eos_token_id in gen else gen
reasoning, content, tool_calls = parse_output(clean_decode(tok, gen))

Use sampling, not greedy: greedy decoding looped in testing.

Serving with vLLM and an OpenAI-compatible client

The community plugin (AntonProkopyev/AliceAI-T5-35B-A0.6B-vLLM) loads the merged repo unchanged, but it only serves /v1/completions and adds the UL2 frame itself. alice_openai_shim.py sits in front of it and provides /v1/chat/completions with tools, reasoning and streaming (simulated). It also samples by default (the harness sends no temperature), merges duplicate tool calls, caps parallel calls, never lets a turn continue as the user, retries an empty reply after a user or tool message, shows an answer that landed inside <think> as the reply, and cuts a repetition loop after its first cycle (v5 once looped a story for 13,221 tokens). serve_smoke.py exercises it with the openai Python client.

Training

Data

One example per assistant turn: the encoder reads everything before it, the decoder learns that turn, then </s>.

Sources: real agent trajectories (UltraData-SFT-Agent-2609, GLM-formatted copy linked above), plain Q&A from UltraData-SFT-2605 (English IF and Knowledge), greetings and system-prompted chats from smoltalk, a sanitized capture of the pi coding-agent prompt and tool schemas (harness_pi.json), General-Agent trajectories retargeted onto pi's tools, templated pi flows with exact reasoning, and negation-with-task replies (pi_flows.py), and templated greetings, small talk, "no tools" instructions and thank-yous (phrases.py). Targets are filtered to at most 8 parallel calls, 600 reasoning tokens and no repeat of a recent call. Held-out rows are never trained on.

Evaluation

Behaviour probes (probes.py): greetings, "don't use tools", small talk and plain questions under pi-style prompts (with and without a 14-skill list, and with a tool subset) and a non-coding weather / calculator / search assistant; real first calls; answering after a tool result; the second call of a multi-step task; silence after a thank-you or a finished task. A probe passes only if every expectation holds, the reply finished, and it never wrote another turn. The previous adapter (v3) is scored on the same probes.

Held-out real turns (this adapter):

n = 60 held-out rows, greedy, 1,024 new tokens.

These are small samples (133 probes, a few dozen held-out rows) of greedy or single-sample generations. They check for the specific failures above; they are not a capability benchmark. Tool-call arguments are not checked, only the tool name, and the tool-name metric counts a reasonable alternative as wrong.

Limitations

• Small budget. 200 steps is 1,600 examples: enough to move behaviour, not to add agentic competence the base lacks.
• Attention-only adaptation. Experts and embeddings are untouched.
• Templated data. Greetings, small talk, negations, thank-yous and the pi flows are templated, which is why a hash-held-out 20% of their wording is kept for the probes; real-world phrasing outside those templates is untested.
• Tokenizer quirk. The tokenizer inserts a space after every newline when decoding; use clean_decode from alice_chat.py (important for code in tool arguments).
• Data. Reasoning traces come from whatever model produced the upstream data, not from AliceAI. Plain chat comes from English-only sources. No safety tuning was done: the model can and will run bash commands if a harness lets it.
• vLLM. The merged-weights path was verified with v1's checkpoint on the community plugin; re-verify for this one.

Licence and attribution

Apache-2.0, matching the base model (© Yandex LLC) and the data (UltraData: arXiv:2602.09003; smoltalk: Apache-2.0 for its new subsets).