qwen3-1.7b-agent-tool-recovery-study

Agent Tool Recovery: a small-model study and verified-workflow pilot

Research snapshot with mixed and negative results. The adapters did not establish reliable general agent recovery.

本项目研究小模型能否通过故障恢复轨迹训练改善工具调用，以及哪些可靠性问题更适合交给确定性执行程序。恢复训练在受控模板上有效，但组合故障、预算停止、消歧与虚构完成没有稳定解决。后续小规模系统实验支持保留可验证执行工作流；九例开发集中，模型输入理解没有超过简单规则。目前结束追加训练，发布权重、实验材料与限制，供复查和复现。

The repository contains six validation-selected Qwen3-1.7B LoRA adapters (Success-v3 and Recovery-v3, seeds 17/23/41), executable synthetic environments, data, evaluation traces, and two small calendar workflow pilots. The root adapter is Recovery-v3 seed 17, epoch 1, chosen as the first prespecified seed and the adapter used in the workflow pilot, not because it had the best test result. This is an adapter, not a merged model or a hosted agent service.

Findings and scope

1. Recovery-context SFT improved recovery on familiar synthetic task templates. The original v2 experiment also increased unsupported completion versus the prompt baseline, failing its frozen safety criterion. Original results.
2. Shared boundary supervision produced very strong v3 validation results. Independent structural confirmation still exposed failures in repeated transient faults, multiple-candidate clarification, retry-budget stopping, and composed faults. Validation success was not confirmation of broad reliability.
3. A state-contrast retry pilot shifted behavior from premature stopping toward excessive retrying. It failed the preregistered gate. Pilot results.
4. In four fixed calendar cases, a deterministic workflow completed the three confirmable cases and retained uncertainty for the fourth. Model execution arms completed none. The initial guard also missed a success claim hidden under informational; final result text is now generated from verified runtime state. Report and correction.
5. In a subsequent nine-case development suite, simple rules and model interpretation both handled 9/9 correctly, including 7/7 confirmable completions. The model took about 29.8 seconds of local inference versus under 0.01 seconds for rules, excluding model loading. One field-validation fix was made using these cases; they are not an independent holdout. Hybrid pilot.

The practical deliverable is the verified workflow and test harness. These results do not establish incremental model value over simple rules for the tested calendar inputs, nor cross-service reliability.

Independent confirmation: preserve the scoring history

The following are the original frozen recovery-mode scores, not a replacement or silently corrected leaderboard. The denominator is 432 recovery tasks per seed. Complete-mode tasks have a separate denominator of 480.

Read these three artifacts together; their scoring definitions differ:

• Frozen results and machine-readable summary: unchanged historical outputs. The original complete-mode scorer could penalize proactive lookup that avoided a prescribed error.
• Narrow post-hoc sensitivity audit: removes only the mandatory observed-error requirement for completed-outcome tasks in complete mode, preserving other checks. Complete successes become Success 386/332/334 versus Recovery 335/347/351, by seed 17/23/41. Recovery-mode scores are unchanged in this audit.
• Broader scoring erratum and its data: also tightens completion evidence, retry stopping and candidate-selection checks. It reports complete successes Success 308/303/281 versus Recovery 316/310/332, and recovery successes Success 122/138/129 versus Recovery 247/222/288. These are post-hoc diagnostics, not a new independent confirmation experiment. The analysis implementation is included.

Neither post-hoc analysis replaces the frozen record. Safety and generalization remain unresolved under these readings. Counts from correlated synthetic templates should not be interpreted as broad deployment guarantees.

Training and checkpoint selection

• Base: Qwen/Qwen3-1.7B, revision 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e.
• Target-only LoRA SFT: rank 16, alpha 32, dropout 0.05; attention and MLP projection modules.
• Learning rate 1e-4, microbatch 1, gradient accumulation 16, up to two epochs; Apple MPS training. Seeds 17, 23, 41.
• v3 arms: 3,616 supervised decisions and 137,033 target tokens per epoch per arm. This matches supervision quantity, not every contextual token or computational cost.
• Both arms receive the same boundary examples for permission refusal, ambiguity, and persistent failure. Recovery differs by recovery context on the matched underlying training tasks.
• Selection uses validation safety eligibility, complete-task success, fewer unsupported completions, then earlier epoch. Recovery 17/23/41 and Success 17/41 select epoch 1; Success 23 selects epoch 2.
• Training protocol, v3 amendment, confirmation protocol, selection record.

The six exact adapter paths and original weight hashes are in release_manifest.json. v2 weights, unselected epochs, retry-pilot weights, base weights, caches, and raw console logs are excluded from this first release. Reports for those experiments remain available. Historical reports retain original local paths and hashes for provenance; relocate paths when rerunning.

Use and reproduce

Download this repository using huggingface_hub.snapshot_download with its Hub repository ID, then run the following from the downloaded snapshot:

python -m pip install -e '.[dev]'
python -m pytest -q

The workflow tests require no model download. For a fresh model-free workflow report, use the included library directly:

from recovery_agent.verified_workflow import WorkflowPolicy, pilot_cases, run_case, score_case

for case in pilot_cases():
    trace = run_case(case, WorkflowPolicy(), expose_state=True, constrained=True)
    print(case['task_id'], score_case(case, trace))

For model use, install the recorded ML dependencies (python -m pip install -e '.[ml]') in a compatible Python environment. Load the root adapter locally as follows; other selected adapters use their relative path from the manifest:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

revision = '70d244cc86ccca08cf5af4e1e306ecf908b1ad5e'
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3-1.7B', revision=revision)
base = AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-1.7B', revision=revision)
model = PeftModel.from_pretrained(base, '.')

This example loads on CPU by default; choose a supported device/dtype for practical inference. Base weights must be downloaded separately. Use recovery_agent.model_pipeline.TransformersPolicy and recovery_agent.training_data.prompt_messages for the actual action interface: greedy generation, enable_thinking=False, structured public task context and tool observations. Ordinary chat prompting does not reproduce the experiment. The adapter emits tool_call, ask_user, or final JSON; it does not execute tools itself.

Data generators, frozen synthetic datasets, manifests and complete evaluation traces are included. See project documentation and the experiment-specific run instructions. Several historical scripts require local model cache files and checkpoint paths. Rerunning those evaluations invokes inference; training scripts launch training. Do not overwrite the released result files when reproducing.

Limitations and intended use

Intended for inspection, controlled reproduction, and studying failure modes. The trained policy was studied with one small base-model family, two synthetic task domains, and narrow tool schemas. CRM/schema tooling in the source is not evidence of successful trained-model transfer. The workflow pilots use a simulated service with explicit idempotency and effect-query assumptions; they do not demonstrate reliable operation against a real calendar provider.

The adapters can retry beyond budget, choose incorrect actions, fail to clarify, and assert completion without evidence. They are not validated for autonomous production mutations. The nine-case hybrid pilot uses explicit IDs, emails and ISO timestamps and cannot establish general natural-language understanding. Test sets inspected during this work are now regression artifacts, not fresh holdouts for future tuning.

Licensing and provenance

The original project code, documentation, synthetic datasets, evaluation artifacts, and released LoRA adapters are licensed under the Apache License, Version 2.0, to the extent of the project contributors' rights. See LICENSE for the full terms and NOTICE for attribution. This license permits commercial use, modification, and redistribution subject to its terms; it does not imply model reliability or fitness for a particular use.

The base model is distributed by Qwen under Apache-2.0; consult the upstream model repository. The adapters were produced by LoRA fine-tuning the pinned base revision; this snapshot does not bundle the base weights. The upstream license is preserved in licenses/Qwen3-1.7B-LICENSE. Third-party materials and dependencies retain their own licenses; see THIRD_PARTY_NOTICES.md. The explicit project license was added on 2026-10-09; model weights and recorded experiment results were not changed by this licensing update.

The release includes SHA256SUMS for integrity checking. Synthetic tasks and sandbox interactions are experimental artifacts, not real customer or calendar records. The model card summarizes measured results and explicitly retains the unsuccessful experiments and scoring amendments.