Qwen3-8B-CC-SFT-v2

English | 简体中文

Qwen3-8B-CC-SFT-v2 is a failure-targeted continued-SFT checkpoint designed to improve coding-agent state preservation and continuation across Claude Code native context compaction. It is initialized from Qwen3-8B-CC-SFT-v1 and trained on compact-aware, context-correct supervision distilled from GLM-5.3 under real Claude Code native compaction.

Key Result

Best observed canonical303 Pass@1: 52.48% (159/303) — 4090 run of this same checkpoint at T=0.5 (temperature selected on an independent dev set; see the table below).

Best observed configurations of this same checkpoint:

• T=0.5 was selected on an independent 60-task development set (task-id and repo disjoint from canonical303), then evaluated once on canonical303 — no post-hoc selection on this benchmark.
• The T=0.5 run solved +15 tasks vs the original T=0.3 run (paired McNemar p = 0.18; 303/303 completed, 1 environment failure counted as unsolved) — a positive but not statistically significant difference at single-rollout power.

Same-harness paired comparison on the original 4090 runs — the clean Base → V1 → V2 training-improvement evidence:

Internal canonical evaluation: 303 SWE-smith-derived, repo-disjoint tasks, Claude Code native context compaction ON, Pass@1, 1 rollout per task. Not SWE-bench Verified.

The same v2 checkpoint achieved 154/303 (50.83%) Pass@1 in an independent A100 reproduction run, compared with 144/303 (47.52%) in the original 4090 run.

V2 vs V1 (paired, identical 303 tasks, same 4090 harness):

• +38 solved tasks
• +12.54 percentage points
• McNemar exact test p = 0.0002
• 95% paired bootstrap CI = [6.3, 18.8] percentage points

Cross-Machine Reproduction

The same SFT-v2-E1 model weights were re-evaluated on the identical canonical303 task set on independent A100 hardware:

Held identical: SFT-v2-E1 checkpoint (weights unchanged), canonical303 task identities, REAL40K context, Claude Code native compaction, temperature 0.3, top_p 0.95, max 32 agent calls, Pass@1, 1 rollout per task. The A100 run completed 303/303 valid environments with 0 infrastructure failures. Observed difference: +10 solved tasks (+3.30 pp); Wilson 95% CI for the A100 run: [45.22%, 56.41%].

The model weights are unchanged — this is a cross-machine performance reproduction, not a bitwise-identical, trajectory-identical, or runtime-identical execution.

Reproduction Notes

Runtime-level serving differences existed between the two environments:

The compact-fix implementation also differs slightly between the two serving stacks. Runtime-level compaction frequency differed between the two serving environments, so this experiment should be interpreted as a cross-machine performance reproduction rather than trajectory-level deterministic reproduction.

Paired Evaluation

Because both models were evaluated on exactly the same 303 tasks with the same protocol, per-task outcomes can be paired:

Net +38 tasks in favor of v2 (McNemar discordant pairs 68 vs 30; exact p = 1.56e-4, reported as 0.0002).

Why v2?

Trajectory-level failure analysis of v1 under Claude Code native context compaction showed that, after a compaction event, the agent frequently lost or corrupted execution state, including:

• completed vs pending work (what was done vs still to do)
• modification state (which files / symbols were edited)
• test execution state and test results
• the current hypothesis and the next action

v2 specifically targets agent-state compression and post-compaction continuation. It is not "more data + another SFT": every added supervision segment is derived from a real native-compaction boundary.

Training Data

Standalone dataset: liangzhidanta/claude-code-glm53-swesmith-compact-trajectories

Two core supervision types:

1. Compact Summary — the real Claude Code native compact request is forwarded to the teacher (GLM-5.3), and the teacher-generated state summary becomes the training target. Goal: faithful state compression, especially done / not-done status, modification state, test state, current hypothesis, and pending action.
2. Post-Compaction Continuation — the real model-visible post-compaction context is the prefix, and the subsequent assistant reasoning / text / tool calls are the target. Goal: learn to continue execution from a compressed state.

Dataset statistics (from the published dataset manifest)

Training recipe used for this checkpoint (COMPACT_FOCUSED, ≤40960 tokens)

Context-Correct Training Format

A naive conversion concatenates pre-compaction history + summary + post-compaction history into one training context. At real inference time, however, the pre-compaction trimmed history is no longer visible to the model — so the naive full-concat view creates a train–inference context mismatch and leaks history that the deployed model will never see.

SFT-v2 therefore constructs context-correct segments: each training example is split at real compaction boundaries, and each prefix is rebuilt from the actual model-visible wire request at that point. Post-compaction targets are trained only against inference-visible state.

Native Compaction Behavior

Measured over the canonical303 evaluation (Claude Code native auto-compaction ON):

Trigger rate is essentially unchanged: v2 does not improve by avoiding compaction. Instead, under nearly identical compaction-trigger rates, v2 requires fewer compaction events per triggered task (3.16 → 2.65, ≈16.2% relative reduction) while achieving substantially higher task success — consistent with more efficient post-compaction continuation. Thrashing (83 → 84 tasks) is not improved and remains an open limitation. For reference, the Qwen3-8B base model triggered compaction on 124/303 tasks (227 events).

Training Details

Losses are reported as training information only, not as capability metrics.

Training curve

Masked SFT loss over 367 optimizer steps (1 epoch):

Evaluation Protocol

No-compaction runs are treated as diagnostic controls, not the primary deployment-aligned evaluation: the target agent environment uses Claude Code native context compaction. SWE-bench Verified results are not reported in this release; the results above use our SWE-smith-derived canonical evaluation.

Model Lineage

Qwen/Qwen3-8B
        ↓
execution-verified GLM-5.3 trajectories
        ↓
Qwen3-8B-CC-SFT-v1
        ↓
trajectory-level failure analysis (native compaction failure modes)
        ↓
compact-aware targeted supervision (189 sources → 240 native compact events)
        ↓
context-correct continued SFT
        ↓
Qwen3-8B-CC-SFT-v2
        ↓
verifiable GRPO (planned)

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "<this-repo>"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")

This checkpoint uses the original Qwen3 tokenizer and chat template. For coding-agent evaluation, the model was served through SGLang with a Qwen3 reasoning parser and tool-call parser, and used behind a Claude-Code-compatible harness with native context compaction enabled.

Limitations

1. Task pool shared with v1 (same tasks, different-trajectory supervision).
2. Compact summaries are generated by GLM-5.3 and may contain residual errors.
3. 84 of 303 tasks still exhibit autocompact thrashing (unchanged vs v1's 83); thrashing is not solved by v2.
4. Evaluated only at temperature 0.3.
5. The internal 303-task benchmark is not SWE-bench Verified; SWE-bench Verified results are not reported in this release.
6. Performance may depend on harness/tool configuration; tool-use performance should not be interpreted as standalone raw-model benchmark performance.

Citation

@model{qwen3_8b_cc_sft_v2,
  title={Qwen3-8B-CC-SFT-v2: Compact-Aware Coding Agent},
  author={liangzhidanta},
  year={2026},
  url={https://huggingface.co/liangzhidanta/Qwen3-8B-CC-SFT-v2}
}