qwen3.6-35b-a3b-swe-lora-sft-v01

A LoRA adapter (rank 16) for Qwen/Qwen3.6-35B-A3B, supervised fine-tuned on 1,489 repository-level bug-fix samples (SWE-smith and SWE-Gym) to produce a unified diff from an issue description plus repository context. The adapter was trained with Megatron-SWIFT (ms-swift 4.2.3) on 4 x NVIDIA A100-SXM4-80GB for one epoch (93 optimizer steps) and exported in PEFT format. Targets are human- or commit-derived patches; no distillation from another model and no reinforcement learning were used.

A second-stage adapter continued from this one with execution-filtered self-generated fixes is published at jasonzhanglll/qwen3.6-35b-a3b-swe-lora-stage2-v01.

This repository contains only the adapter weights. The base model must be obtained separately from the base repository at the exact revision below.

Base model

The adapter configuration (adapter_config.json) records this repository id in base_model_name_or_path and the commit in revision. The file as written by the training job (adapter_config.original.json, kept verbatim) had both fields set to null because the base weights were mounted from a local path inside the training container; no other field was changed.

Training data

Combined set sft_v01_combined (assembled 2026-10-11), built from two public datasets with pinned revisions:

Totals: 1,489 training samples from 66 repositories (SWE-smith 989 + SWE-Gym 500) and a 141-sample development split from 5 repositories that are disjoint from every training repository. Difficulty tiers in train: T1 1,060 (one file, at most 2 hunks and 20 diff lines), T2 415, T3 14. Token budgets, measured with the base tokenizer and chat template: whole sample at most 4,096 tokens, assistant turn at most 1,500 tokens; over-budget samples were dropped, never truncated (train p50 / p90 / max total tokens 2,908 / 3,716 / 4,095; assistant p50 / p90 / max 311 / 597 / 1,425). The training set contains about 4.29 M tokens.

Sample shape (ms-swift messages format, chat template qwen3_5, non-thinking target):

• System prompt, verbatim: You are an expert software engineer. Given a repository issue and the relevant source files, produce a minimal, correct fix as a unified diff that applies withgit applyfrom the repository root. Do not modify tests unless the issue requires it.
• User turn: # Issue (problem statement), # Failing tests (FAIL_TO_PASS test names only, at most 20), # Repository context (for each file touched by the fix, the buggy version: the whole file if it has at most 120 lines, otherwise windows of 60 lines on each side of every hunk plus the first 15 lines and the enclosing def/class line, with right-aligned line numbers).
• Assistant turn: The fix changes N file(s): <paths>. followed by one ```diff fence containing a git unified diff that applies with git apply from the repository root of the buggy tree.

Filtering and controls applied while building the set:

• SWE-smith: 18,033 rows with an empty problem statement dropped; repositories removed if they belong to a public benchmark (5 repositories), are not Python (91), carry a copyleft or non-allowlisted license (5), or were reserved for the SWE-Gym source (10). Of 2,534 candidate rows built from 60 repositories, 2,067 built successfully; dedup removed 9 exact fix duplicates and 405 rows sharing a (repository, FAIL_TO_PASS set) with an earlier row; a per-repository cap of 18 samples then bounded the train pool.
• SWE-Gym: every row was fetched and built; 2 rows dropped because the fix touches test paths, 10 stored patches failed to apply at their base commit, 1 build error, 1,715 rows (70 %) dropped because the real issue with its context does not fit 4,096 tokens; dedup removed 14 rows.
• Cross-source: 0 duplicate normalized diffs; a global cap of 8 % per repository removed nothing (largest repository: getmoto/moto, 113 samples, 7.6 %).
• Licenses: only repositories licensed MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, PSF-2.0 or MPL-2.0 were used (SPDX identifiers from the GitHub license API cross-checked with the license text at each base commit); copyleft repositories were excluded. The dataset cards' MIT tags do not by themselves license the third-party repository code; the per-repository licenses are the operative record.
• Benchmark-contamination control: an exclusion list of 46 repositories was built from public identity metadata only (all repositories of SWE-bench Full, Verified, Lite, Multimodal and SWE-bench Pro public splits, plus the 11 repositories of the public SWE-bench Pro V2 HARD-51 instance ids). 5 SWE-smith repositories were hit and removed; 0 SWE-Gym repositories were hit. Benchmark contents (problem statements, reference patches, hidden tests, answers or judge prompts) were never read during data construction.
• Validation: every training and development sample was re-applied on a rebuilt tree at its base commit (git apply --check and git apply), with the touched files compared byte for byte against the reference; 0 failures.

Training setup

Checkpoint selection rule: lowest development loss among the saved checkpoints, ties resolved toward the latest step. The published adapter is checkpoint-93, the final step.

Provenance of these numbers: the training job's own logging.jsonl and loss log written by ms-swift (evaluation every 25 steps on the 141-sample development split), consistent with the values printed in the job's log stream; the first logged training loss, at step 1, was 0.39816.

What is in the adapter

adapter_model.safetensors holds 62,060 bfloat16 tensors with 1,279,457,280 parameters in total (about 2.57 GB). All tensor names follow the PEFT convention base_model.model.model.language_model.layers.<i>.<module>.lora_{A,B}.weight, with lora_A of shape [16, in_features] and lora_B of shape [out_features, 16].

Three parameter figures describe this adapter: 939.0 M (938,997,760) trainable parameters in the fused training view, 1,279.5 M (1,279,457,280) parameters stored in the exported adapter, and a 2.57 GB file (2,568,923,184 bytes). The difference of 340,459,520 (340.46 M) is the LoRA-A matrices of projections that Megatron keeps fused but Hugging Face splits, which the export duplicates once per split target (routed-expert linear_fc1 -> gate_proj/up_proj: 335.54 M; Gated DeltaNet in_proj -> four in_proj_* targets: 2.95 M; attention qkv -> q_proj/k_proj/v_proj: 0.66 M; shared-expert linear_fc1 -> gate_proj/up_proj: 1.31 M), all measured from the tensor header; the function computed is unchanged, since a fused weight $W=[W_1;W_2]$ with update $BA$ splits into $B_1A$ and $B_2A$ with the same $A$. The four pipeline ranks hold unequal shares (ranks 0 and 2 carry one more Gated DeltaNet layer than ranks 1 and 3), and only rank 0 logged its count (234.7745 M).

The per-expert LoRA tensors account for 98.3 % of the adapter parameters. shared_expert_gate is matched by the target_modules regex but has no stored LoRA weights; a loader that injects LoRA into it will warn about missing keys and leave it as a zero-initialised (no-op) LoRA, which is harmless. inference_mode is false in the saved configuration, as written by the trainer; PEFT switches it to inference mode when loading with the default is_trainable=False.

Loading

Verification status of this section: the snippets were checked for configuration and key-name compatibility against transformers 5.19.0 and PEFT 0.21.2 (the adapter configuration parses with PeftConfig.from_pretrained; the tensor-name prefix matches Qwen3_5MoeForConditionalGeneration). End-to-end loading of the 35B base with this adapter was not executed while preparing this card. Evaluation results will be reported separately.

Important compatibility note: routed-expert LoRA and transformers

In transformers 5.x the routed experts of this architecture are implemented as fused parameters (mlp.experts.gate_up_proj, mlp.experts.down_proj) rather than as per-expert nn.Linear modules. PEFT's module-based LoRA therefore has nothing to attach the 61,440 per-expert tensors of this adapter to: PeftModel.from_pretrained will load the attention, linear-attention and shared-expert LoRA weights (620 tensors, about 1.7 % of the adapter parameters), silently drop the per-expert tensors as unexpected keys, and warn about the missing shared_expert_gate keys. A model loaded that way is not the trained model. To apply the adapter in full, use either a serving stack with LoRA support for fused MoE experts and for the Gated DeltaNet projections (see the vLLM section), the training framework's own loader (ms-swift, which exported this adapter), or the merged full weights produced by the same training job (checkpoint-93-merged, about 70 GB in Hugging Face format; not published in this repository).

Transformers + PEFT

The base checkpoint is a Qwen3_5MoeForConditionalGeneration (vision-language) model whose text decoder lives under model.language_model. The adapter's tensor names and its target_modules regex (^(model\.language_model(?=\.).*\.(...))$) match that structure, so the base must be loaded with AutoModelForImageTextToText (or the class directly). AutoModelForCausalLM resolves to Qwen3_5MoeForCausalLM, a text-only wrapper whose layers live under model.layers; its module names do not match the regex and the adapter will not attach to it. The architecture is implemented natively in current transformers releases (5.19.0 was checked; the base configuration carries no auto_map), so trust_remote_code is not required.

import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen3.6-35B-A3B"
REVISION = "995ad96eacd98c81ed38be0c5b274b04031597b0"
ADAPTER = "jasonzhanglll/qwen3.6-35b-a3b-swe-lora-sft-v01"

tokenizer = AutoTokenizer.from_pretrained(BASE, revision=REVISION)
model = AutoModelForImageTextToText.from_pretrained(
    BASE, revision=REVISION, dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)  # see the compatibility note above
model.eval()

messages = [
    {"role": "system", "content": "You are an expert software engineer. Given a repository issue and the relevant source files, produce a minimal, correct fix as a unified diff that applies with `git apply` from the repository root. Do not modify tests unless the issue requires it."},
    {"role": "user", "content": "# Issue\n...\n\n# Failing tests\n...\n\n# Repository context\n..."},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
).to(model.device)
out = model.generate(**inputs, max_new_tokens=1500, do_sample=True, temperature=0.7, top_p=0.8, top_k=20)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

vLLM (OpenAI-compatible server with LoRA)

The adapter is meant to be served as a LoRA module on top of the pinned base revision. The command below uses the same chat and tool-calling flags as the base-model service it is compared against; --max-lora-rank 16 matches the adapter rank. Check that your vLLM version supports LoRA for Qwen3_5MoeForConditionalGeneration including the fused MoE experts and the linear-attention projections; without that support the per-expert part of the adapter is not applied.

vllm serve Qwen/Qwen3.6-35B-A3B \
  --revision 995ad96eacd98c81ed38be0c5b274b04031597b0 \
  --dtype bfloat16 --tensor-parallel-size 4 \
  --trust-remote-code \
  --enable-lora --max-lora-rank 16 \
  --lora-modules sft_v01=jasonzhanglll/qwen3.6-35b-a3b-swe-lora-sft-v01 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --served-model-name qwen3.6-35b-a3b

Request the adapter by its module name and keep thinking disabled:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="sft_v01",
    messages=[
        {"role": "system", "content": "You are an expert software engineer. ..."},
        {"role": "user", "content": "# Issue\n...\n\n# Failing tests\n...\n\n# Repository context\n..."},
    ],
    temperature=0.7, top_p=0.8, max_tokens=1500,
    extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)

Non-thinking mode

The adapter was trained on non-thinking targets (add_non_thinking_prefix true): the assistant turns contain no reasoning trace. Use the model in non-thinking mode (enable_thinking=False in apply_chat_template, or chat_template_kwargs={"enable_thinking": false} in the OpenAI-compatible request) so that inference matches the training format. The sampling values above (temperature 0.7, top_p 0.8, top_k 20, no repetition penalty) follow the base model's recommendation for non-thinking mode.

Intended use

Repository-level patch generation for Python projects: given an issue description, the names of failing tests and the relevant source context, produce a minimal unified diff that applies with git apply. The adapter is a research artifact intended for evaluation of supervised fine-tuning on software-engineering data and as a starting point for further training (for example execution-feedback continued SFT).

Limitations

• Small training set (1,489 samples, Python only, about 4.29 M tokens) and a single epoch of LoRA; the base model's general capabilities are otherwise unchanged.
• 66 % of the training samples are synthetic SWE-smith bugs with generator-written problem statements, mostly single-file, small diffs (tier T1 is 71 % of train). The 4,096-token budget removed 70 % of SWE-Gym rows, so the real-issue subset is biased toward short fixes.
• The repository context given to the model is cut around the gold hunks, so fault localisation is partly provided; the model learned patch writing more than repository search. No agentic tool-use or multi-turn samples were used.
• No reinforcement learning, preference optimisation or distillation was applied.
• The adapter contains per-expert LoRA weights that plain transformers + PEFT cannot attach to the fused expert implementation (see the compatibility note); use a stack with fused-MoE LoRA support or the merged weights.
• This card reports no benchmark results. Evaluation of this adapter (together with the base model and later training stages) is carried out separately under a fixed protocol and will be reported once all compared weights are frozen.
• The training-data licenses were classified mechanically from API metadata and license text; copyleft and unknown cases were excluded, but no legal review was performed.

Files

License

This adapter is released under the Apache License 2.0, the license of the base model Qwen/Qwen3.6-35B-A3B (the base repository declares license: apache-2.0 and ships the license text, which is reproduced here as LICENSE). Use of the adapter requires the base model and is subject to the base model's license. The training data were drawn from repositories under permissive licenses (MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, PSF-2.0); the SWE-smith and SWE-Gym dataset cards are tagged MIT.

Citation

Base model:

@misc{qwen3.6,
  title  = {Qwen3.6-35B-A3B},
  author = {Qwen Team},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Qwen/Qwen3.6-35B-A3B}},
  note   = {revision 995ad96eacd98c81ed38be0c5b274b04031597b0}
}

Training data:

@misc{yang2025swesmith,
  title  = {SWE-smith: Scaling Data for Software Engineering Agents},
  author = {Yang, John and Lieret, Kilian and Jimenez, Carlos E. and Wettig, Alexander and Khandpur, Kabir and Zhang, Yanzhe and Hui, Binyuan and Press, Ofir and Schmidt, Ludwig and Yang, Diyi},
  year   = {2025},
  eprint = {2504.21798},
  archivePrefix = {arXiv}
}

@misc{pan2024swegym,
  title  = {Training Software Engineering Agents and Verifiers with SWE-Gym},
  author = {Pan, Jiayi and Wang, Xingyao and Neubig, Graham and Jaitly, Navdeep and Ji, Heng and Suhr, Alane and Zhang, Yizhe},
  year   = {2024},
  eprint = {2412.21139},
  archivePrefix = {arXiv}
}