Qwen3.6-35b-lora

V6 supervised fine-tuning

This repository contains the preserved LoRA adapter checkpoint, tokenizer and native chat template. It requires the original base model Qwen/Qwen3.6-35B-A3B at revision 995ad96eacd98c81ed38be0c5b274b04031597b0; it is not a standalone merged model.

Adapter SHA256: 420b8682d0e042f110f521a28a6f85e95491b343d1fb73bafb669e3b0004db91. Size: 130,065,504 bytes; 380 FP32 tensors / 32,501,760 parameters. This is the final step-94 V6 adapter.

Training

Fresh original base; attention-only LoRA rank 32, alpha 32, dropout 0; BF16 base and FP32 adapter. Experts, routers, vision, embeddings and LM head were frozen. One epoch, 94 optimizer updates, LR 1e-5, microbatch 1, accumulation 16, warmup ratio 0.05, weight decay 0.01, gradient clipping 1.0, seed 42. Native chat template, assistant-only loss including reasoning/tool calls, whole records up to 16384 tokens, no packing or answer truncation.

The frozen dataset contains 1498 training records / 4,001,044 supervised tokens and 82 validation records: 459 coding records (2,000,835 tokens), 141 math (800,324), 134 base-answer anchors (799,743), and 764 data-analysis workflows (400,142). Sources are SWESmith trajectories, Nemotron-SFT-Math-v4, FineEnvs data-agent-sft, and reference-checked base answers to math/science prompts including OpenScienceReasoning-2. Reference agreement does not prove every reasoning step; upstream execution was not universally replayed. Raw training data are not distributed here.

After a VM interruption, training resumed the exact optimizer/scheduler/RNG checkpoint at step 47; the exported final adapter is step 94. This was a smaller mixed-data experiment, not a controlled replacement of only the earlier dataset's weak reasoning. See training_provenance.json for the preserved run metadata.

Evaluation

Budget-limited, single-attempt subsets; these are not full official benchmark scores.

Terminal used a 128K context, 16K response limit, 64 turns and concurrency 8. HLE V6/NCA used concurrency 32 and native 262144 context without an explicit output cap; the historical base used concurrency 16 and an 81920 output cap. The historical base Terminal heap result includes a verifier download failure. QEMU verifier dependency failures are infrastructure errors, not demonstrated model errors. SWE results use the repaired transport series; NCA's eight tasks combine a prior single qutebrowser attempt with seven subsequent tasks. ASI uses fractional scoring, not a pass rate. Small samples, stochastic generation and protocol differences limit conclusions. NCA does not consistently improve every benchmark over V6.

Native reasoning can overrun generation limits; agents can fail to finish or produce required files. No claim of contamination freedom or general superiority is made. Lexical exclusions do not rule out semantic overlap or upstream exposure.

Loading

Use Transformers with Qwen3_5MoeForConditionalGeneration support and PEFT (the recorded training environment used PEFT 0.18.1). The BF16 base requires substantial GPU memory. The following downloads a fixed Hub revision and applies only this adapter:

import torch
from huggingface_hub import HfApi, snapshot_download
from transformers import AutoTokenizer, Qwen3_5MoeForConditionalGeneration
from peft import PeftModel

repo = "Ruixiao/Qwen3.6-35b-lora"
# Resolve once and record this immutable revision for reproduction.
revision = HfApi().model_info(repo).sha
adapter_dir = snapshot_download(repo, revision=revision)
tokenizer = AutoTokenizer.from_pretrained(adapter_dir)
base = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3.6-35B-A3B", revision="995ad96eacd98c81ed38be0c5b274b04031597b0",
    torch_dtype=torch.bfloat16, device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_dir, is_trainable=False)
model.eval()

Use the included native chat template and its enable_thinking option for text conversations. Only language attention was trained; multimodal improvements were not evaluated. No new GPU inference test is implied by publication or file-hash verification.

License and contents

Released under Apache-2.0. The original base LICENSE is included unchanged (Copyright 2026 Alibaba Cloud); tokenizer/template files originate from the pinned base. This release adds the trained adapter and experiment documentation. Dataset sources retain their respective licenses; no raw dataset, credentials, optimizer state or merged base weights are included. SHA256SUMS.json records the release payload's file hashes.