Qwen3.6-35b-nca

NCA preference continuation of V6

This repository contains the preserved LoRA adapter checkpoint, tokenizer and native chat template. It requires the original base model Qwen/Qwen3.6-35B-A3B at revision 995ad96eacd98c81ed38be0c5b274b04031597b0; it is not a standalone merged model.

Adapter SHA256: 9bb6cc8bdd71bd126112a1e9b6965352b509f3ca6e8938be3276757d81edeaa8. Size: 130,065,504 bytes; 380 FP32 tensors / 32,501,760 parameters. The NCA adapter already includes the continued V6 weights: apply it directly to the original base, do not stack it on top of V6.

Training

Continued a copy of V6 SFT with NCA_pair preference loss (not standard DPO). The reference was the exact frozen V6 adapter. One epoch of 184 training pairs (149 coding, 35 math), 23 held-out pairs, 23 optimizer updates, beta 0.1, LR 2e-6, microbatch 1, accumulation 8, two warmup steps, linear schedule, weight decay 0.01, gradient clipping 1.0, seed 42. Complete prompt/answer branches fit 2048 tokens. BF16 base, FP32 attention-only rank-32/alpha-32 adapter, fresh optimizer. The exact starting V6 SHA256 is 420b8682d0e042f110f521a28a6f85e95491b343d1fb73bafb669e3b0004db91.

Preference sources: UltraInteract_pair, revision 3692843c72c3a297c7914cdf76c0ffce5e38a234, with coding checks using training-split CodeContests, revision 802411c3010cb00d1b05bad57ca77365a3c699d6. V6's source data are inherited through its starting adapter. These mostly short responses differ from long benchmark agent trajectories. No raw training data are distributed here.

CPU normalization checks and exact reference tensor checks passed. Pre-update dtype/OOM repairs occurred before any optimizer update. The held-out NCA objective worsened from 1.38629 to 1.39223, while preservation token loss improved slightly. Manual adjudication of the 23 paired generated answers found both models had 12 correct, one wrong and ten cap-censored unknown outcomes; candidate output tokens increased by 3.9%. This is mixed evidence, not proof of improvement.

Retrospective label-validity limitations

All 207 historical pairs (184 training / 23 validation), including 168 coding pairs / 336 coding branches, were structurally inspected. Six training rejected branches (indices 28, 33, 62, 102, 162, 175) executed an empty regex match instead of the visible nested-fence program. Their execution receipts do not prove the intended program failed; this also does not establish that the complete responses were correct. No chosen or validation branch had this extraction defect.

Four of seven specifically examined coding domains had malformed or out-of-domain tests; 85 other scaffold domains were unaudited. This targeted audit is not a population error-rate estimate. Valid public failure witnesses remain for the kth-digit, calendar and Trichotomy negatives. Although the old factorial source suite was malformed, an independent valid-input test at N=3 confirms its rejected program is wrong (2,12 instead of 2,6); it is not an additional extraction-mismatch case. These defects invalidate blanket claims of functional verification but do not establish a causal explanation for benchmark differences. The original trained checkpoint is preserved without retroactively changing labels or scores.

A later DPO feasibility check approved zero pairs and did not train another model. This release is the first NCA attempt only. See training_provenance.json; its original planned benchmark scope predates the subsequently authorized SWE/ASI evaluation shown below.

Evaluation

Budget-limited, single-attempt subsets; these are not full official benchmark scores.

Terminal used a 128K context, 16K response limit, 64 turns and concurrency 8. HLE V6/NCA used concurrency 32 and native 262144 context without an explicit output cap; the historical base used concurrency 16 and an 81920 output cap. The historical base Terminal heap result includes a verifier download failure. QEMU verifier dependency failures are infrastructure errors, not demonstrated model errors. SWE results use the repaired transport series; NCA's eight tasks combine a prior single qutebrowser attempt with seven subsequent tasks. ASI uses fractional scoring, not a pass rate. Small samples, stochastic generation and protocol differences limit conclusions. NCA does not consistently improve every benchmark over V6.

Native reasoning can overrun generation limits; agents can fail to finish or produce required files. No claim of contamination freedom or general superiority is made. Lexical exclusions do not rule out semantic overlap or upstream exposure.

Loading

Use Transformers with Qwen3_5MoeForConditionalGeneration support and PEFT (the recorded training environment used PEFT 0.18.1). The BF16 base requires substantial GPU memory. The following downloads a fixed Hub revision and applies only this adapter:

import torch
from huggingface_hub import HfApi, snapshot_download
from transformers import AutoTokenizer, Qwen3_5MoeForConditionalGeneration
from peft import PeftModel

repo = "Ruixiao/Qwen3.6-35b-nca"
# Resolve once and record this immutable revision for reproduction.
revision = HfApi().model_info(repo).sha
adapter_dir = snapshot_download(repo, revision=revision)
tokenizer = AutoTokenizer.from_pretrained(adapter_dir)
base = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3.6-35B-A3B", revision="995ad96eacd98c81ed38be0c5b274b04031597b0",
    torch_dtype=torch.bfloat16, device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_dir, is_trainable=False)
model.eval()

Use the included native chat template and its enable_thinking option for text conversations. Only language attention was trained; multimodal improvements were not evaluated. No new GPU inference test is implied by publication or file-hash verification.

License and contents

Released under Apache-2.0. The original base LICENSE is included unchanged (Copyright 2026 Alibaba Cloud); tokenizer/template files originate from the pinned base. This release adds the trained adapter and experiment documentation. Dataset sources retain their respective licenses; no raw dataset, credentials, optimizer state or merged base weights are included. SHA256SUMS.json records the release payload's file hashes.