Open-weight model
Qwen1.5-MoE-A2.7B-MidAlign
by Dinhtracducanh ducanhdinh/Qwen1.5-MoE-A2.7B-MidAlign
Qwen1.5-MoE-A2.7B-MidAlign is an open-weight model from Dinhtracducanh, released under Apache License 2.0. Its published files total 518.6 MB. It draws 560 downloads a month.
LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B theo baseline MidAlign, Alternate Training: step chan = task step (LM loss), step le = contrastive align step. (prompt bi mask), tren pool gop cua cac tap: squad, snli, mmlu.
Model Card
By Dinhtracducanh, published under apache-2.0, revision 15d496a3af5c.
LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B theo baseline MidAlign, Alternate Training: step chan = task step (LM loss), step le = contrastive align step. (prompt bi mask), tren pool gop cua cac tap: squad, snli, mmlu. - lblosscoef = 0.01, numexperts = 60, topk = 4 tieng Anh va cau target tai layer 12 (block 0-indexed = 11 / 24 layer), temperature = 1.5. task (SQuAD/SNLI/MMLU, English-only) >= 1 lan. Align = pool nho 121571 cap (nguon: flores, ntrex, ted; 500 cap/ngon ngu, boc ngau nhien seed 42), lap lai de du 724736 cap align / epoch (dung nhu paper). - worldsize = 8, alignbatchsize = 16 (per-rank, anchor), taskbatchsize = 16 (per-rank), taskmicrobatchtokens = 4096. - Range layer [8…
Read Dinhtracducanh's full model card
Qwen1.5-MoE-A2.7B-MidAlign (task = SQuAD + SNLI + MMLU)
LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B theo baseline MidAlign, Alternate
Training: step chan = task step (LM loss), step le = contrastive align step.
Alternate Training
- Task step (chan):
L_task = L_LM + lb_loss_coef * L_LB.L_LMchi tinh tren phan dap an (prompt bi mask), tren pool gop cua cac tap: squad, snli, mmlu. - squad: 87599 sample (doc 87599, bo 0)
- snli: 549367 sample (doc 549367, bo 0)
- mmlu: 87691 sample (doc 99842, bo 12151)
lb_loss_coef= 0.01,num_experts= 60,top_k= 4- Align step (le): contrastive Eq.(1) MidAlign (oneway, negative trong mini-batch 4) giua mean-pooled hidden state cau tieng Anh va cau target tai layer 12 (block 0-indexed = 11 / 24 layer), temperature = 1.5.
- Moi epoch: 5662 task step + 5662 align step. TASK la anchor: moi epoch dung HET 724657 sample task (SQuAD/SNLI/MMLU, English-only) >= 1 lan. Align = pool nho 121571 cap (nguon: flores, ntrex, ted; 500 cap/ngon ngu, boc ngau nhien seed 42), lap lai de du 724736 cap align / epoch (dung nhu paper).
- Contrastive: effective batch 16/rank, mini-batch 4, global_negatives = True, mask_false_negatives = True, lang_balance = True.
- world_size = 8, align_batch_size = 16 (per-rank, anchor), task_batch_size = 16 (per-rank), task_micro_batch_tokens = 4096.
LoRA
- Range layer
[8, 16)(0-indexed), attention / router / experts + shared expert. - Rank: attention 8, router 8, experts 8; alpha = 16, dropout = 0.1.
Training
- 3 epoch, gradient all-reduce thu cong (khong DDP), checkpoint chi giu ban moi nhat.
- Diagnostics (log theo tung step:
diagnostics/loss_log.jsonl):
Task steps (L_LM, L_LB, L_task_total, token acc)
Align steps (contrastive loss)
Identity and Version
- Repository
- ducanhdinh/Qwen1.5-MoE-A2.7B-MidAlign
- Publisher
- Dinhtracducanh
- Task
- Not stated by the source
- Modality
- Other
- Library
- peft
- Parameters
- Not stated by the source
- Languages
- moe
- Revision
- 15d496a3af5c2f429e05a11c67c8b9a4b1af22c5
- First published
- 2026-08-20
- Last updated
- 2026-10-07
Files and Weights
8 files, 518.6 MB in total. The weights are 2 files totalling 512.7 MB in pt, safetensors.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| adapter_model.safetensors | Weights | 170.3 MB | 60db75ca3b25 |
| trainer_state.pt | Weights | 342.4 MB | 2d3653d958cb |
| adapter_config.json | Configuration | 71.9 KB | — |
| README.md | Documentation | 2.0 KB | — |
| diagnostics/align_loss_curve.png | Other | 78.4 KB | a29394d2d48d |
| diagnostics/loss_log.jsonl | Other | 5.5 MB | 734184affc54 |
| diagnostics/task_loss_curve.png | Other | 209.4 KB | a8050d4cc5ef |
| .gitattributes | Repository | 1.8 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 512.7 MB
Released by Dinhtracducanh through its official repository on Hugging Face. Read the license.
Built From
- Adapter of Qwen/Qwen1.5-MoE-A2.7B
- Derived from Qwen/Qwen1.5-MoE-A2.7B
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 512.7 MB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About Qwen1.5-MoE-A2.7B-MidAlign
Can I use Qwen1.5-MoE-A2.7B-MidAlign commercially?
Yes. Qwen1.5-MoE-A2.7B-MidAlign is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.