Open-weight model
Qwen1.5-MoE-A2.7B-SNLI-Task-Only
by Dinhtracducanh ducanhdinh/Qwen1.5-MoE-A2.7B-SNLI-Task-Only
Qwen1.5-MoE-A2.7B-SNLI-Task-Only is an open-weight model from Dinhtracducanh, released under Apache License 2.0. Its published files total 985.7 MB.
Day la LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B, mot mo hinh Mixture-of-Experts, tren task SNLI (Stanford Natural Language Inference) — phan loai quan he giua premise/hypothesis thanh entailment / neutral / contradiction.
Model Card
By Dinhtracducanh, published under apache-2.0, revision f809affbf34a.
Day la LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B, mot mo hinh Mixture-of-Experts, tren task SNLI (Stanford Natural Language Inference) — phan loai quan he giua premise/hypothesis thanh entailment / neutral / contradiction. (tuong ung khoang 1L/3 -> 2L/3), giong het setting cua code alignment finetuning. thanh phan mot rank rieng qua rankpattern cua PEFT: - alpha = 32, dropout = 0.05 Ltotal = LLM + lblosscoef LLB - LLM: cross-entropy chuan tren token tiep theo, CHI tinh tren phan (prompt bi mask, giong instruction-tuning/SFT tieu chuan). - LLB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune. - lblosscoef (lambda) =…
Read Dinhtracducanh's full model card
Day la LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B, mot mo hinh Mixture-of-Experts, tren task
SNLI (Stanford Natural Language Inference) — phan loai quan he giua premise/hypothesis thanh
entailment / neutral / contradiction.
Prompt format
Premise: <premise>
Hypothesis: <hypothesis>
Question: What is the relationship between the premise and the hypothesis? Choose one: entailment, neutral, or contradiction.
Answer: <entailment|neutral|contradiction>
Cau hinh LoRA
- Layer duoc finetune:
[8, 16)trong tong so24layer (tuong ung khoang 1L/3 -> 2L/3), giong het setting cua code alignment finetuning. - Module duoc gan LoRA: attention, router, experts trong khoang layer tren, moi
thanh phan mot rank rieng qua
rank_patterncua PEFT: - attention: r = 16
- router: r = 4
- experts: r = 16
- alpha = 32, dropout = 0.05
Loss
Loss MoE tieu chuan:
L_total = L_LM + lb_loss_coef * L_LB
L_LM: cross-entropy chuan tren token tiep theo, CHI tinh tren phan<label_word><eos>(prompt bi mask, giong instruction-tuning/SFT tieu chuan).L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.lb_loss_coef(lambda) = 0.01num_experts= 60,top_k= 4
Du lieu
SNLI, doc tu data/english_task/snli/train.json (list object {"premise", "hypothesis", "label"}, label
0/1/2 = entailment/neutral/contradiction).
Diagnostics
Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png,
diagnostics/loss_curve_smoothed.png va diagnostics/accuracy_curve.png (do chinh xac xap xi
tren token dau tien cua nhan, tinh tren batch train, da lam min moi 50 step).
Identity and Version
- Repository
- ducanhdinh/Qwen1.5-MoE-A2.7B-SNLI-Task-Only
- Publisher
- Dinhtracducanh
- Task
- Not stated by the source
- Modality
- Other
- Library
- peft
- Parameters
- Not stated by the source
- Languages
- moe, nli
- Revision
- f809affbf34aadda441ea9ba18ebf7dc0ad83cca
- First published
- 2026-09-24
- Last updated
- 2026-09-24
Files and Weights
10 files, 985.7 MB in total. The weights are 2 files totalling 984.5 MB in pt, safetensors.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| adapter_model.safetensors | Weights | 327.6 MB | 3052b0c144d9 |
| trainer_state.pt | Weights | 656.9 MB | bb8c8d3f27e7 |
| adapter_config.json | Configuration | 161.1 KB | — |
| README.md | Documentation | 2.0 KB | — |
| diagnostics/accuracy_curve.png | Other | 58.5 KB | — |
| diagnostics/loss_curve.png | Other | 87.1 KB | a1a39b5f77f9 |
| diagnostics/loss_curve_smoothed.png | Other | 75.9 KB | — |
| diagnostics/loss_log.jsonl | Other | 898.1 KB | — |
| diagnostics/loss_log.jsonl.bak-failed-runs | Other | 408 B | — |
| .gitattributes | Repository | 1.6 KB | — |
License and Download
- License
- apache-2.0
- Access
- Open weights, no gate
- Download size
- 984.5 MB
Released by Dinhtracducanh through its official repository on Hugging Face. Read the license.
Built From
- Adapter of Qwen/Qwen1.5-MoE-A2.7B
- Derived from Qwen/Qwen1.5-MoE-A2.7B
- Trained on (disclosed) snli
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 984.5 MB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Questions About Qwen1.5-MoE-A2.7B-SNLI-Task-Only
Can I use Qwen1.5-MoE-A2.7B-SNLI-Task-Only commercially?
Yes. Qwen1.5-MoE-A2.7B-SNLI-Task-Only is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.