LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B theo baseline MidAlign, Alternate Training: step chan = task step (LM loss), step le = contrastive align step. (prompt bi mask), tren pool gop cua cac tap: squad, snli, mmlu. - lblosscoef = 0.01, numexperts = 60, topk = 4 tieng Anh va cau target tai layer 12 (block 0-indexed = 11 / 24 layer), temperature = 1.5. task (SQuAD/SNLI/MMLU, English-only) >= 1 lan. Align = pool nho 121571 cap (nguon: flores, ntrex, ted; 500 cap/ngon ngu, boc ngau nhien seed 42), lap lai de du 724736 cap align / epoch (dung nhu paper). - worldsize = 8, alignbatchsize = 16 (per-rank, anchor), taskbatchsize = 16 (per-rank), taskmicrobatchtokens = 4096. - Range layer [8…
Independent publisher
Dinhtracducanh
ducanhdinh
Models in Library2
Datasets in Library0
Models on Hugging Face43
Followers—
Compare the published evidence
This profile connects the 2 model and dataset records shown below to their upstream cards and sources. The library counts describe SAVRN's coverage, rather than this publisher's entire output.
Reported licenses: apache-2.0. Open a record to review its specific license and conditions.
Compare model specifications, source dates, and hardware estimates on the individual records. Publisher identity verification is separate from a model's quality, deployment readiness, or suitability for your work.
Models
Day la LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B, mot mo hinh Mixture-of-Experts, tren task SNLI (Stanford Natural Language Inference) — phan loai quan he giua premise/hypothesis thanh entailment / neutral / contradiction. (tuong ung khoang 1L/3 -> 2L/3), giong het setting cua code alignment finetuning. thanh phan mot rank rieng qua rankpattern cua PEFT: - alpha = 32, dropout = 0.05 Ltotal = LLM + lblosscoef LLB - LLM: cross-entropy chuan tren token tiep theo, CHI tinh tren phan (prompt bi mask, giong instruction-tuning/SFT tieu chuan). - LLB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune. - lblosscoef (lambda) =…