Qwen2.5-3B-Base Medical SFT (QLoRA + Trainable Full Embeddings)
Official production SFT checkpoint for clinical and biomedical reasoning, fine-tuned on 45,000 verified medical instruction examples (Dataset-Medical-SFT45k-RaR-GRPO-V1).
Model Details
- Base Model:
Qwen/Qwen2.5-3B (Pinned official revision: 3aab1f1954e9cc14eb9509a215f9e5ca08227a9b)
- Fine-tuning Technique: QLoRA NF4 (double quantization) with full shared trainable embeddings (
embed_tokens master weights in FP32).
- LoRA Parameters: $r=16$, $\alpha=32$, dropout=0.05, target modules:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
- Trainable Parameters: 341,098,496 (~11.0% of total 3.09B parameters):
- Shared Input/Output Embeddings: 311,164,928 parameters (FP32)
- LoRA Adapter Matrices: 29,933,568 parameters
- Sequence Context: 4,096 tokens max length (512 token overlap, no packing, ChatML format with
<|im_end|>).
- Optimization: PagedAdamW8bit, cosine learning rate scheduler to 0, effective batch size 32 (4 microbatch $\times$ 8 gradient accumulation).
- Training Duration: 2 complete epochs (2,870 optimizer steps, 91,788 windows seen, 128,620,906 supervised tokens).
Evaluation Results
Evaluated every 150 steps across the full 2,500 validation examples (3,548,077 supervised tokens):
| Domain / Split |
Examples |
Initial Loss (Step 150) |
Final Loss (Step 2,870) |
Initial PPL |
Final PPL |
| Overall Validation |
2,500 |
1.1954 |
1.0207 |
3.30 |
2.78 |
| MedQuad (Formal Medical Q&A) |
375 |
1.0447 |
0.8814 |
2.84 |
2.41 |
| MUSER (Clinical Case Reasoning) |
1,500 |
1.1716 |
0.9965 |
3.23 |
2.71 |
| iCliniq (Patient-Doctor Dialogue) |
33 |
2.3913 |
2.1633 |
10.93 |
8.70 |
| HealthCareMagic (Telemedicine) |
592 |
2.3874 |
2.2180 |
10.89 |
9.19 |
- Early stopping counter: 0 bad evaluations (monotonic loss reduction across all 20 evaluation checkpoints).
How to Use
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model_id = "Qwen/Qwen2.5-3B"
peft_model_id = "Matycaba2016/qwen2.5-3b-base-qlora-embeddings-sft45k-v1"
tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
base_model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
model = PeftModel.from_pretrained(base_model, peft_model_id)
model.eval()
messages = [
{"role": "system", "content": "Sos un médico especialista y asistente clínico de alta precisión. Respondé con fundamentación médica exhaustiva."},
{"role": "user", "content": "¿Cuáles son las pautas actuales para el manejo de la cetoacidosis diabética en adultos?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=True, temperature=0.7, top_p=0.9)
response = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
print(response)
