Qwen3.5-0.8B Medical (MLP-only fine-tune, experiment 1)
Medical disclaimer
This model is a research experiment. It is not a medical device and must not be used for
diagnosis, treatment, triage, or any clinical decision-making.
- Outputs may be inaccurate, incomplete, outdated, or confidently wrong, and may include
fabricated reasoning, drug names, doses, or references.
- The model has not been clinically validated, safety-tested, or reviewed by medical
professionals.
- It is not a substitute for advice from a qualified healthcare professional. If you have
a medical concern or emergency, contact a licensed clinician or your local emergency services.
- Do not enter personal health information or patient data.
- The authors accept no liability for any use of this model or its outputs.
Model description
Qwen/Qwen3.5-0.8B fine-tuned directly (no LoRA/adapters) on medical reasoning data, to turn a
general-purpose model into a domain-focused one. Overfitting to the training data was an
accepted trade-off of this experiment; the goal was to study the training/validation loss behaviour.
Training details
| Item |
Value |
| Base model |
Qwen/Qwen3.5-0.8B |
| Dataset |
FreedomIntelligence/medical-o1-reasoning-SFT (en config) |
| Split |
80 / 10 / 10 train / validation / test (seed 42) |
| Trainable parameters |
MLP blocks (gate/up/down proj) of the first 5 layers (~55M params, 7.3%) |
| Frozen |
Attention / DeltaNet layers, embeddings, norms, output head, all other MLPs |
| Epochs |
10 |
| Optimizer |
AdamW, lr 1e-5, cosine schedule, 3% warmup |
| Effective batch size |
16 |
| Max sequence length |
1536 tokens |
| Precision |
bf16 autocast, fp32 master weights for trained layers |
| Regularisation |
dropout 0.0, weight decay 0.0, label smoothing 0.0 (baseline run) |
| Checkpoint uploaded |
<best validation loss, epoch N> |
| Hardware |
<GPU used> |
Loss is computed on the assistant response only (reasoning + answer); the prompt is masked.
Results
|
Loss |
Perplexity |
| Base model, test split |
<x.xx> |
<x.xx> |
| This model, test split |
<x.xx> |
<x.xx> |
Best validation loss <x.xx> at epoch <N>. See loss_curve.png for train vs validation loss.
<Describe the plateau / overfit behaviour you observed.>
Intended use
- Research into domain adaptation, partial fine-tuning, and overfitting behaviour of small models.
- Offline experimentation with medical-style question answering and reasoning formats.
Out-of-scope use
- Clinical decision support, diagnosis, treatment recommendations, or patient-facing tools.
- Any setting where an incorrect answer could cause harm.
Limitations and biases
- Small (0.8B) model trained on a single synthetic dataset; knowledge is narrow and may be wrong.
- Training data reasoning was generated by GPT-4o and verified automatically, not by clinicians,
so it can contain errors and inherit biases.
- English only. Deliberate overfitting means generalisation beyond the training distribution is
likely weaker than the base model's.
- The model may produce long "thinking" text before an answer, and can loop in thinking mode.
How to use
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "adityag-india/qwen3.5-0.8b-medical-mlp5"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "What are common causes of microcytic anemia?"}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
ids = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
Citation / acknowledgements
Base model: Qwen team. Dataset: FreedomIntelligence, medical-o1-reasoning-SFT.
