kev-9B — Production Alert-Triage Decision Model
A fine-tuned decision head that has been triaging real production alerts, resident, for 72 days — zero errors. Not a demo: alertmanager → this model → triaged log → daily aggregation → weekly reports, every day.
Production Record (the part that matters)
| Metric |
Value |
How to verify |
| Resident deployment |
72+ days, no restart |
triage log grows daily |
| Real alert stream |
1,902 raw → 123 deduped types |
zero processing errors |
| Human arbitration (40 disputed cases) |
kev 90% vs static rules 10% |
reasons published per case |
| Agreement with static rules |
17.9% — and that's the point |
disagreement = the value proposition |
Why the disagreement is the feature, not the bug: kev downgrades high-frequency auto-recovering alerts (the #1 source of alert fatigue) and escalates orchestration-core outages that static rules treat as routine.
Method
- Architecture: Qwen3.5-9B + LoRA + a 256-d rank head (classification head retrained, not just LM adaptation)
- Objective: RPS ordinal loss (severity is an ordered scale: info < warning < critical — train it that way)
- None-injection (p=0.1): teaches the head to abstain, not everything deserves a label
- Calibration: LBFGS temperature fit post-training — when it says 0.7 confidence, it means 0.7
- Training data: 123 deduplicated real alert types, 10-source multi-benchmark synthesis
Honest Negative Result (read this before deploying anything)
We ran the obvious next experiment: fine-tune it on its own production data (5-fold grouped CV, 4-arm temperature calibration). Result: 0 gain over the deployed version — and we did not deploy it, per preset criteria defined before the experiment.
That's the discipline this model is proud of: judge the upgrade on pre-registered criteria, not vibes. Training receipts and the arbitration table are in the ModelScope card (below).
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-9B", torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "jjshin/kev-9B-LoRA-ranker")
tok = AutoTokenizer.from_pretrained("jjshin/kev-9B-LoRA-ranker")
# rank head: model.head (256-d → 3 classes: info / warning / critical)
Files: adapter_model.safetensors (173MB) + head.pt (rank head) + tokenizer — load LoRA then attach the head, see training_config.json for wiring.
Receipts
- Full card (bilingual, with the 40-case arbitration table & A/B eval on 8 suites): ModelScope theGAI/kev-9B-LoRA-ranker
- A 10B sibling (kev-0.8B) exists for size comparison; the 9B head is the production one
中文：kev-9B = 生产告警分诊决策模型——常驻生产 72 天零错误、40 条分歧人工仲裁胜率 90%（静态规则 10%）、回流实验 0 增益按预设判据不部署。LoRA + 256 维排序头 + RPS 序数目标。完整双卡（含仲裁表与八套件 A/B）见 ModelScope。
