Thaki-SecEmbed-8B-Preview

Security-domain specialized experimental preview.

한국어 카드:MODEL_CARD.ko.md

Preview release. This model is evaluated on one task only: retrieving MITRE ATT&CK® technique documentation from analyst-style queries in English and Korean. Do not read the results as general security or general-domain retrieval performance.

Headline. Security-domain specialized experimental preview. No statistically clear difference from the stock Qwen3-Embedding-8B baseline under the preregistered held-out evaluation. On general retrieval: no catastrophic regression under the preregistered 10% threshold; significant FiQA degradation was observed.

Model summary

Thaki-SecEmbed-8B is a LoRA specialization of Qwen/Qwen3-Embedding-8B. It retrieves ATT&CK technique documentation for security-analyst queries. Queries can be questions, claims, keywords, evidence requests, analyst searches, or TTP hypotheses, in English or Korean.

Intended use

• ATT&CK technique retrieval. Given a query, rank ATT&CK technique and sub-technique documents. Typical uses are candidate retrieval for TTP mapping, analyst search over ATT&CK, and a first-stage retriever in front of a reranker.
• Research on domain-specialized embeddings.

Out of scope

• General-domain retrieval as a primary use. On four public retrieval sets the adapter shows no catastrophic regression versus the base model, but it is not tuned for general retrieval, and finance retrieval (FiQA-2018) drops measurably. See General retrieval retention. For general-purpose retrieval, use the base model.
• Retrieval over other security corpora, such as CTI reports, logs/telemetry, detection rules, or vulnerability databases. These were not evaluated.
• Automated detection, attribution, or response decisions without human review.
• Languages other than English and Korean.

Usage

Load the base model explicitly at the pinned revision, as below. From v0.0.1 the adapter_config.json also names the base model and revision, so PEFT auto-loading works too; v0.0.0 keeps the evaluated file, whose base path is a local training path.

Only the query side receives an instruction. Documents are embedded as raw text. Pooling uses the last token (left padding) followed by L2 normalization, the same as the base model.

Instruction used in training and evaluation:

Given a security analyst query, retrieve documentation of the adversary technique or tactic it describes.

Query format:

Instruct: {instruction}
Query: {query}

import torch
import torch.nn.functional as F
from peft import PeftModel
from transformers import AutoModel, AutoTokenizer

INSTRUCTION = ("Given a security analyst query, retrieve documentation of the adversary "
               "technique or tactic it describes.")

BASE, REV = "Qwen/Qwen3-Embedding-8B", "1d8ad4ca9b3dd8059ad90a75d4983776a23d44af"
tok = AutoTokenizer.from_pretrained(BASE, revision=REV, padding_side="left")
base = AutoModel.from_pretrained(BASE, revision=REV, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "ThakiCloud/Thaki-SecEmbed-8B-Preview").eval()

def embed(texts, max_len):
    enc = tok(texts, padding=True, truncation=True, max_length=max_len, return_tensors="pt")
    with torch.no_grad():
        h = model(**enc).last_hidden_state[:, -1]          # last token (left padding)
    return F.normalize(h.float(), p=2, dim=-1)

q = embed([f"Instruct: {INSTRUCTION}\nQuery: {x}" for x in ["Can malware download files from the C2?"]], 512)
d = embed(["<ATT&CK technique document text>"], 2048)     # documents: raw text, no instruction
scores = q @ d.T

Dimensions

The primary embedding dimension is 4096. This Preview does not train with a Matryoshka (MRL) objective. Truncated dimensions (1024 / 512 / 256) are a diagnostic only: they are not optimized, and this card makes no Matryoshka claim. Held-out nDCG@10 after truncation and L2 re-normalization (583 queries; Δ vs stock with paired-bootstrap 95% CI):

Training

Data (public only)

• Source: MITRE ATT&CK STIX, Enterprise ATT&CK v19.2 (enterprise-attack.json, from https://github.com/mitre/cti). Used under the ATT&CK Terms of Use.
• Retrieval corpus: every live ATT&CK technique and sub-technique, rendered into one document each by a fixed renderer. The training positives use the same renderer, so document formatting cannot reveal the answer.
• Queries: generated by a teacher LLM from grounded ATT&CK source spans. The generator was ThakiCloud/Qwen3.8-27B-NVFP4-GPTQ-txt (revision 615fbec2b9beb5a0dfb0858a7be036e354dff20a), ThakiCloud's NVFP4 quantization of Qwen/Qwen3.8-27B, served with z-lab/Qwen3.8-27B-DFlash2 as a speculative-decoding draft. The served files were checked byte-for-byte against those public repositories. There are six archetypes (analyst search, claim, evidence request, keyword, question, TTP hypothesis), in English and Korean. Generated queries passed deterministic gates before use: format, answer/ID leakage, and source grounding. Rejected queries were dropped, not regenerated.
• No internal, customer, or proprietary data is included in this release.

Hard-negative mining

• Candidate negatives came from the union of BM25 and the stock Qwen3-Embedding-8B rankings.
• False-negative removal is a fixed, deterministic rule. For each query it excludes two kinds of candidate:
• other techniques linked to the same ATT&CK source anchor, since these are also correct answers;
• the gold technique's parent and sibling sub-techniques.
• Nemotron-3-Embed-8B was not used for mining. It served only as an evaluation baseline, which keeps it an independent comparison.

Objective

• Contrastive InfoNCE: one positive plus hard negatives plus in-batch negatives, at τ = 0.05.
• Plus a listwise distillation term from Qwen/Qwen3-Reranker-8B (revision 77d193c791ed757ca307ee72715aa132723da912): teacher-softened listwise KL, student temperature fixed at τ=0.05.
• The teacher temperature is applied only to the teacher's logits. There is no T² rescaling.
• Teacher scores were computed once and cached. They were bit-exact reproducible under the fixed scoring configuration.
• KD weight λ_KD = 4, selected by a pre-registered 0.6B loss canary from {0, 1, 4} (3 seeds each, tuning-dev split disjoint from the held-out set). In the canary, contrastive-only training (λ_KD = 0) fell well below the untuned 0.6B model, and distillation recovered and exceeded it.
• The canary used Qwen3-Embedding-0.6B and served only to eliminate λ options. It is not evidence of 8B performance.

Hyperparameters

Evaluation

Evaluation set: a held-out set of generated queries from the same generator, built before training and never used for mining, hard-negative selection, model selection, or query generation.

• It covers 17 parent techniques, which is a narrow slice of ATT&CK.
• Its queries come from the same generator and distribution as the training queries.
• The results therefore measure in-distribution ATT&CK technique retrieval. They are not an out-of-distribution or real-analyst-traffic estimate.
• Size: 583 queries, materialized from 595 planned; slots that failed the generation gates were dropped, not replaced.

Metric: nDCG@10 (primary). Recall@1/5/10 and MRR@10 are also reported. Retrieval runs over the full ATT&CK technique corpus.

Baselines: BM25 (no instruction), stock Qwen3-Embedding-8B (same instruction), and stock Nemotron-3-Embed-8B.

Pre-registered verdict vs the base model: no statistically clear difference. Δ nDCG@10 (Thaki-SecEmbed-8B − stock Qwen3-Embedding-8B) = +0.016 [-0.010, +0.043] (paired bootstrap over queries, 95%). No statistically clear difference from the stock baseline under the preregistered held-out evaluation. This does not establish equivalence, and we make no claim of improvement over the base model.

On this pre-registered ATT&CK-grounded held-out benchmark, Thaki-SecEmbed-8B outperformed Nemotron-3-Embed-8B (Δ nDCG@10 +0.033 [+0.009, +0.057]), while showing no statistically clear difference from its Qwen3-Embedding-8B base model. The held-out queries share the training generator and distribution, so this is not evidence of a general advantage over Nemotron. Nemotron was run with its own official input format (query:/passage: prefixes, mean pooling), checked against its model card example.

Per-slice Δ nDCG@10 vs stock (95% CI). No slice shows a significant regression, but the point estimates for analyst_search and evidence_request are negative:

General retrieval retention

Measured against the stock Qwen3-Embedding-8B on four public retrieval sets at 4096 dimensions, each with its own query instruction, nDCG@10, paired bootstrap 95% CI. The pre-registered rule flags a set as a catastrophic regression only if the relative drop exceeds 10% and the CI upper bound of the difference is below 0. Sets are judged one by one; results are not averaged.

No catastrophic regression under the preregistered 10% threshold; significant FiQA degradation was observed. This is not evidence that general capability is preserved. Finance retrieval (FiQA-2018) drops by 0.040 nDCG@10 (-6.2% relative), and the drop is statistically clear. If you retrieve over financial Q&A, prefer the base model.

Training variability across three independently trained seeds

Same recipe and data, different seeds. The released seed was fixed before any held-out result was seen; the other two are reported only and were never used for selection.

Mean 0.689, range 0.676–0.705 (stock 0.670). The seed-to-seed spread is about as large as the released adapter's difference from the base model, which is one reason this Preview makes no improvement claim.

Limitations

• Narrow evaluation. The held-out set covers 17 parent techniques and shares its generator with the training data, so reported gains may not transfer to real analyst queries or to other ATT&CK regions.
• Synthetic queries. All queries are LLM-generated, and stylistic artifacts of the generator may be learned.
• Finance retrieval regression. FiQA-2018 nDCG@10 is 0.040 lower than the base model (-6.2% relative, 95% CI excludes 0). Retention was measured on four sets only; other domains were not tested.
• Truncated dimensions are not trained with MRL. See Dimensions.
• ATT&CK version drift. The model was trained on Enterprise ATT&CK v19.2. Techniques added, renamed, or deprecated later are not reflected.
• Teacher dependence. Training used knowledge distillation from Qwen3-Reranker-8B, so the model inherits the teacher's preferences over hard negatives.

Reproducibility

• Base model artifacts were verified by sha256 against the pinned HF revision before every run.
• Training and evaluation inputs (queries, qrels, corpus, teacher scores) are pinned by sha256 in an append-only decision ledger. Pre-registration was committed before results were opened.
• Not released in this Preview: the held-out queries, qrels, and query-to-technique mapping. They are kept sealed so the benchmark stays independent for future model versions.
• Described here instead: the ATT&CK source version, how the benchmark was built (595 planned → 583 materialized deterministically; failed slots dropped, not replaced), the metric definitions, paired-bootstrap confidence intervals, aggregate results, and all eight slice results.

Upstream models and licenses

Attribution

MITRE ATT&CK® is a registered trademark of The MITRE Corporation. Thaki-SecEmbed-8B is not affiliated with, sponsored by, or endorsed by The MITRE Corporation.

Training data was derived from MITRE ATT&CK under the ATT&CK Terms of Use. This repository does not redistribute ATT&CK content.

Revision history

Citation

@misc{han2026thakisecembed,
  title        = {Thaki-SecEmbed-8B-Preview: Security-Domain Specialized Experimental Embedding Model},
  author       = {Han, Hyo-Jung},
  year         = {2026},
  organization = {ThakiCloud},
  url          = {https://huggingface.co/ThakiCloud/Thaki-SecEmbed-8B-Preview}
}