ettin-150m-memory-reranker-ft-v1

Ettin-150M memory reranker

An English cross-encoder for ranking passages from notes, procedures, decisions and technical documentation. Give it a query and candidate passages; it returns one relevance score per pair. It is a fine-tune of Ettin Reranker 150M and runs locally through ONNX Runtime.

This revision adds a Vulkan FP32 graph, model.vulkan.onnx; the learned weights and the CUDA FP16 graph are unchanged. It omits the DirectML graph, which remains in the previous revision.

Quick start

Install tokenizers, numpy and onnxruntime-gpu[cuda,cudnn]==1.24.4. Keep model.onnx, model.onnx.data and the tokenizer from the same package together. This CUDA example assumes that package is in downloaded-model and scores two passages:

import numpy as np
import onnxruntime as ort
from pathlib import Path
from tokenizers import Tokenizer

root = Path("downloaded-model")  # The exact downloaded serving package.
tokenizer = Tokenizer.from_file(f"{root}/tokenizer.json")
tokenizer.no_padding()
tokenizer.no_truncation()
ort.preload_dlls()
session = ort.InferenceSession(
    f"{root}/model.onnx", providers=[("CUDAExecutionProvider", {"use_tf32": "0"})]
)
if "CUDAExecutionProvider" not in session.get_providers():
    raise RuntimeError("CUDA did not initialize; check the ONNX Runtime CUDA dependencies.")
query = "What must happen before a database migration?"
passages = [
    "Take a verified backup before applying the migration.",
    "The dashboard theme uses a blue background.",
]
query_tokens = tokenizer.encode(query, add_special_tokens=False)
if len(query_tokens.ids) > 128:
    raise ValueError("Query exceeds the qualified 128-token content limit.")
scores = []
for passage in passages:
    passage_tokens = tokenizer.encode(passage, add_special_tokens=False)
    passage_tokens.truncate(1022)
    pair = tokenizer.post_process(query_tokens, passage_tokens, add_special_tokens=True)
    inputs = {
        "input_ids": np.array([pair.ids], dtype=np.int64),
        "attention_mask": np.array([pair.attention_mask], dtype=np.int64),
    }
    scores.append(float(session.run(None, inputs)[0].reshape(-1)[0]))
print(sorted(zip(passages, scores), key=lambda item: item[1], reverse=True))

Higher scores rank first; raw scores are not probabilities. The input envelope is 128 query-content tokens plus 1,022 passage-content tokens and three special tokens, a maximum of 1,153 tokens. The example rejects an oversized query and truncates the passage, matching the tested input policy.

Evaluation

Upstream and fine-tuned Ettin scored the same 970 pools of 50 candidates, with all 48,500 pairs judged. The table uses the 873 answerable pools that contain at least one useful passage, so it measures ordering after retrieval. “Upstream” is the exact starting checkpoint without Daecore fine-tuning.

Grades 2 and 3 count as useful. Hit asks whether at least one useful passage appears, precision measures the useful share of returned positions, and nDCG rewards higher grades near the top. Many pools contain several useful passages, so random ordering already reaches 97.30% Hit@20; early precision and graded ranking are more informative than deep Hit. The comparison runs both models in PyTorch FP16 with matched tokenization on reused development inputs. The evaluation companion has every cutoff through 20, all-query results, model identities and metric code.

On public data, both models rerank the same 50 candidates that upstream Gemma retrieved for each query, scored with official relevance judgments:

These are nDCG@10 scores for reranking fixed candidate pools, not dense Gemma scores. Fine-tuning lowered FiQA and slightly raised SciFact. Later continuation recipes did not recover FiQA while keeping Daecore ranking quality, so the original learned weights remain.

Training data and objective

Training passages come from generated organizational documents and one person's project documentation, much of it AI-written. Queries include full questions, keywords, identifiers and pasted fragments. Model judges assigned grades from 0 to 3, and several passages can be useful for one query. ListNet learns ordering within each candidate list, and a pairwise term emphasizes distinctions between neighboring grades. Lists rotate through the full labeled pools, so useful, partly useful and unhelpful passages stay in the comparison.

Runtime and limits

model.onnx is the CUDA FP16 graph and model.vulkan.onnx the Vulkan FP32 graph; both read model.onnx.data. The Vulkan derivation rewrites bounded mask operations and keeps the learned weights. The FP32 graph's results were also checked for correctness on the CPU execution provider; no CPU latency guidance is given.

Vulkan requires onnxruntime==1.24.4 and the native WebGPU plugin onnxruntime-ep-webgpu==0.4.0, registered explicitly. Add the plugin device to the session options with dawnBackendType=Vulkan, and create the session without a providers list; the plugin name alone does not select Vulkan. serving.vulkan.json records the tested provider options.

On 970 Daecore candidate pools, CUDA and Vulkan produced the same Hit@5, Hit@10 and Hit@20; Hit@3 differed by one query and nDCG@10 by 0.0004. The paths are not bit-exact, so close scores can swap order. Checks used Windows x64 with an NVIDIA RTX 3060 Ti (8 GB); other GPUs and Linux are untested, even where Vulkan is available. Inside Daecore, Ettin reorders up to 50 deduplicated BM25 and Gemma candidates; the retrieval pipeline overview describes that path and result selection.

The task-specific panel is mostly generated and model-labeled, with no human reference, and was reused during development; training-seed variation is unmeasured. A reranker cannot recover a passage missing from its candidates, and several relevant passages may repeat one fact. These results do not establish unique-evidence coverage or downstream agent success.

License

Apache-2.0. The package includes upstream attribution and a modification notice.