commercecore-expansion-relevance-singletoken-v1

CommerceCore Expansion — relevance adapter, single-token label attempt

A separate project from arghya2030/commercecore-qwen3-1.7b. No shared weights, no shared training run, no shared serving process.

Status: research checkpoint, a rejected hypothesis, preserved as evidence. This adapter is NOT the model of record for relevance — arghya2030/commercecore-expansion-match-v1 (v1, shared adapter, 0.526 accuracy) remains the best real, adopted result. Read why this one is published anyway, below.

What this is

A QLoRA rank-16 adapter on an independently-pinned original Qwen/Qwen3-1.7B base, testing the fifth of five distinct hypotheses for why the shared adapter's match_relevance accuracy (0.526) falls short of the strongest tested frontier model, claude-sonnet-5 (0.583).

The hypothesis this tests

Two earlier dedicated-relevance-adapter attempts (see the main project README) collapsed: the first predicted only the majority class (exact); the second, with class-weighted loss, still never predicted substitute or complement even once. Direct tokenization showed exact is the only single-subword-token label among the four (substitute, complement, irrelevant are each 2 tokens) — a plausible structural cause distinct from class imbalance.

This adapter remaps each of the four classes to a single distinct letter token (A/B/C/D) for training and generation, removing the token-length variable entirely, then maps predictions back to real label names only for scoring.

Real, measured result — the collapse is fixed, but it's still not a win

Full dev-set evaluation (n=4,000, natural class distribution):

Not a collapse — all four classes are predicted, including complement, which got zero correct predictions in the second (2-token) attempt. This confirms token length was a real, partial cause of that earlier failure.

But it's not an improvement either: 0.5115 is marginally below the shared adapter's 0.526, and both remain short of claude-sonnet-5's 0.583. The remaining gap looks like substitute acting as an over-predicted catch-all, stealing recall from complement and irrelevant.

Verdict

Five distinct, falsifiable hypotheses for the relevance shortfall have now been tested (majority-class collapse under high LR / no weighting; class-balanced oversampling within the shared adapter; 5x data scaling within the shared adapter; class-weighted 2-token dedicated adapter; class-weighted single-token dedicated adapter, this one) — none has closed the gap. The shared adapter (v1) remains the adopted model for match_relevance. This checkpoint is published for transparency and as a base for any future attempt, not as a replacement.

What this is NOT

• Not the model of record for relevance — use arghya2030/commercecore-expansion-match-v1 instead.
• Not a win against frontier models.
• Not load-tested or evaluated under production traffic conditions.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B")
model = PeftModel.from_pretrained(base, "arghya2030/commercecore-expansion-relevance-singletoken-v1")
tokenizer = AutoTokenizer.from_pretrained("arghya2030/commercecore-expansion-relevance-singletoken-v1")

TOKEN_TO_LABEL = {"A": "exact", "B": "substitute", "C": "complement", "D": "irrelevant"}

prompt = (
    "Classify the query-product relevance. Respond with exactly one letter:\n"
    "A = exact match, B = substitute, C = complement, D = irrelevant.\n"
    "Query: wireless bluetooth headphones\n"
    "Product: Sony WH-1000XM5 Wireless Noise Canceling Headphones\n"
    "Answer:"
)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=4, do_sample=False)
letter = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()[:1]
print(TOKEN_TO_LABEL.get(letter, "invalid"))

Full evaluation methodology and data provenance: github.com/arghya05/commercecore-expansion.