vtok101-attribution-baselines

Data-attribution scores over lamsheeper-data-attribution/Qwen3.5-4B-d0-vtok101-lora-seeds: 3 function counts x 7 document counts x 4 seeds, scored by 12 methods.

Each training document defines one synthetic constant function, and each query asks for one function's value. The ground truth for a query is the set of documents describing its function, so a method is measured by how far up its ranking those documents come.

Every document in the pool describes some function, which is the easy case: real attribution never has that luxury. The arm that removes it, one same-template hard negative per document, is in lamsheeper-data-attribution/vtok101-distr-attribution-baselines under the same run/group paths.

Layout

{run}/{group}/scores.npz          the primary method's [query, train] matrix
{run}/{group}/scores.{method}.npz  the group's other methods
{run}/{group}/metrics.json        AUC and Recall@R, overall and per function
{run}/{group}/config.json         every argument the ranker was given

run is f{functions}_{docs}d_sd{seed}, matching the adapter repo's subfolders. group is one of ekfac_margin, trak, logra_margin, tracin_margin, source_margin, bm25, embed.

Every method differentiates the margin on both sides. That is TRAK as published, and an ablation for the others -- the influence-function derivation fixes the training-side gradient to the training objective -- so those groups carry a _margin suffix and the cross-entropy grid stays at {run}/{group}/. The margin's advantage on this corpus is carried substantially by answer-token identity rather than by function attribution; see filter/story/WHY_MARGIN_LATE_LAYERS.md in the source repository before quoting these numbers.

The v2 rerun, under v2/

Everything above describes the original grid, which is at the top level and is frozen. The active sweep is v2, under the v2/ prefix, and it is a rerun rather than an extension: the numbers at the top level were produced before a padding bug was found in Qwen's linear-attention layers, which let pad tokens into per-sample gradients and moved the unprojected methods by +0.37 to +0.42 AUC when fixed. filter/story/padding_bug.md in the source repository is the account of it, and filter/baselines/SWEEP_V2.md is the plan for the rerun. Do not mix the two prefixes in one comparison.

v2/{run}/{group}/scores.{method}.{arm}.npz
v2/{run}/{group}/metrics.json      keyed "method@arm"
v2/{run}/{group}/config.json       every argument, plus provenance

Three things differ beyond the fix:

• Arms. Each job scores several training-side objectives in one pass, sharing the query gradients and the curvature: ce (cross-entropy), margin, and margin_docw (the margin scaled per document by 1 - mean_t p_t, TRAK's residual). The measurement side is always the margin. So a v2 file names both its method and its arm, where a v1 file named only its method and put the training loss in the directory suffix.
• Groups. TRAK and LoGra are merged into sketch, since they differ only in what is done with the same projected gradients and the arms made that pass the expensive half of both. {run}/logra_margin/scores.npz under v1 is v2/{run}/sketch/scores.logra.margin.npz under v2.
• Conventions are pinned, not defaulted. Padding is trimmed dynamically at collate time on every path, the curvature is fitted from cross-entropy, every loss is oriented so that helpful is positive, and queries are capped at four per function. v1 left these to the ranker's defaults, which is how a re-scored cell could differ from its neighbours without anything recording it. Every v2 file now carries convention, orientation, git_commit and git_dirty, in the npz and in config.json.

The orientation matters for anyone reading v1 too: under v1 the margin was a utility while cross-entropy was a loss, so a margin/cross-entropy pairing came out inverted and consumers negated it by filename. Under v2 every kind is a loss and nothing needs negating. A file with no convention field is v1 by construction.

The v2 grid is a subset: documents per function in {1, 5, 10, 30, 50}, seeds 1001-1003, function counts 25 and 50 first (stage 1) and 100 after (stage 2).

What is in a scores.npz

scores          float32 [n_query, n_train]
train_uids      corpus uid of each column
train_func      the function each training document describes
train_role      "constant" for a real document, "distractor" for a decoy
train_source    the function a decoy shadows, empty for a real document
query_uids      query uid of each row
query_func      the function each query asks about
query_correct   whether the model answers that query correctly

float32 rather than float16, which was tried: an unprojected EK-FAC score is a dot product of two 10^8-dimensional gradients and runs past float16's 65,504 ceiling, which ties most of the matrix and reads as chance.

train_role and train_source are what let a decoy be scored as a third document class -- signal, its own twin, everything else -- rather than merely as a negative. metrics.json reports decoy_auc (a real document against the query's own decoys) and decoy_top1 (how often a decoy takes the top rank) wherever there are decoys.

Methods

• ekfac group: if-ekfac (EK-FAC influence functions), grad-dot (no curvature) and grad-sim (the cosine). One pass produces all three, since they differ only in what happens to the representation after it is computed.
• trak group: trak, in the dual form over factorised random projections.
• logra group: logra, projected gradients with a block-diagonal Fisher.
• tracin group: tracin and tracin-cos, the learning-rate-weighted sum of gradient dot products along the training run, on raw and unit-normalised gradients respectively.
• source group: source, the same trajectory with each segment preconditioned by the unrolled sum (1 - (1 - eta*lam)^T)/lam in the eigenbasis of a projected Fisher. That factor is TracIn's learning-rate weight at a segment width of one step and an inverse Fisher in the limit, so SOURCE sits between the two methods above it here, on the same gradients and the same checkpoints. After Bae et al., Training Data Attribution via Approximate Unrolled Differentiation.
• bm25 group: bm25 and bm25-prompt, Okapi BM25 over the raw text, with and without the query's answer token.
• embed group: embed and embed-base, cosine similarity under BAAI/bge-base-en-v1.5 and under this suite's own base model read at its last token.

Padding is excluded from the Fisher, and attention runs under sdpa. Both choices, and why they matter more than they look, are in filter/DATTRI_PARITY.md of the source repository.

The retrieval baselines saturate this benchmark

bm25 and embed are not attribution methods and should not be read as ones. They are here to bound what the gradient methods are being credited for, and the bound is not comfortable: BM25 scores 1.000 AUC and 1.000 recall@R on both arms, including against the hard negatives.

It is a lookup, not a discovery. Every document describing <B01> contains the token <B01> and no other document does, so the query names its own answer and BM25 reads it off. The whole method is the tokenizer -- keeping <B01> whole through glued punctuation is worth 0.999 AUC, and a plain whitespace split scores 0.500 and puts a decoy top for every query.

The embedding pair is the other end. Both sit at chance (0.51 and 0.53), because a dense vector of a 512-token document cannot preserve the identity of one rare token in it, whether or not the encoder has that token in its vocabulary.

So: a gradient method scoring below 1.0 here is being beaten by grep, and any claim made from this grid has to be made against that rather than against chance.

The trajectory groups cover half the grid, and a sibling of each run

tracin and source score a trajectory rather than a finished model, and these sweeps kept no mid-run checkpoints. The trajectories are a separate retrain published under {run}/ckpt/ in the adapter repository, covering two of the four seeds, so their cells exist only for seeds 1001 and 1002 and their figures should be read against the other methods restricted to those seeds.

A retrain is a sibling of the published run, not its history: the recipe is not reproducible run to run. What makes the comparison fair anyway is that the last checkpoint of each trajectory is that trajectory's own final model, so these methods attribute to a model that reaches the same accuracy by its own path. See manifest["reproducible"] and models/v4/ckpt_arm.py in the source repository.

The bm25 and embed groups cover one seed, for the opposite reason: they never load the model, so all four seeds of a cell would give the same matrix.

Contributing

Anyone in the lamsheeper-data-attribution organization can fill in gaps:

git clone <repo> && cd influence-benchmarking-hops && uv sync
hf auth login
.venv/bin/python filter/baselines/selftest.py --suite vtok101
filter/baselines/launch.sh --suite vtok101 0 1

Workers coordinate through this repository's file listing, so they do not need to know about each other.