slm-parameter-audit

SLM card-vs-artifact parameter audit

An autonomous audit of small-language-model repos on the Hugging Face Hub, comparing the parameter count each model card claims against the parameter count actually present in the artifact (safetensors header), cross-checked with config.json and the training/export code where present.

This is a snapshot, not a live feed. It was regenerated on 2026-09-20 03:37 UTC from the live verification store. The store grows as new models are checked, so the numbers below describe the state of the store at that moment. Treat any figure here as "as of 2026-09-20".

Scope

In-scope models are small, from-scratch language models from the independent SLM community — roughly a few hundred thousand to a few hundred million parameters. A finetune of a big lab's small model is generally not what this community builds; training from scratch in the small/ultra-small range is the point. Models outside that range are recorded as out_of_scope and excluded from the checked counts.

Method

For each model: 1. Read the model card and record the claimed parameter count. 2. Parse the safetensors header and count stored tensor elements. 3. Exclude non-parameter buffers (__metadata__, rope.cos/rope.sin, _extra_state 1-element entries). 4. For tied models, exclude a redundant lm_head from the intended param count (but note when the file stores one, since that inflates the file size). 5. Cross-check against config.json and the training/export code. 6. Record the outcome.

A clean outcome means the card and the artifact agree within rounding (e.g. a card saying "about 303M" vs 303,350,784 stored is clean).

Current numbers (as of 2026-09-20 03:37 UTC)

The full per-outcome breakdown is in findings.json.

What "mismatch" covers

The 85 mismatches span several distinct failure modes, each with its own label in findings.json:

• card_param_mismatch (57) — the headline number on the card does not match the artifact.
• card_param_mismatch_untied_head (4) — the card's tied count is right, but the checkpoint stores a redundant untied lm_head, so the file is larger than the card implies.
• name_param_mismatch_minor (4) — the param count in the repo name is off from the artifact by a small margin.
• card_precision_mismatch (2) — the dtype/precision claim is wrong.
• card_dtype_mismatch (2) — the stored dtype differs from the card.
• Plus smaller counts (1 each): card_param_mismatch_and_untied_head, untied_lm_head_and_precision_mismatch, card_counts_masks_as_params, card_intermediate_mismatch, card_says_tied_but_untied, architecture_mislabel_and_broken_script, card_tag_understates, card_omits_layernorm_params, card_size_overstated, card_active_param_mismatch, config_layer_mismatch, card_config_mismatch, card_param_overstated_minor, card_param_and_config_mismatch, name_param_mismatch, card_arch_nit.

Caveats

• Element counts come from the safetensors header; __metadata__ is not a tensor and is excluded.
• Non-parameter buffers (rope tables, _extra_state) are excluded from param counts.
• For tied models the redundant lm_head (where present) is excluded from the intended count, but the file size reflects what is actually stored.
• Rounding is tolerated: "about 303M" vs 303,350,784 is clean.
• This is a snapshot. The live verification store is the source of truth and grows over time.

Reproduction

Each check is recorded with the repo id, the claimed count, the measured count, and a one-line note. The store is maintained by @Compactbot on behalf of @CompactAI. To regenerate this file, re-aggregate the live verification store and rewrite findings.json + this README with the new numbers.