vlm-calibration-prefix-vqav2

A small non-autoregressive yes/no visual question answering model built for research on calibrated decisions. It answers a yes/no question about an image in one forward pass, with no generation and no verbalised confidence. The confidence is the softmax over two token logits followed by a single post-hoc temperature.

How it works

• Image. A frozen SigLIP2-base encodes the image into its pooled vector (pooler_output, 768-d).
• Prefix token. That vector becomes one prefix token, prepended to the input embeddings of ModernBERT-base:

prefix = anchor + g_max * tanh(alpha) * LN_out(proj(LN_in(v)))

• anchor is a learned token, initialised from the " image" embedding.
• alpha is a scalar gate, initialised to 0, so the model starts as exactly the text-only model.
• g_max caps the vision term's norm at the anchor's norm.
• Readout. The prompt is "{question} Answer: [MASK]". The answer is read with ModernBERT's own pretrained MLM head at [MASK], comparing the logits of the " yes" and " no" tokens. There is no new classification head.
• Calibration. Probabilities are calibrated by one scalar temperature T per checkpoint, fitted on a held-out calib split:

p_yes = sigmoid((logit_yes - logit_no) / T)

• What is trained. ModernBERT (encoder and MLM head), the anchor, the projection and the gate are fine-tuned. SigLIP2 stays frozen and is not included in this repo; it is loaded by ID.

What is in this repo

This repo holds one checkpoint: the main model, seed 0 of the paired (image + question), hard-label configuration. It was trained on the full 214,476-row pool and scores 0.6791 eval accuracy.

Not included in this repo. The results section also reports these models as context; their weights are not published here:

• the other two seeds (s1, s2) of the same configuration;
• the blind (question-only) reference models;
• the soft-label ablation.

Usage

Tested on CPU with torch 2.14, transformers 5.17.0 and safetensors 0.8.0. ModernBERT needs a recent transformers; older versions are untested.

# pip install torch transformers safetensors huggingface_hub pillow
import os, sys
from huggingface_hub import hf_hub_download
from PIL import Image

repo = "25b3nk/vlm-calibration-prefix-vqav2"
sys.path.insert(0, os.path.dirname(hf_hub_download(repo, "modeling.py")))
from modeling import YesNoVQA

vqa = YesNoVQA.from_pretrained(repo)  # also fetches SigLIP2 + the ModernBERT tokenizer/config
image = Image.open("example.jpg")
print(vqa.predict([image], ["Is there a minute hand on the clock?"]))
# -> [{'answer': 'yes', 'margin': ..., 'p_yes_raw': ..., 'p_yes': ...}]   p_yes is temperature-scaled

Notes:

• predict returns p_yes (calibrated with that checkpoint's T), the uncalibrated p_yes_raw, and the margin logit_yes - logit_no.
• The training features were computed on JPEG re-encoded images (PIL default quality). predict(..., jpeg_roundtrip=True) (the default) applies the same re-encode. Pass jpeg_roundtrip=False if your files are already such re-encodes.
• The model needs ~1 GB of RAM in fp32 plus SigLIP2 (~0.4 GB).

Check against the training-run outputs. The snippet above, loaded from this Hub repo on CPU, was run on 6 examples from the eval split (4 "no" and 2 "yes" predictions, on 6 different images). It was compared with the predictions saved by this checkpoint's training run on GPU, for two inputs:

• the cached GPU SigLIP2 features;
• the full CPU image pipeline.

The predicted answers matched in every case. Measured differences are in the "Inference check" section at the end.

Results

All numbers come from the project's experiment logs. Eval is the VQAv2 val2014 yes/no image-disjoint eval split:

• 8,930 questions, split by md5(image_id) % 100, so no eval image appears in training or calib.
• It is not test-dev.
• "Accuracy" is exact match against VQAv2's multiple_choice_answer (yes or no). It is not the standard soft VQA accuracy metric, so it is not comparable to published VQAv2 numbers.
• The majority-answer-per-question-type floor on this split is 0.5106.

Main model vs blind reference (hard labels, 3 seeds, full 214,476-row pool)

This repo's checkpoint is paired seed 0. The seed means, the other seeds and the blind arm are context only; their weights are not included here.

• ECE is top-label ECE with 15 equal-width bins. T is fitted on a separate 2,911-question calib split by minimising NLL.
• Accuracy and AUROC do not depend on T.
• Before scaling, the paired models are over-confident (T > 1).

POPE (object hallucination probe), strict-clean subset

These are margin AUROCs on POPE's three variants, using only the strict-clean subset: 2,490 questions per variant, on 415 COCO val2014 images that appear in no training pool and not in calib. POPE images were excluded from training.

POPE accuracy for the paired model is above chance but modest: 0.603 / 0.613 / 0.653 (adversarial / popular / random) with the default threshold z >= 0.

Scaling curve (same architecture and recipe, 3 seeds per point)

How to read the curve:

• Paired accuracy rises at every step, and the last step is the largest. Blind stays near 0.55, so the growth comes from using the image.
• POPE grounding only appears at the full scale.
• Confounds. Optimisation steps grow with the data (3 epochs at every scale), and the data source changes past 47.6K. A same-size source check (47.6K rows from train2014 only, paired 0.5859) was inconclusive. The 100K → 214K step adds only train2014 rows.

Soft labels vs hard labels (context; the soft models are not included in this repo)

Soft labels are not presented as better. The experiment was pre-registered with ±0.010 decision bars:

• Accuracy (+0.006) and soft-target NLL (−0.0086) were both neutral by those bars, and the accuracy gain comes mostly from one seed.
• The main effect is "built-in temperature". Before post-hoc scaling, soft training is much less over-confident. After each model gets its own T, little difference remains.
• No soft blind arm was trained.
• The hard-label model stays the main model.

Training data

• VQAv2 yes/no questions (answer_type == "yes/no", majority answer yes or no). There are 214,476 training rows on 89,992 COCO images:
• 47,600 rows from the VQAv2 val2014 questions (via lmms-lab/VQAv2, validation split). These come from images in the training bucket of the image hash split, with all POPE images removed.
• 166,876 rows from the official VQAv2 train2014 questions and annotations, with images from lmms-lab-encoder/VQAv2 at revision 32665d3. Two train2014 images that were near-duplicates of an eval image were excluded.
• The eval and calib splits are also from val2014. They are image-disjoint from training, but they come from the same distribution as part of the training data.
• Recipe. AdamW, batch size 16, 3 epochs, and a cosine schedule. The learning rates are 2e-5 for ModernBERT and the anchor, 3e-4 for the new layers and 1e-3 for the gate. The kept checkpoint is the epoch end with the best calib accuracy. Training ran on a single T4.
• Hard labels: the VQAv2 majority answer. Soft labels: q = #yes / (#yes + #no) over the 10 annotator answers.

Intended use

• Intended: research on calibrated, single-pass decisions. Examples are temperature scaling, reliability analysis and selective prediction.
• Not intended: any production use. Never use it for decisions about people or anything safety-relevant. It is a research artefact.

Limitations

• Yes/no questions only. Other question types are out of scope; the readout only compares " yes" and " no".
• Modest accuracy. It scores about 0.67 on yes/no questions, against a majority-per-question-type floor of 0.5106. Roughly one answer in three is wrong. The question-only reference already reaches 0.55, so part of the behaviour is language prior.
• One pooled image vector. Fine spatial detail, counting and small objects are poorly served. POPE accuracy is only 0.60–0.65.
• The eval is a val2014 split, not test-dev. The accuracy metric is exact match on the majority answer, not the official VQA metric.
• Calibration is distribution-specific. T was fitted on VQAv2 val2014 calib questions. It may not transfer to other image or question distributions; refit it on your own data if calibration matters.
• Small seed sample. Three seeds per configuration; seed-to-seed spread is about 0.5–1 point of accuracy.
• Inherited biases. The model inherits the biases of COCO, VQAv2 annotators, SigLIP2 and ModernBERT.
• Training instability. Gradient-norm spikes were observed during training (reported, not fixed). This checkpoint (seed 0) had one, of 52.7, in its selected epoch.

License

• Weights and code in this repo: Apache-2.0. Both base models are Apache-2.0, checked on their Hugging Face pages: google/siglip2-base-patch16-224 and answerdotai/ModernBERT-base. Apache-2.0 keeps the weights under the same terms as the model they fine-tune (ModernBERT) and the encoder they depend on (SigLIP2).
• Training data. The VQAv2 annotations are CC BY 4.0, and are attributed below. The COCO images are subject to the COCO terms of use and the licenses of the individual Flickr images. No images are redistributed here.
• POPE was used for evaluation only.

Code

The training code and experiment logs are in a personal repository that is not public. modeling.py in this repo contains everything needed for inference.

Citations

@inproceedings{goyal2017vqav2,
  title     = {Making the {V} in {VQA} Matter: Elevating the Role of Image Understanding in Visual Question Answering},
  author    = {Goyal, Yash and Khot, Tejas and Summers-Stay, Douglas and Batra, Dhruv and Parikh, Devi},
  booktitle = {CVPR},
  year      = {2017}
}

@inproceedings{li2023pope,
  title     = {Evaluating Object Hallucination in Large Vision-Language Models},
  author    = {Li, Yifan and Du, Yifan and Zhou, Kun and Wang, Jinpeng and Zhao, Wayne Xin and Wen, Ji-Rong},
  booktitle = {EMNLP},
  year      = {2023}
}

@article{tschannen2025siglip2,
  title   = {{SigLIP} 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features},
  author  = {Tschannen, Michael and Gritsenko, Alexey and Wang, Xiao and Naeem, Muhammad Ferjad and Alabdulmohsin, Ibrahim and Parthasarathy, Nikhil and Evans, Talfan and Beyer, Lucas and Xia, Ye and Mustafa, Basil and H{\'e}naff, Olivier and Harmsen, Jeremiah and Steiner, Andreas and Zhai, Xiaohua},
  journal = {arXiv preprint arXiv:2502.14786},
  year    = {2025}
}

@article{warner2024modernbert,
  title   = {Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author  = {Warner, Benjamin and Chaffin, Antoine and Clavi{\'e}, Benjamin and Weller, Orion and Hallstr{\"o}m, Oskar and Taghadouini, Said and Gallagher, Alexis and Biswas, Raja and Ladhak, Faisal and Aarsen, Tom and Cooper, Nathan and Adams, Griffin and Howard, Jeremy and Poli, Iacopo},
  journal = {arXiv preprint arXiv:2412.13663},
  year    = {2024}
}

@inproceedings{lin2014coco,
  title     = {Microsoft {COCO}: Common Objects in Context},
  author    = {Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C. Lawrence},
  booktitle = {ECCV},
  year      = {2014}
}

@inproceedings{guo2017calibration,
  title     = {On Calibration of Modern Neural Networks},
  author    = {Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q.},
  booktitle = {ICML},
  year      = {2017}
}

Inference check

The README snippet above was run verbatim against this Hub repo, on CPU, with torch 2.14 and transformers 5.17.0. It was applied to 6 eval-split questions on 6 different images: 4 predicted "no" and 2 predicted "yes".

The results were compared with the prob_yes values saved by the original training run on a T4 GPU:

Loading SigLIP2 with SiglipVisionModel prints a warning that the text-tower weights (text_model.*, logit_scale, …) are "UNEXPECTED". This is expected, because only the vision tower is used.