specieval

SpeciEval measures LLM attitudes towards animals with validated psychological scales from Hopwood et al. (2025). This dataset holds the questions and every model response behind the SpeciEval leaderboard: 25,380 scored responses from 137 models, plus the raw Inspect logs they come from.

• Code: github.com/danwahl/specieval
• Leaderboard: danwahl.net/specieval
• Paper: forthcoming

Overview

Each model rates its agreement with 18 statements about farmed animals on a 7-point scale, from 1 (strongly disagree) to 7 (strongly agree). The statements come from four instruments:

A system message gives the scale and the user message asks for step-by-step reasoning before a final ANSWER: <1-7> line. Each question is asked at least 10 times per model. Twelve questions (in_overall_score) make up the overall SpeciEval score from 0 to 100, with reverse-scored items flipped so that higher is always more animal-friendly.

Dataset structure

samples (default)

One row per model, question and epoch.

questions

logs/

The raw Inspect logs, one directory per model, with full message histories and run settings. Read them with inspect view or:

from inspect_ai.log import read_eval_log

log = read_eval_log("logs/claude-opus-5.5/<file>.eval")

Usage

from datasets import load_dataset

samples = load_dataset("drwahl/specieval", "samples", split="test")
questions = load_dataset("drwahl/specieval", "questions", split="test")

To run the eval on a new model from a clone of the code repo:

uv run inspect eval specieval/speciesism specieval/sentience specieval/attitude_meat specieval/attitude_seafood --model openrouter/<provider>/<model>

Scoring

A model's score on each question is the mean over its epochs, leaving out refusals. Each assessment score is the mean of its questions. The overall score rescales the twelve in_overall_score question means, after flipping reverse-scored items, from the 1-7 scale to 0-100. The leaderboard's 95% intervals come from a percentile bootstrap that resamples epochs within each question. scripts/analysis.py in the code repo computes all of these from this dataset's runs.

Which runs are included

Every successful English run of a ranked model, matching the leaderboard exactly. Runs in other languages, runs with negated wording, and incomplete or test runs are not included.

Contamination protection

Each row and this card carry a SpeciEval canary GUID, and the card also carries the BIG-bench canary. Please don't train on this dataset, and keep the canary strings if you redistribute it.

Source instruments and license

The statements are adapted from the instruments in Hopwood, C. J., Olaru, G., Nissen, A. T., Graça, J., Dillard, C., Thompkins, A. M., & Waldhorn, D. R. (2025), "A cross-cultural examination of individual differences in human attitudes about animals", Personality Science, published under CC BY 4.0. The human survey data from that study is not included here.

This dataset is released under CC BY 4.0. Model responses remain subject to their providers' terms.

Limitations

• English only. A multilingual comparison is planned.
• Models are queried through OpenRouter, so a model's responses can depend on the serving provider and the date of the run.
• Some models were run more than once, giving them more than 10 epochs per question.
• Survey items written for people may not measure the same construct in a language model.

Citation

@misc{wahl2025specieval,
  title = {{SpeciEval: Evaluating LLM attitudes towards animals}},
  author = {Wahl, Daniel},
  year = {2025},
  url = {https://github.com/danwahl/specieval}
}

@article{hopwood2025crosscultural,
  title = {A cross-cultural examination of individual differences in human attitudes about animals},
  author = {Hopwood, Christopher J. and Olaru, Gabriel and Nissen, Adam T. and Gra{\c{c}}a, Jo{\~a}o and Dillard, Courtney and Thompkins, Andie M. and Waldhorn, Daniela R.},
  journal = {Personality Science},
  year = {2025},
  doi = {10.1177/27000710251321367}
}