llm-watermark-eval-generations

LLM Watermark Evaluation — Inputs and Generations

Every input given to the generator and every text it produced during full-scale evaluations of the Thicket watermark against published watermarking schemes. All schemes were run with the same model, prompts, sampling settings and seeds, so each row of one config has an exact counterpart in every other config of the same generator (same prompt_id, same sample). The dataset lets readers inspect what went into the model and what came out, compare schemes on identical inputs (quality, diversity, length) and reuse the texts for robustness or detection experiments.

Each config is one stage (split type) and each of its splits is one scheme (the names in the tables below), so all splits of a config share one schema:

from datasets import load_dataset
ds = load_dataset("RootAccess4Life/llm-watermark-eval-generations", "main")                 # DatasetDict, one split per scheme
one = load_dataset("RootAccess4Life/llm-watermark-eval-generations", "main", split="thicket")

ELI5 questions — Qwen3.5-9B

One split per watermark scheme in each of the configs below; the counts are rows.

Other generator models

The ELI5 protocol on other generators, one <scheme>__<model> split each in the ELI5 configs: the same questions, sampling settings and seeds; every prompt re-rendered from question with the generator's own chat template (Gemma has no system role, so the multilingual system prompt opens the user turn); generators without a thinking mode run multilingual and benchmarks without a reasoning budget (the whole generation is the answer, reasoning is empty).

Each scheme used one fixed key; the keys are released in keys.json and explained in KEYS.md, so the detection results can be reproduced with the project's code.

Splits

Fields

Generation

Model Qwen/Qwen3.5-9B, fp16, PyTorch SDPA attention; batched sampling with fixed per-batch seeds; each watermark applied inside the sampler by its official or reference implementation (TextSeal, Gumbel-max and SynthID-Text from the TextSeal codebase; KGW, Unigram, SWEET and EWD through the Hugging Face transformers watermark logits processor or re-implementations of their papers; Thicket from its authors' code). The unwatermarked reference (none, when present) uses the same sampler without a watermark. The <scheme>__<model> configs use meta-llama/Llama-3.1-8B-Instruct, google/gemma-2-9b-it, mistralai/Mistral-7B-Instruct-v0.3 the same way (fp16), with the prompt handling described above.

Not included

• GPQA: its 30 questions and the corresponding answers are left out because GPQA's authors ask that its examples not be published online (benchmark-leakage prevention). All other benchmarks are included.
• The negatives of the false-positive tests (Wikipedia passages and unwatermarked same-model answers to other ELI5 questions).
• Copy-paste attack documents (watermarked fragments pasted into Wikipedia text).
• attacks and documents of the runs that finished before per-text records were kept (their attacked texts and documents were not stored; see the tables above).
• Evaluation scores: no p-values, detector outputs or metrics.

Sources and licenses

Generated texts: CC BY 4.0. They are outputs of the generator named in model, and their use is also subject to that model's terms: - Qwen/Qwen3.5-9B: subject to the Qwen3.5 model license (see the model card). - meta-llama/Llama-3.1-8B-Instruct: Built with Llama. Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved; outputs are also subject to the Llama 3.1 Acceptable Use Policy. - google/gemma-2-9b-it: Gemma Terms of Use (ai.google.dev/gemma/terms) and the Gemma Prohibited Use Policy. - mistralai/Mistral-7B-Instruct-v0.3: Apache 2.0.

Attacked texts (attacks, c4_attacks): the rephrasings are outputs of meta-llama/Llama-3.2-3B-Instruct (Built with Llama; Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved), the paraphrases of kalpeshk2011/dipper-paraphraser-xxl, the back-translations of facebook/nllb-200-distilled-600M (model license CC BY-NC 4.0).

documents / c4_documents: CC BY-SA 4.0. These splits contain text from English Wikipedia (wikimedia/wikipedia, dump 20231101.en; © Wikipedia contributors, https://en.wikipedia.org), redistributed under CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); the generated passages they embed are marked by true_spans. All other splits: CC BY 4.0.

Inputs keep the licenses of their sources: ELI5 (Reddit questions, as distributed in the Pavithree/eli5 Hugging Face dataset; also the English multilingual questions), the multilingual question pool (shanearora/CaLMQA, CC BY 4.0), GSM8K (MIT), MATH-500 (MIT), AIME (public competition problems), MMLU (MIT), ARC (CC BY-SA 4.0), HellaSwag (MIT), WinoGrande (CC BY), HumanEval (MIT), MBPP (CC BY 4.0).

Citation

Paper under review; citation to be added.