Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods

Abstract

Large Language Models (LLMs) are trained on web-scale corpora, increasing the risk that benchmark test data appears in training sets and inflates reported performance. We present a systematic literature review of 55 studies on LLM benchmark contamination through late 2025. Our contributions are: (1) a four-tier contamination taxonomy (Exact, Syntactic, Semantic, Task-Level; T1–T4); (2) a comparative analysis of five detection families (string-matching, likelihood-based, membership inference, LLM-prompted detection, and benchmark auditing), including access assumptions and failure modes; (3) a synthesis of contamination evidence on MMLU, GSM8K, HumanEval, and HellaSwag by measurement construct; (4) a comparative evaluation of mitigation strategies across lifecycle points, access assumptions, and evidence maturity; and (5) a Contamination Transparency Card (CTC) framework for future releases. Across studies, no detection method is consistently reliable across contamination tiers, model-access settings, and training stages. We identify instruction tuning as a persistent blind spot, note that RL/post-training contamination auditing is only beginning to mature, and report inflation estimates spanning roughly 6%–40% under benchmark- and setting-dependent assumptions.

Four-Tier Contamination Taxonomy

Mechanism-first tiers map overlap type to expected detectability and mitigation leverage, unifying fragmented prior terminology.

Contamination Detection Families

Five detection families along the white-box to black-box access spectrum. Probing methods need corpus or log-probability access; prompting methods use outputs alone.

What Benchmarks Show

Evidence is stratified by benchmark and measurement family. Hit rates, likelihood scores, and benchmark-level deltas measure different constructs and are not collapsed into one metric.

Mitigation Landscape

Static, inference-time, and dynamic strategies trade coverage, standardization, and cost. No single fix covers all tiers.

Contamination Transparency Card

A minimal five-dimension disclosure framework modelled on Model Cards and Datasheets. Absence of contamination evidence is not evidence of absence.

Takeaways

• Substantial contamination signals recur across major benchmarks, with reported inflation of roughly 6%–40% under setting-dependent assumptions.
• No detection family is reliable across all tiers, access settings, and training stages.
• Instruction fine-tuning contamination remains a critical blind spot for current detectors.
• RL / post-training contamination auditing is only beginning to mature.
• Adopt CTC-style disclosure, strengthen evaluation-of-evaluation infrastructure, and prioritize IFT- and RL-targeted detection.

BibTeX

@inproceedings{nourbakhsh-etal-2026-llm,
    title = "Are {LLM} Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods",
    author = "Nourbakhsh, Erfan  and
      Sirjani, Mohammad Sadegh  and
      Mousavi, Amir  and
      Nguyen, Khoa  and
      Quarles, John  and
      Xie, Mimi  and
      Slavin, Rocky",
    editor = "Mille, Simon  and
      Gehrmann, Sebastian  and
      Schmidtov{\'a}, Patr{\'i}cia  and
      Du{\v{s}}ek, Ond{\v{r}}ej  and
      Fadaee, Marzieh  and
      Lo, Kyle  and
      Santus, Enrico  and
      Stanovsky, Gabriel",
    booktitle = "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics ({GEM})",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.gem-main.50/",
    doi = "10.18653/v1/2026.gem-main.50",
    pages = "518--539",
    ISBN = "979-8-89176-423-1",
    abstract = "Large Language Models (LLMs) are trained on web-scale corpora, increasing the risk that benchmark test data appears in training sets and inflates reported performance. We present a systematic literature review of 55 studies on LLM benchmark contamination through late 2025. Our contributions are: (1) a four-tier contamination taxonomy (Exact, Syntactic, Semantic, Task-Level; T1{--}T4); (2) a comparative analysis of five detection families (string-matching, likelihood-based, membership inference, LLM-prompted detection, and benchmark auditing), including access assumptions and failure modes; (3) a synthesis of contamination evidence on MMLU, GSM8K, HumanEval, and HellaSwag by measurement construct; (4) a comparative evaluation of mitigation strategies across lifecycle points, access assumptions, and evidence maturity; and (5) a Contamination Transparency Card (CTC) framework for future releases. Across studies, no detection method is consistently reliable across contamination tiers, model-access settings, and training stages. We identify instruction tuning as a persistent blind spot, note that RL/post-training contamination auditing is only beginning to mature, and report inflation estimates spanning roughly 6{\%}{--}40{\%} under benchmark- and setting-dependent assumptions."
}