Eval Washing
Eval washing describes a communication and procurement practice in which vendors or internal teams tailor evaluations to produce desired results — through favourable benchmark selection, contaminated test data or unrealistic input distributions.
Eval washing is the practice of showcasing selective or flattering evaluation results to make an AI system look safer or more capable than it is in real use.
Explanation
The mechanism resembles greenwashing: the claim is not false, its framing is. Typical patterns are benchmarks whose tasks appeared in training data, safety tests run only with English prompts, results reported without confidence intervals, or comparisons against outdated model versions. Because robust evaluation is expensive and domain-specific, buyers easily mistake marketing figures for measurements. Countermeasures are in-house evaluation sets built from real customer requests, disclosure of data cut-offs and test distributions, inclusion of failure cases, and contractual metrics re-measured in production.
Marketing Relevance
Anyone buying AI tooling makes budget decisions on such numbers. Small in-house evaluation sets protect against bad investments far better than any vendor slide showing top scores.
Example
A vendor advertises 96 percent accuracy on invoice extraction. The customer's test with 200 real documents including scans and handwriting yields 71 percent — the benchmark consisted solely of digitally generated PDFs.
Common Pitfalls
The costliest mistake is importing vendor benchmarks into business cases unchecked. Without an in-house test set you can prove neither progress nor regression.
Origin & History
As public leaderboards spread from 2023, the incentive shifted to optimising scores rather than capability. After several studies demonstrated benchmark contamination in 2025, the term eval washing established itself in 2026 in procurement guidance and governance discussions as a counterpart to greenwashing and AI washing.
Marketing Use Cases
In-house evaluation set of 100 to 300 real requests per use case.
Procurement questionnaire requiring test distribution, data cut-off and failure cases.
Regular re-measurement of vendor claims in production.
Documented failure-case collection as part of model selection and acceptance.
Frequently Asked Questions
How do you spot eval washing?
By what is missing: if test distribution, data cut-off, language coverage and failure cases are unstated, the number cannot be verified. Comparisons made only against older competitor versions are another clear signal.
How large should an in-house evaluation set be?
For most marketing applications 100 to 300 realistic cases suffice if they mirror the true distribution, including edge cases. Representativeness matters more than size.
Is eval washing the same as AI washing?
No. AI washing means marketing a product as AI-powered when it barely uses AI. Eval washing comes later: the AI exists, but its performance evidence is selectively presented.
Related Services
Go deeper: Agentic AI Hub · Model comparison 2026