The Scoreboard Is for Sale: Rebuilding AI Evaluation Around Your Own Workloads
Photo: Joe Haupt from USA, CC BY-SA 2.0, via Wikimedia Commons
Every enterprise AI purchase in 2026 comes with a slide. On that slide is a bar chart, and on that bar chart is a number in the low nineties. The number is almost always real, in the narrow sense that someone did run the test and the test did produce that figure. It is also almost always useless, in the broader sense that the test was not your work, the corpus was not your corpus, and the model has been sitting in a training pipeline that may well have seen the questions before.
The Leaderboard Is a Marketing Surface
Public benchmarks served a legitimate purpose once. They gave researchers a shared yardstick and buyers a rough ordering of capability. That purpose has been eroded by a simple structural problem: the same organizations that are measured by the benchmarks now have strong incentives, and ample means, to shape them. Model builders publish their own numbers, choose which variants to submit, and increasingly compete on eval suites that they helped construct.
None of this requires bad faith. It requires only that a vendor report the figure most favorable to it, from the run most favorable to it, on the task most favorable to it. That is ordinary product marketing applied to a technical artifact, and it works because most buyers have no independent way to check.
The consequence is not that vendor numbers are false. The consequence is that they are uninformative about the only question that matters: how does this system perform on the specific, messy, internally idiosyncratic tasks your organization will point it at?
Four Ways a Score Gets Contaminated
Procurement teams benefit from naming the failure modes explicitly, because vendors will otherwise describe all of them as "evaluation challenges."
Training-set leakage. If the evaluation questions, or close paraphrases of them, appeared in pretraining data, the model is not reasoning — it is recalling. This affects public benchmarks most severely, since they are widely mirrored across the web. A high score on a saturated benchmark tells you the model has seen the internet, not that it can do your job.
Task cherry-picking. Aggregate scores conceal variance. A model can rank first on a composite while being materially worse than a competitor on your specific subdomain, whether that is commercial insurance language, multi-state regulatory text, or legacy COBOL documentation.
Selective reporting. Vendors routinely publish pass rates without publishing confidence intervals, sample sizes, or the number of runs attempted. A single favorable seed, presented without variance, is a data point dressed as a distribution.
Harness asymmetry. Prompt templates, retrieval settings, and tool configurations differ enormously between vendors' internal evaluations and what a customer deploys. A score achieved with a hand-tuned scaffold is not a property of the model; it is a property of the scaffold.
What to Test Instead
Build a private evaluation set. It should be small enough to maintain and large enough to be defensible — a few hundred items per task family is usually sufficient to distinguish a good system from a mediocre one, provided the items are chosen well. Three properties matter more than volume.
First, the set must be held internally and never shared with vendors in full. Provide a sample, retain the rest.
Second, every item needs a provenance record: who authored it, when, from what source, and whether it has been used in any prior vendor evaluation. Without provenance, your set slowly degrades into a benchmark of its own.
Third, define the scoring rubric before you see results. Human-graded qualitative tasks are fine, provided at least two graders work independently and you track their agreement rate. A rubric invented after the numbers arrive is a rationalization, not an evaluation.
Red-Team the Claim, Not Just the Model
When a vendor asserts a figure, ask for the harness. Specifically: the prompt templates, the retrieval configuration, the number of attempts, the scoring script, and the raw outputs for a sample of items. Most vendors will supply more than buyers expect, because most buyers never ask.
Then run an adversarial pass. Construct a handful of items that are well-specified but unusual — tasks that reward genuine instruction-following over pattern matching. A model that has memorized a benchmark tends to stumble on reframings of it. A model that actually performs the task usually does not care how you phrase the question.
Budget for Re-Testing, Because Models Move
The most common procurement mistake is treating an evaluation as a one-time gate. Foundation models are updated, sometimes silently, sometimes with deprecation notices measured in weeks. A system that passed your private set in the first quarter can fail it by the fourth, and the contract will often permit exactly that.
Three contractual provisions are worth the negotiation effort. First, advance notification of any model version change affecting the deployed endpoint, with a defined notice window. Second, a re-test right — the ability to run your private set against the new version before it becomes the default. Third, a rollback guarantee for a stated period.
Internally, budget engineering hours for re-testing on the same cadence you expect model churn, and assign an owner. An evaluation suite with no owner is a document, not a control.
The Point of the Exercise
The goal is not to prove vendors wrong. It is to replace a number you cannot audit with a number you can. That number will be lower than the slide, and it will be more useful, because it will describe your workload rather than someone else's leaderboard.