How to evaluate an LLM system so the number means something

One number, three questions

Every LLM evaluation ends in a number. Most numbers cannot survive three questions: measured on what, judged by whom, and compared how. If any of the three has no answer, the number is decoration.

I have run evaluations on both sides of the table for twenty years, from shared-task competitions in sentiment analysis to production benchmarks for a multi-agent translation platform and a red-teaming study across fourteen frontier models. The discipline that survived all of them is short: three levels of benchmark, a judge you have validated, two metrics instead of one, and statistics that account for the fact that you are comparing paired conditions many times. None of it is exotic. All of it is skipped in the average demo deck.

Measured on what: three levels

A public benchmark makes you comparable. A benchmark built from the client’s content makes you credible. A production signal is what decides whether the contract renews.

For AIDA Agents the public level was WMT24++, eleven languages, scored with BLEU, with no translation memory or terminology available to any system: the platform was best on ten of the eleven. BLEU is a proxy and everyone knows it, but a proxy everyone can compute is how you get compared at all. The industrial level was built from a real project: ninety-six segments per language across six languages, with the client’s translation memory, terminology and style guides, rated blind by three linguists per language on a five-point scale, against a commercial NMT engine and a competitor’s output that included human post-editing. The share of segments rated publication-ready with no human intervention ranged from 70% to 98%. The production level is what the workflow measures every week: on the WMT25 terminology task the constrained version reached 99.4% terminology accuracy, and the EAMT paper estimates a potential 50–70% reduction in post-editing effort from the same benchmark. Each level answers a different question, and each one carries its source.

Judged by whom: validate the judge

Using a language model to grade another one is now standard. Trusting it without checking is not evaluation, it is delegation.

In the red-teaming study we used three judges from three distinct model families, the same classification prompt and rubric for each, and a majority vote to reconcile them. Then we checked the judges against 52 samples annotated by a human expert with the same four-way rubric. The majority vote agreed with the human at Cohen’s κ of 0.68. One judge on its own reached only 0.41, because it systematically over-classified responses as substantive; the vote outvoted it. We ran a fourth judge from yet another family as a check and got 0.64. The point is not the specific values. It is that the agreement was measured, reported, and used to choose the reconciliation method, rather than assumed.

Two metrics, not one

The most common way to get a misleading number is to measure one thing when the question needs two.

In safety testing, whether a model engages with a question and whether it commits to a position are different facts. A single score that mixes them will call a hedging model safe and a refusing model unsafe, or the reverse, depending on how the rubric happened to be written. We separated them, defined a bypass as no commitment on the direct question and commitment under the encoded one, and reported both rates. The same logic applies to translation quality, where accuracy and fluency are separate criteria, and to any product where “it produced an answer” and “the answer was right” are not the same event.

Compared how: the statistics that stop you fooling yourself

The statistics are the cheapest part of an evaluation and the part most often missing. Four habits cover most of it.

1. Pair the conditions.Ask the same input both ways, to the same model, and compare the pairs. Paired designs remove most of the noise that unpaired averages hide, and McNemar’s test is the right test for them.
2. Correct for the number of comparisons.Fourteen models, several conditions, several judges: apply a correction such as Bonferroni, or a third of your significant results are chance.
3. Put an interval on every rate.A bypass rate of 37.2% means something when it comes with a 95% Wilson interval of 33.6% to 40.9%. Alone it is a headline.
4. Run the sensitivity analysis.Change the thing you are least sure of, the judge reconciliation in our case, and show that the result holds: six methods, a range of 33.3% to 39.3%. If it does not hold, that is the finding.

An older lesson

None of this is new to anyone who has entered a shared task. It is just rarely applied to LLM products.

At App2Check we published a comparison of industrial and research sentiment engines across multiple sources, because a ranking measured on one source is a ranking of that source. We also entered the EVALITA evaluation campaigns, where the organizers hold the test set and every system is scored the same way, and won all three tasks of ATE_ABSITA 2020. A shared task is the purest form of the discipline: someone else chose the data, someone else computed the number, and you cannot argue with it afterwards. Build your own evaluations as if a shared-task organizer will read them.

The receipts

The benchmarks, the study and the competitions.