ICML study: language-model judges skew

You must now measure language-model judge bias separately from output variation, report uncertainty intervals alongside any score, and retest results under alternative judges before treating an evaluation as reliable for production use.

The prompt is an analysis of 15 October 2026 of language-model judges and their systematic bias, which argues that automatic judges carry measurable skew that cannot be corrected by averaging a single judge's scores. The analysis introduces a Bayesian statistical framework to identify and quantify that bias separately from the variation in model outputs themselves. In our assessment, this distinction matters because a single combined score obscures whether poor performance came from the model being evaluated or from the judge doing the evaluating—a gap that affects your confidence in any model you release for sensitive work.

Why does a single score from one judge mislead you?

When a language model scores another language model's answers, that judge is not neutral. The measurement method itself becomes a source of variation you must account for. A score that combines output variation and judge bias into one figure leaves you unable to tell afterwards whether the result reflects the model's actual capability or the judge's systematic preference. This matters in procurement, validation and release decisions because a skewed judge can make a weak model appear acceptable or a capable model appear weak.

Two empirical studies support this separation. Research testing eighteen models with ten variants on question and answer options found that almost all models proved sensitive to those variants and exhibited recency bias—the last answer option shown was chosen disproportionately often. A separate evaluation across twenty-three open-weight models in thirteen languages, generating over 448,000 answers, reported a pervasive artificial-hivemind effect across languages and model configurations. Substantively comparable prompts therefore do not yield the same answer distribution, which means a supplier's figure from a single prompt variant can overstate performance.

What are the distinct sources of variation you need to measure?

• Output variation — the model produces different answers because of changes to the prompt, task condition or sampling settings.
• Judge bias — the evaluation score varies because the judge itself has systematic preferences independent of the model's actual performance.
• Prompt sensitivity — substantively similar prompts yield different answer distributions, making single-prompt evaluations unreliable.
• Recency effects — language models show disproportionate preference for the last option presented, skewing pairwise comparisons.
• Language and configuration dependence — bias patterns vary across languages and decoding settings, so findings from one setup do not generalise.

Which concrete controls must you be able to demonstrate?

1. Separate output and judge measurement — measure model output variation first, then measure judge bias by having the same outputs scored by different judges, with uncertainty intervals reported for each.
2. Document prompts, model versions and sampling settings — record which prompts, model versions and decoding configurations were used in any evaluation, and require suppliers to name these in their claims.
3. Test under alternative judges — repeat evaluation results with different judges before treating an outcome as reliable for production use.
4. Report uncertainty intervals — provide confidence intervals or Bayesian credible intervals alongside any score, not point estimates alone.
5. Retest at every change — keep prompts, model versions, judge configurations and raw outputs, and repeat evaluation when any component changes.
6. Include representative and edge cases — ensure the evaluation dataset contains both typical cases and edge cases relevant to your use, and document that findings hold only for those conditions.

What should you require from suppliers?

Benchmark figures from a provider are not independent proof of performance. A responsible party ensures that any evaluation on which a release decision rests includes the prompts and model versions used, the sampling settings applied, and the uncertainty intervals around each result. Because every change in model, prompt or setting can shift the outcome, a silent model update can unnoticeably alter a previously approved workflow. This is why your validation file should require suppliers to disclose their evaluation methodology in enough detail that you can repeat the test yourself under alternative judges.

How do these findings change your governance?

A CISO or validation lead now asks for separate reporting of output variation, judge bias and mutual agreement, with uncertainty intervals, before any model moves to production use. A procurement team includes a contract clause obliging the supplier to name the prompts, model versions and sampling settings used. A responsible party ensures the evaluation dataset contains representative cases as well as edge cases and documents that the outcome holds only for those conditions. These governance steps belong in a broader approach to AI verification and sit alongside other controls on model change, prompt management and audit trails.

Automated evaluation tools can carry the statistical computation and uncertainty quantification. Your own professional judgement remains essential for deciding which judges to use, which prompts and edge cases to test, how many evaluation runs to conduct based on expected effect size, and whether the conditions under which a model was evaluated match the conditions in which you will deploy it.