How many repeats a prompt reliability test needs

You must design reproducibility tests for model outputs statistically before you run them, fixing the sample size, effect threshold and reliability measure in advance. A fixed number of repeats—such as five—is not a universal standard but the result of a specific case with defined parameters. Consistency alone does not prove correctness, and you must test validity against an independent reference.

The prompt is an analysis of 14 October 2026 of sample size and reproducibility in prompt testing, which argues that the required number of repeats depends on the smallest relevant effect size, the number of test items, the confidence level and the chosen measure. A financial-classification study using 90 per cent confidence and a 10 per cent margin of error arrived at 1,317 test items and five repeats per model under the most conservative measure. In our assessment, this means you cannot treat five repeats as a default; you must calculate the sample size your own deployment requires before testing begins, and you must record that calculation as part of your control framework.

Why does the number of repeats depend on what you are trying to detect?

The smaller the variation you want to be able to demonstrate reliably, the more repeats you need. A methodological analysis using Monte Carlo power simulations shows that five iterations provide sufficient discriminating power only for large instability; medium effects require at least fifteen iterations, and small effects remain hard to demonstrate below forty iterations. The number required therefore depends on the chosen metric, the analysis design and the smallest difference you still consider relevant to your use case.

What must you fix before the test begins?

1. Define the smallest effect size you need to detect — the margin of variation below which you will treat outputs as acceptably stable for your deployment.
2. Specify the number of test items — the sample size drawn from the population of documents, questions or scenarios on which you will deploy the model.
3. Choose your reliability measure and confidence level — decide whether you will use agreement statistics, consistency metrics or another measure, and set your confidence threshold in advance.
4. Record the model version, prompt and conditions — document which model, prompt text, temperature setting and provider configuration the test applies to.
5. Plan the repeat structure — treat repeats as nested measurements within a model-item combination rather than as independent observations, to avoid artificially inflating precision.

What is the difference between reproducibility and correctness?

Reliability and validity measure two different things. A model can give exactly the same label on repetition and yet systematically predict the wrong outcome. High consistency therefore says nothing about factual correctness. Financial-classification research makes this distinction explicit: models gave consistently the same labels without predicting the external outcome well. In a grant-review study, prompt conditions shifted the scores, while the correspondence between model-generated scores and human reviewers remained at or below chance level. A stable answer that is consistently wrong remains wrong. You must report reproducibility and validity separately and test the output against an independent reference.

When does an earlier test no longer apply?

A one-off reproducibility test within a single fixed prompt-model combination shows only the properties of that configuration; it does not automatically say anything about stability after later changes. As soon as the prompt, model, provider, temperature or a connected tool changes, the earlier test does not automatically count as evidence for the new configuration. Research in Political Analysis reports that the same prompt can yield different variances and relationships over time. You cannot rely on a fixed snapshot. You must include periodic re-evaluation as a recurring control in the management process and record in the deployment policy that a model version change triggers a new measurement round before output is used again.

What should your control framework record?

Because reconstructing the basis of a conclusion is harder when the design was not recorded, fix the statistical plan—including the measure, margin of error and power target—before the test. A protocol that fixes the design in advance and treats every relevant change as grounds for a new measurement round combines a predefined sample design with explicit recording of the configuration and an independent validity check. Anyone who neglects this reports a figure whose validity may remain limited to the tested configuration and which cannot simply be extrapolated to a changed configuration. In our assessment, this creates a significant risk that regulators and internal auditors will be unable to verify whether the conclusion applies beyond the tested configuration.

No tool can substitute for your own judgment about what effect size matters in your deployment or what independent reference proves correctness. Spreadsheets and statistical software can carry the calculation; the choice of threshold, the decision on what constitutes an acceptable margin, and the selection of your validity check remain yours alone.