US GAO discusses testing challenges in federal AI

You must verify performance claims independently before awarding a contract for any AI system, set measurable thresholds in the agreement, require retesting when the system changes, and document your evaluation method so that you can defend the decision if performance falls short.

An analysis of 11 November 2026 of independent testing and performance verification before AI procurement argues that buyers should not treat supplier claims or model cards as sufficient evidence of fitness for purpose. The analysis draws on federal acquisition frameworks, including practices from the Department of Energy and Department of the Interior, which require measurable performance thresholds, pre-award testing and contractual obligations to retest after system changes. In our assessment, this principle applies equally to private organisations: a one-off supplier demonstration or self-report lacks the baseline, method and context needed to support a procurement decision, and the burden of verification now falls on the buying organisation, not the vendor.

Why supplier claims alone are not enough

A vendor's own statement about system performance is not an independent test. Academic research into public-sector AI procurement shows that self-reports often omit the baseline data, testing methodology and operational context required to assess whether a claim is meaningful. A figure without comparison—accuracy of 92 per cent, for instance—tells you nothing about whether the system will work for your use case, your data or your tolerance for error. The same gap applies to general compliance certifications: a model certified as compliant with a regulatory framework does not settle your own obligations under that framework, nor does it prove the system will perform acceptably in your environment.

What does the contract need to say?

Translate your performance requirements into enforceable terms, not general reliability promises. The contract should specify measurable thresholds for each material performance metric, define the test data and methodology that will be used to verify those thresholds, require the supplier to notify you of any material change to the model or its training data, and give you an audit right with concrete inspection evidence. Include a clause that allows you to retest the system after any change and to refuse deployment or rescind the contract if performance falls below the agreed threshold. If the system processes personal data, the contract must also address compliance with data protection law, including data retention, access controls and breach notification.

Which controls must you be able to demonstrate?

1. Set measurable performance thresholds before award — define what acceptable performance looks like for your use case, expressed in quantifiable terms tied to your operational data and tolerance for error.
2. Conduct independent testing with your own or representative validation data — do not rely on the vendor's test results; use a ring-fenced dataset that reflects your actual operating conditions.
3. Require change notification and reassessment — the contract must obligate the supplier to notify you of any model update, retraining or data change, and to allow you to retest before the new version goes live.
4. Document your evaluation method and findings — keep a record of what you tested, how you tested it, what the results were and why you accepted or rejected the system; this record is your defence if performance later falls short.
5. Assign verification responsibility explicitly — designate an internal team or engage an independent third party to carry out pre-award testing and ongoing monitoring; do not leave it ambiguous.

What happens if you skip this step?

The burden of proof has shifted. If a dispute arises over substandard performance, a supplier will argue that you accepted the system based on their demonstration or certification. You will then need to show that you conducted your own due diligence, set clear thresholds and monitored compliance. If you cannot produce that evidence, you have little recourse. Conversely, if you can demonstrate that you verified performance independently, set measurable thresholds, required retesting after changes and documented your findings, you have a factual basis for holding the supplier accountable.

How do you build this into procurement policy?

Add a pre-award verification phase to your AI procurement process. Before you issue a purchase order, you must complete independent testing with your own validation data and confirm that the system meets your thresholds. Include in your procurement policy a refusal option: if the supplier cannot provide sufficient verifiable evidence, you do not proceed. Assign this function explicitly—do not assume it will happen. For material or higher-risk systems, budget for external evaluation expertise if you lack it internally. Make the testing requirement and the refusal option visible to procurement teams and to the business stakeholders who request the system, so that performance verification becomes a normal part of the decision, not an afterthought.

Automated tools can help you structure testing, log results and track changes to the system after deployment. What they cannot do is judge whether the system is fit for your purpose or whether the risk it carries is acceptable to your organisation. That remains your professional responsibility, informed by the evidence you have gathered and the thresholds you have set.