Surgeons followed wrong AI advice: make oversight measurable

You must now treat automation bias as a measurable system risk in any workflow where AI advice influences a consequential decision. Make that advice explicitly contestable, maintain logs of every override, and test your oversight controls with deliberately incorrect suggestions to verify they actually work.

An analysis of 9 November 2026 of how surgeons followed deliberately wrong AI advice in a controlled trial argues that the presence of a human reviewer does not reliably catch errors in AI recommendations. In a controlled experiment, nineteen surgeons followed 60.6% of deliberately incorrect AI suggestions, whilst following correct suggestions in 76.9% of cases. In our assessment, this figure demonstrates that the human step in a decision chain cannot be assumed to work by itself; it must be designed, measured and tested as an active control rather than treated as a formal box to tick.

Why does a human reviewer still follow wrong AI advice?

The surgeons in the study were experienced professionals working under realistic conditions. More than three in five wrong suggestions were followed anyway. This phenomenon, called automation bias, describes the tendency of people to rely on an automated recommendation even when they have the expertise to question it. The mechanism operates under several conditions: time pressure, confidence in the system, and the presentation of advice as a default choice rather than an option to be contested. Formal oversight—a signature, a review button, or an assigned staff member—does not prevent this. Meaningful oversight requires that the reviewer has time, expertise, access to source information, genuine authority to reject the advice, and a workflow that makes rejection the easier path, not the harder one.

What failures does automation bias create?

• Overreliance on system output — professionals follow recommendations without independent verification, especially under time or cognitive pressure.
• Narrow practical authority — reviewers hold formal responsibility but lack the workflow conditions to exercise it, such as time or access to underlying data.
• Confidence drift — exposure to AI advice changes a professional's own confidence in their judgment, making them less likely to contradict the system.
• Organisational incentives against deviation — workload, performance metrics or escalation friction discourage reviewers from rejecting recommendations.
• Procedural compliance without substance — oversight controls become mandatory steps that do not meaningfully alter the decision.

Which concrete controls must you be able to demonstrate?

1. Map where AI advice has become a default — identify every workflow in which AI output is presented as the starting assumption rather than as a contestable option.
2. Make advice explicitly contestable — redesign workflows so that rejecting or modifying AI recommendations requires no more effort than accepting them, and ensure reviewers know they have authority to do so.
3. Maintain and audit override logs — record every instance in which a reviewer accepts, modifies or rejects AI advice, and review these logs periodically to detect patterns of overreliance or systematic deviation.
4. Test oversight with deliberately wrong advice — run scenario-based tests in which you introduce incorrect AI suggestions and measure whether your reviewers catch them; record the outcomes and use them to refine your controls.
5. Distribute responsibility contractually — if you use AI advice supplied by an external provider, specify in your contract which oversight measures the provider and your organisation each take, and which party bears liability for failures.

How should you weight these controls?

The heaviest controls belong on decisions with real consequences. A low-stakes recommendation may need only a log; a decision that affects patient safety, legal liability or significant resource allocation needs scenario testing and measured response times. The trap is creating so much measurement that oversight becomes a ritual that no one reads. An override log that sits unreviewed is simply another formal layer. In our assessment, restraint in the design of controls is as important as their presence.

What can tooling do, and what remains your responsibility?

Workflow systems can enforce that advice is presented as contestable, can log every decision point, and can trigger scenario tests on a schedule. They cannot judge whether a reviewer's reasoning is sound, whether they had genuine access to the information they needed, or whether the organisational culture actually permits them to deviate. That remains your professional judgment and your governance responsibility. The measure of oversight is not the existence of a control; it is whether that control changes behaviour in the direction you intend.