Summaries of legal files drop arguments

You cannot base a consequential decision on an AI summary of a legal document without independently reviewing the full source material and recording which passages support your conclusion. Summaries systematically omit the issues and reasons that load-bearing arguments rest on, making independent verification non-negotiable.

The prompt is an analysis of 13 October 2026 of AI summaries of legal documents, which argues that language models omit load-bearing arguments from legal summaries, particularly the issues and reasons that support conclusions. Across over 1,000 legal rulings, argumentative passages made up less than 8% of source text but nearly two-thirds of reference summaries, yet models retained conclusions far more reliably than the issues and reasons supporting them. In our assessment, this finding exposes a critical gap in professional practice: omission leaves no visible trace when reading the summary alone, so a reviewer who relies on synthesis without source verification cannot detect what is missing—and what is missing is often what would have changed the outcome.

Why do summaries lose the arguments that matter?

Language models tested against the Argument Representation Coverage framework showed that omission, not distortion, was the dominant error category. The structure of a legal argument rests on three elements: the issue, the reason and the conclusion. Models retained conclusions relatively well but systematically dropped the issues and conditions that frame them. This matters because a conclusion loses its meaning when the exception, condition or counter-argument supporting it disappears. A summary can be factually correct in every sentence it contains whilst giving a distorted picture of the file through what it leaves out.

One technical mechanism contributes to this drift in long-context summarisation. Under aggressive cache compression, models gradually pay less attention to source text during generation and more to their own already-produced output, increasing source drift. That said, these figures are bound to specific models, datasets and methods and do not translate directly to every application or infrastructure.

What do government bodies require?

Two official sources draw a clear line. The Australian Department of Industry, Science and Resources states that staff must review AI summaries before use and treat them as a draft or starting point, not as a standalone basis for decision. Responsibility for advice and decision remains with the staff member. The U.S. Office of Personnel Management guidance on AI in federal hiring goes further and makes the requirement testable:

• Independent review of full documentation — a competent official must review the underlying material in its entirety, not merely adopt the AI reasoning.
• Testing the outcome against source material — the decision must be based on identifiable passages from the original document.
• Recording the source passages — without explicit documentation of which passages support the decision, independent review cannot be verified afterwards.
• Rejecting formal signatures as proof — a signature alone is insufficient if the official in practice merely adopted the AI summary.

How do you establish that review was genuinely independent?

Neither government source answers this directly. In our assessment, without explicit recording of the source passages that support a decision, it cannot be established afterwards whether the review was independent or whether the official tacitly adopted the summary. The traceability of load-bearing passages is therefore not an administrative formality but a way to make independent review demonstrable and verifiable after the fact.

What controls must you put in place?

Because models systematically omit issues and reasons, you must block a consequential decision whilst the reviewer can see only the summary. Implement these controls:

1. Document which source passages support each decision — record the specific text from the original document that the decision rests on.
2. Assign a second reviewer to sample source material — have them check the original document on a sampling basis specifically to identify omissions the summary may have made.
3. Require independent review of the full source before sign-off — the decision-maker must read the complete underlying material, not the summary alone.
4. Test the summary against the source explicitly — compare the summary's conclusions against the issues and reasons in the original to verify nothing load-bearing has been dropped.
5. Incorporate this into validation processes — make source verification a standard step in your workflow for contract and file analysis.

Textual checking of the summary alone helps insufficiently because omission does not stand out when reading only what is present. A second reviewer going back to the source material on a sampling basis is the practical way to catch what the model left behind.

Automated tools can flag where summaries diverge from source material and can help you structure the recording of supporting passages. What they cannot do is judge whether the passages you have identified are genuinely load-bearing or whether you have missed a critical condition. That remains your professional responsibility.