Study shows LLMs leak personal data in notes

You must now determine which systems feed raw sensitive text directly to a language model, and move those flows behind a retrieval and sanitisation layer before the model receives any context.

The prompt is an analysis of 19 October 2026 of privacy-preserving retrieval and sanitisation as distinct processing layers, which argues that language models can leak irrelevant personal data despite privacy instructions embedded in prompts. A peer-reviewed study tested language models drafting clinical notes and found they included information about third parties that did not belong in the output, even when they appeared to recognise privacy rules correctly. In our assessment, this finding means that any organisation giving a complete record as context to an external model now carries a real leak risk, and the separation of retrieval, sanitisation and generation into distinct processing layers has moved from optional to necessary.

The core problem is architectural. A model that correctly paraphrases an instruction such as "do not mention information about third parties" does not reliably act on it. In a generative model, recognising a rule and complying with it are not the same process. Because the prompt asks the model to follow a rule while the raw sensitive source is already in its context, privacy instructions alone do not reliably prevent reproduction. An instruction changes the probability of leakage, not the possibility.

What does the evidence show about privacy instructions in prompts?

The peer-reviewed study tested language models on synthetically enriched transcripts and found that models tested could include irrelevant personal information about third parties in clinical notes, even when given privacy instructions. The researchers also found that separating generation and privacy editing, combined with structured output and deterministic post-processing, produced the greatest reductions among the tested approaches. However, none of the measures examined eliminated leakage entirely. Because the test used synthetic data, the exact percentages do not map one-to-one onto production environments, but the direction of the evidence is relevant: you cannot assume that an instruction in the prompt keeps sensitive information within bounds.

A separate analysis of privacy-preserving retrieval-augmented generation describes that separation concretely: privacy extraction, semantic analysis and reconstruction take place separately, after which the generation component receives only a sanitised context. The original text is used solely for retrieval indexing. The authors report less exposure under the grey-box attacks examined, but acknowledge residual leakage under adaptive and multi-query attacks and give no absolute guarantee.

How should you structure the flow of sensitive data through a model?

The protection should sit before the model (what goes in) and after the model (what comes out), rather than only in the model itself. The core is that the raw sensitive source stays behind a retrieval and privacy layer, while the generation model receives only a minimal, structured and sanitised payload. Four layers, each with its own task, make that controllable:

1. Retrieve only the facts the task needs — do not pass the complete record to the model.
2. Sanitise the context before it reaches the model — remove names, identifiers and information not required for the generation task.
3. Generate output from the sanitised payload — the model works only with what it has been given.
4. Verify and filter the output — apply deterministic post-processing and access controls outside the model.

What are the failure modes you must now guard against?

Because vector stores, evaluation datasets and model output can create additional copies of sensitive information, the data flows you must account for extend beyond the prompt itself:

• Leakage through model output — sensitive information reproduced or inferred from the model's generated text despite privacy instructions.
• Exposure in retrieval context — raw sensitive data held in the retrieval layer and vulnerable to access or logging.
• Logging of prompts and intermediate states — complete records retained in system logs, audit trails or model evaluation datasets.
• Re-identification through derived data — sensitive information reconstructed from vector embeddings, model weights or evaluation outputs.
• Residual leakage under adaptive attack — information disclosed when an attacker queries the system repeatedly or with knowledge of its architecture.

What must your contracts and documentation now cover?

Because the evidence shows that privacy instructions reduce but do not prevent leakage, and because residual leakage persists under adaptive attacks, you should now ensure that:

1. Document the data route per processing step — record which systems feed raw sensitive text to the model and which have moved that flow behind a retrieval and sanitisation layer.
2. Require logging minimisation in supplier contracts — specify retention periods for prompt logs and the right to independent leak tests.
3. Account for derived data flows in privacy documentation — ensure your privacy and compliance review covers vector stores, evaluation datasets and model output as sources of sensitive information copies.
4. Record approval before sensitive output enters a record — because none of the measures examined offered complete protection, require documented human approval of any AI-drafted note or summary before it enters a permanent record.

Tooling can enforce the separation of retrieval, sanitisation and generation, and it can log which data moved through which layer. What tooling cannot do is determine which facts your task actually needs, or judge whether the output is fit to enter a record. That remains your own professional assessment.