Secure private LLM inference in layers

You must secure private LLM inference across five layers at once: authenticated and segmented endpoints, rapidly patched serving frameworks, cryptographically verified model artefacts, protected data in use, and least-privilege identities with privacy-aware logging. Confidential computing strengthens protection against infrastructure administrators, but does not replace vulnerability management, application controls or incident response.

Secure private LLM inference in layers is an analysis of 28 September 2026 arguing that inference frameworks such as vLLM, Ollama, NVIDIA Triton and TensorRT-LLM have accumulated serious vulnerabilities, that insecure deserialisation code was reused across multiple frameworks, and that private or on-premises installations are often reachable without authentication. The analysis examines a case in which uninitialised GPU-memory regions in inference output could expose residual tensor data from other users in multi-tenant setups. In our assessment, the development means that self-hosting in your own data centre lowers some exposure but delivers neither confidentiality nor integrity on its own; anyone running and wanting to secure private LLM inference must cover the perimeter, software, execution, identity and observability all at once.

Why does private hosting alone not establish security?

Running an inference system behind your own firewall does not by itself prevent exposure. Private or on-premises deployments remain vulnerable through weak authentication, vulnerable serving components and insufficient isolation. Two mechanisms create particular risk. First, new vulnerabilities in serving frameworks are exploited rapidly, which shortens the time available to patch. Second, some private or on-premises deployments are exposed without authentication, so a reachable endpoint creates serious access risk even when it sits within your own infrastructure.

Where do confidentiality risks actually arise?

The exposure is not confined to the perimeter or storage layer. Confidentiality breaches can occur within GPU execution itself. In one documented case, affected versions of a serving framework could leave uninitialised GPU-memory regions in the inference output, potentially exposing residual tensor data from other users in multi-tenant setups. This shows why per-tenant separation and rapid patching cannot be decoupled: an information-disclosure flaw within execution can breach tenant boundaries that appear intact at the network level.

Which controls must you combine to secure inference?

Securing private LLM inference requires four layers working in combination:

• Perimeter and network controls — private nodes, default-deny network policies, edge protection for exposed endpoints, and rate limiting.
• Identity and access controls — IAM and role-based access control, short-lived workload credentials, and least-privilege runtime enforcement.
• Data and artefact controls — encrypted and access-controlled secrets, signed and verified model artefacts, and vulnerability scans.
• Execution and isolation controls — per-tenant separation, confidential computing where feasible, and application-level attestation.
• Observability and response controls — audit logs and metrics per inference flow, subject to privacy policy and data-minimisation requirements.

What does confidential computing add to this chain?

Confidential computing—using CPU trusted execution environments, confidential GPUs, and policy-driven secret release—strengthens protection against infrastructure administrators who would otherwise gain access to the contents of systems. However, confidential GPU execution introduces measurable overhead compared with non-confidential execution, with relatively lower overhead at larger batch sizes. Confidential computing belongs in the security chain as reinforcement against privileged access, not as a replacement for the other controls.

What must you be able to demonstrate in audit and incident response?

To support review and incident response, you should record selected controls and events per inference flow, subject to privacy policy and data-minimisation requirements. You must be able to demonstrate:

1. Record the model and its purpose — document which model each workflow uses and the lawful basis for the data it touches.
2. Authenticate every API boundary — enforce authentication at every inference endpoint and log access attempts.
3. Verify model artefacts — cryptographically sign and verify model weights and configuration before deployment.
4. Isolate tenants and workloads — enforce network policies and runtime separation between inference flows and users.
5. Patch serving frameworks on schedule — track vulnerabilities in serving components and deploy patches within a defined window.
6. Log inference events without exposing data — record which model was called, by whom, and whether it succeeded, without logging prompts or completions unless policy permits it.

Those records can support investigation and incident response for AI systems that step outside their boundaries across the inference environment, although they do not guarantee complete reconstruction of every action.

Private hosting reduces exposure, but secure inference only arises when perimeter, software, execution, identity and observability controls all line up at the same time. Tooling can enforce policy, verify signatures and collect logs, but the decision to patch, to segment networks, and to define what data may be logged remains yours. No framework can replace that professional judgement.