Anatomy of an LLM data leak
Apr 11, 2026 · 9 min read
How a customer support agent quietly egressed PII across three subsystems - and the four enforcement points that would have stopped it.
We recently walked a Fortune 500 retailer through an internal incident. A customer support assistant had been retrieving order history through an internal API, formatting it, and including raw email and phone fields in responses to agents.
No single component was misbehaving. The retrieval API was working as designed. The model was working as designed. The chat surface was working as designed. The system as a whole had no policy boundary, so nothing said no.
There were four points where enforcement would have caught this: at retrieval (column-level masking), at the model boundary (output inspection for PII patterns), at the tool layer (purpose-bound access), and at egress (final redaction). Defense in depth - but only if the depth exists.
The lesson is not 'add a filter to your prompt.' The lesson is that AI systems compose policy gaps the way distributed systems compose failures.
Further reading
