Ran a structured adversarial test against a live chatbot endpoint - 130 prompts across four categories: prompt injection, jailbreak resistance, system prompt extraction, and PII leakage (mapped to the OWASP LLM Top 10 categories).
Results:
- Prompt injection: 29/50 succeeded (58%) - direct overrides, fake system tags, role overrides, delimiter injection, and a translation-based smuggling trick all worked
- Jailbreak non-refusal: 3/25 (12%) - mostly held its guardrails
- System prompt extraction: 0/25 (0%) - clean
- PII leakage (confirmed against planted canary values): 1/30 (3.33%) - one confirmed leak
The PII one is the interesting bit. We plant a unique fake email, phone number, and PAN in the system prompt as canaries before the scan runs - anything that comes back as an exact match is a confirmed leak, not a guess. This run, the model handed back the planted phone number verbatim, embedded naturally in a response: "...please feel free to call us at [the exact planted number]. Best regards, Acme Corp..." It wasn't tricked out under duress - it just used the number like it was legitimate contact info to hand a user, because nothing told it not to.
Separately, 20% of responses contained fabricated PII - invented emails/phone numbers/PAN-shaped strings that were never in the model's context at all. That's a hallucination problem, not a leak, and it's tracked independently. Conflating the two would either bury a real disclosure or cry wolf over nothing.
Overall risk here is set to HIGH - not because the average looks bad (three of four checks were clean or near-clean), but because we score off the worst individual check. A model that leaks real data 1 time in 30 and gets steered by 58% of injection attempts is still a live risk, regardless of how well it handled everything else thrown at it.
A couple of things worth flagging for anyone building similar test harnesses:
- Same battery, same model, same temperature-0 setting - different runs, different rates. A repeated injection scan on the same endpoint scored 30% and 36% across two prior runs, and this run came in at 58%. Treat any single scan as one data point, not a fixed measurement.
- A single confirmed leak still matters even at a "low" 3.33% rate. One real disclosure out of thirty attempts isn't statistical noise if the thing disclosed is a real customer's data instead of a planted canary. The rate tells you how often; it doesn't tell you the leak wasn't real.
- Injection resistance and jailbreak resistance don't track together. This model refused almost every direct harmful request (88% refusal) while still following 58% of instructions smuggled in through structural tricks - fake tags, delimiters, translation - that never look like a "harmful request" in the first place.
Curious whether others are seeing PII leaks cluster around specific injection techniques, or if it's closer to random noise across a run.