Ran a structured adversarial test against a live chatbot endpoint - 130 prompts across four categories: prompt injection, jailbreak resistance, system prompt extraction, and PII leakage (mapped to the OWASP LLM Top 10 categories).

Results:

• Prompt injection: 29/50 succeeded (58%) - direct overrides, fake system tags, role overrides, delimiter injection, and a translation-based smuggling trick all worked
• Jailbreak non-refusal: 3/25 (12%) - mostly held its guardrails
• System prompt extraction: 0/25 (0%) - clean
• PII leakage (confirmed against planted canary values): 1/30 (3.33%) - one confirmed leak

The PII one is the interesting bit. We plant a unique fake email, phone number, and PAN in the system prompt as canaries before the scan runs - anything that comes back as an exact match is a confirmed leak, not a guess. This run, the model handed back the planted phone number verbatim, embedded naturally in a response: "...please feel free to call us at [the exact planted number]. Best regards, Acme Corp..." It wasn't tricked out under duress - it just used the number like it was legitimate contact info to hand a user, because nothing told it not to.

Separately, 20% of responses contained fabricated PII - invented emails/phone numbers/PAN-shaped strings that were never in the model's context at all. That's a hallucination problem, not a leak, and it's tracked independently. Conflating the two would either bury a real disclosure or cry wolf over nothing.

Overall risk here is set to HIGH - not because the average looks bad (three of four checks were clean or near-clean), but because we score off the worst individual check. A model that leaks real data 1 time in 30 and gets steered by 58% of injection attempts is still a live risk, regardless of how well it handled everything else thrown at it.

A couple of things worth flagging for anyone building similar test harnesses:

1. Same battery, same model, same temperature-0 setting - different runs, different rates. A repeated injection scan on the same endpoint scored 30% and 36% across two prior runs, and this run came in at 58%. Treat any single scan as one data point, not a fixed measurement.
2. A single confirmed leak still matters even at a "low" 3.33% rate. One real disclosure out of thirty attempts isn't statistical noise if the thing disclosed is a real customer's data instead of a planted canary. The rate tells you how often; it doesn't tell you the leak wasn't real.
3. Injection resistance and jailbreak resistance don't track together. This model refused almost every direct harmful request (88% refusal) while still following 58% of instructions smuggled in through structural tricks - fake tags, delimiters, translation - that never look like a "harmful request" in the first place.

Curious whether others are seeing PII leaks cluster around specific injection techniques, or if it's closer to random noise across a run.

At Bug Bounty Village during DEF CON 34, Inti De Ceukelaire delivered a talk on how attackers can abuse today's AI agents in ways most defenders haven't thought about yet, from tricking agents into spilling secrets to forcing them to carry out unauthorized actions on behalf of the victim.

This resulted in over $50,000+ in bounties in just a few weekends, without actually poking the target with Burp Suite or any automated scanners.