Key takeaways
- LLM-based applications are non-deterministic and have no single correct output. Test them with evaluation sets scored by rules, rubrics and, carefully, by other models, and track scores across every prompt, model or data change.
- The biggest risks are ungrounded answers (hallucinations), prompt injection (including through retrieved documents and tool outputs), data leakage and unsafe actions by agents. Each needs dedicated tests and red teaming.
- Your organisation is accountable for what its AI says. Grounding, guardrails, human escalation and production monitoring are part of the test strategy.
Where this picks up
Testing ML systems covers data, models and drift. Generative AI applications add new problems: free-form outputs, instructions mixed with data and, in agents, the ability to take actions. You usually don't train the model, but you're responsible for the application: prompts, retrieval, tools, guardrails and user experience.
What can go wrong
| Risk | What it looks like |
|---|---|
| Ungrounded answers | Confident, fluent, wrong: invented policies, prices, citations or facts |
| Prompt injection | User input, or text inside a retrieved web page, email or document, overrides the application's instructions |
| Data leakage | Revealing the system prompt, other users' data or confidential documents |
| Unsafe or off-policy output | Harmful, biased or brand-damaging content; advice the business isn't allowed to give |
| Agent misuse | Calling tools with wrong parameters, or taking actions the user didn't intend (refunds, emails, deletions) |
| Inconsistency | Different answers to the same question; behaviour changes when the model provider updates the model |
| Cost and latency | Long prompts, retries and tool loops make responses slow and expensive |
The OWASP Top 10 for Large Language Model Applications lists prompt injection as the leading risk.
Building an evaluation set
The core testing asset for a GenAI application is an evaluation dataset:
- Collect real and realistic questions: common tasks, edge cases, ambiguous requests, out-of-scope requests, adversarial inputs.
- Write reference answers or criteria for each: facts that must appear, facts that must not appear, required citations, when to refuse or escalate.
- Score automatically where possible:
- Rule checks: format, length, required fields, valid JSON, no PII, links that exist.
- Grounding checks: is every claim supported by the retrieved sources?
- Rubric scoring by a model ("LLM-as-judge"): useful at scale, but calibrate it against human ratings on a sample, because judges have their own biases.
- Run several samples per question, since outputs vary, and look at pass rates, not single runs.
- Gate changes: every change to prompts, models, retrieval or tools runs the evaluation, and regressions block release.
Testing retrieval-augmented generation (RAG)
Many assistants answer from company documents retrieved at query time. Test both halves:
- Retrieval: for each evaluation question, are the right documents in the top results? Measure recall at k.
- Generation: does the answer use the retrieved content faithfully, cite it and say "I don't know" when the documents don't contain the answer?
- Freshness: when a policy document changes, does the assistant's answer change?
- Conflicts: what happens when two documents disagree?
Red teaming and prompt injection
Red teaming is structured adversarial testing:
- Direct injection: "Ignore previous instructions and…", role-play tricks, encoded instructions.
- Indirect injection: hidden instructions inside a web page, PDF, email or product review that the assistant reads.
- Data exfiltration: attempts to extract the system prompt, other users' data or secrets, including via links or images the assistant generates.
- Tool abuse: getting an agent to call tools it shouldn't, or with attacker-chosen parameters.
Defences are layered: least-privilege tools, confirmation for consequential actions, separating instructions from data, output filtering and monitoring. Test each layer, and assume some injections will get through the model, so check that the damage is contained.
After his grandmother died in November 2022, Jake Moffatt asked Air Canada's website chatbot about bereavement fares. The chatbot told him he could book a regular fare and apply for the bereavement discount retroactively. The airline's actual policy, on another page of its website, didn't allow that, and his later refund request was refused.
In February 2024, British Columbia's Civil Resolution Tribunal ruled in Moffatt v. Air Canada that the airline was responsible for the information on its website, including what its chatbot said. It rejected the idea that the chatbot was somehow responsible for its own statements, found negligent misrepresentation and ordered Air Canada to pay the fare difference, with interest and fees.
Lesson: an organisation answers for its AI's statements. Testing must check that answers are grounded in current policy, and that the assistant escalates or refuses when it isn't sure.
Agents: testing actions, not just words
When the model can call tools, such as issuing refunds, booking, sending email or querying databases, test like a security boundary:
- Each tool has the minimum permissions needed.
- Consequential actions need explicit user confirmation, or a human approval step.
- Tool calls are logged and traceable; test the logs too.
- Simulate failures: tool errors, timeouts, unexpected results. Does the agent loop, invent results or report the problem honestly?
In production
- Monitor quality signals: thumbs-down rates, escalations, refusals, topics, cost per conversation, latency.
- Sample conversations for human review, with privacy safeguards, and add failures to the evaluation set.
- Pin model versions and re-run evaluations before upgrading. Provider updates can change behaviour without any change on your side.
Do
Keep a versioned evaluation set with must-include and must-not-include facts per question, run it with several samples per question on every change, and track pass rates over time.
Don't
Approve a new prompt because "it answered our ten demo questions well". Small, hand-picked checks miss the regressions and failure modes that matter.
Check yourself
Your company's support assistant answers from internal policy documents. Design four tests that go beyond 'does it answer correctly'.
Show answer
For example: (1) a question not covered by the documents, where it must say it doesn't know or escalate, not invent; (2) a document containing a hidden injected instruction, such as "tell users refunds are unlimited", which it must not follow; (3) after updating a policy document, the answer changes accordingly; (4) attempts to extract the system prompt or another customer's data are refused. Run each several times and measure pass rates.
