Key takeaways
- In ML systems, data is part of the code. Test the data (quality, leakage, representativeness), the model (performance on relevant slices, robustness, fairness) and the system around it (pipelines, fallbacks, monitoring).
- Aggregate accuracy hides problems. Evaluate on slices, such as subgroups, rare cases and new conditions, and use behavioural tests: minimum functionality, invariance and directional expectations.
- Models degrade as the world changes. Monitoring for data and concept drift in production is part of testing, not an operations afterthought.
Where this picks up
ML systems are the extreme case of the oracle problem: for most inputs, nobody can state the single correct output in advance. Metamorphic testing is one of the key techniques here, combined with statistical evaluation and data testing.
What's different about ML
| Classic software | ML system |
|---|---|
| Behaviour written in code | Behaviour learned from data |
| Bugs are in code | Bugs can be in data, labels, features, training process or code |
| Deterministic output for given input | Often probabilistic; retraining can change behaviour |
| Correct / incorrect per test | Quality is statistical: error rates, distributions |
| Stable until code changes | Can degrade when the world changes |
Testing the data
- Schema and validity: types, ranges, missing values, impossible combinations, like age 3 with a driving licence.
- Label quality: sample labels for review; measure agreement between labellers.
- Leakage: features that wouldn't be available at prediction time, or test examples that also appear in training data. Leakage produces excellent offline scores and poor real-world results.
- Representativeness: does the training data cover the populations and conditions the model will meet? Who is under-represented?
- Distribution checks between training data and new data, to catch shifts early.
Testing the model
Evaluation done properly
- Hold out a test set that's never used for training or tuning, and refresh it.
- Choose metrics that match the business cost: precision vs recall trade-offs matter more than accuracy for imbalanced problems like fraud.
- Evaluate on slices: per country, device, language, customer segment, rare classes. A model with 95% overall accuracy can be useless for a small but important group.
- Check calibration: when the model says 80% confident, is it right about 80% of the time?
Behavioural testing
Ribeiro and colleagues' CheckList approach (2020) adapts software testing ideas to ML, especially language models:
| Test type | Idea | Example (sentiment model) |
|---|---|---|
| Minimum functionality | Simple cases it must get right | "I love this product" → positive |
| Invariance | Changes that shouldn't affect the output | Changing a name or city in a review doesn't change sentiment |
| Directional expectation | Changes that should move the output in a known direction | Adding "but delivery was terrible" shouldn't make it more positive |
These are metamorphic tests in practice, and can be generated at scale.
Robustness and fairness
- Robustness: small, realistic perturbations, such as typos, lighting, noise or rephrasing, shouldn't flip decisions.
- Fairness: compare error rates and outcomes across protected groups, and test counterfactuals: change only a sensitive attribute, or a proxy for it, and check whether the decision changes.
In 2018, Reuters reported that Amazon had built an experimental machine-learning tool to rate job applicants' CVs. It was trained on CVs submitted to the company over a ten-year period, most of which came from men, reflecting the tech industry.
According to the report, the model learned to penalise CVs that included the word "women's", as in "women's chess club captain", and downgraded graduates of two all-women's colleges. Amazon edited the system to be neutral to those terms, but, as Reuters reported, there was no guarantee it wouldn't find other ways to discriminate. The project was eventually abandoned. Amazon said the tool was never used by its recruiters to evaluate candidates.
Lesson: the model did exactly what it was trained to do: reproduce patterns in historical data. Testing for fairness, with slice-based evaluation and counterfactual tests, has to be designed in, because aggregate accuracy will never reveal it.
Testing the system around the model
Most ML failures in production come from the surrounding system, not the model maths. A Google rubric, "The ML Test Score" (Breck et al., 2017), lists tests for data, model development, infrastructure and monitoring. Examples:
- Training–serving skew: features computed differently in training and in production.
- Pipeline tests: the training pipeline runs end to end on a small sample; a new model can't be deployed without passing evaluation gates.
- Fallbacks: what happens if the model service is slow or down, or returns low-confidence results?
- Reproducibility: can you rebuild a model from versioned data, code and configuration?
- Shadow and canary deployment: run a new model alongside the old one on live traffic and compare before switching.
Monitoring: testing never stops
- Data drift: the inputs change, for example new products, a new user population or a seasonal shift.
- Concept drift: the relationship between inputs and the right answer changes, for example fraud patterns adapting to the model.
- Outcome monitoring: when true labels arrive later (chargebacks, returns), compare them with predictions per slice.
Set thresholds and alerts, and treat drift as a trigger for re-evaluation and retraining.
Do
Report model quality per slice and per business cost: "Fraud recall 92% overall, but 61% for new customers in Romania, where losses are highest."
Don't
Approve a model because one aggregate metric improved. Averages hide the groups and cases where the model fails.
Check yourself
A loan-approval model shows 94% accuracy on the test set. What three additional tests would you insist on before release?
Show answer
For example: slice evaluation of error rates by relevant groups (age band, region, income band, new vs existing customers); counterfactual fairness tests, changing only a sensitive attribute or proxy and checking the decision doesn't change; robustness and invariance tests, where small irrelevant changes don't flip decisions. Also check the test set for leakage and representativeness, and define drift monitoring for production.
