A small HR consultancy starts using an AI tool to screen job applications for its clients. The tool is impressive: it ranks candidates, highlights relevant experience and flags gaps. For the first few weeks, consultants read every CV alongside the AI’s ranking. Gradually, as the tool proves reliable, they start reading only the top five.
Then a client asks why an outstanding local candidate, who later joined a competitor, never made the shortlist. When the consultancy checks, the AI had ranked her low because her experience was described in unusual terms. A consultant reading her CV would probably have spotted it immediately.
It’s a counterintuitive risk. The better an automated tool usually performs, the more people tend to trust it, and the less carefully they check. Researchers have studied this pattern for decades, first in aviation and medicine, and now with AI.
One Like insight: A tool that’s right most of the time can make people stop noticing when it’s wrong.
Automation bias
Long before generative AI, researchers studied how people work with automated systems in cockpits, control rooms and hospitals. In an influential 2010 review in the journal Human Factors, Raja Parasuraman and Dietrich Manzey brought together the evidence on what they called automation complacency and automation bias: the tendency to over-rely on automated aids, to miss problems the system doesn’t flag, and to follow its recommendations even when other information suggests they’re wrong. They found these effects occurred in both novices and experts, and couldn’t simply be trained away.
More recently, Fabrizio Dell’Acqua, then at Harvard Business School, explored the same issue with AI in a hiring experiment. Recruiters evaluated job applications with the help of AI recommendations of differing quality. His working paper, titled “Falling Asleep at the Wheel”, reported a striking result: recruiters given a lower-quality AI tended to put in more of their own effort and judgement, while those given a high-quality AI tended to rely on it more and engage less. In some respects, the recruiters with the better AI ended up making less accurate decisions overall, because they stopped checking.
The hiring study was a single experiment, and its details should be checked against the original paper. But it fits a long and consistent body of research: when automation is usually right, people tend to stop paying attention, which is exactly when its rare mistakes slip through.
Designing work so people stay alert
Because automation bias affects experts as well as beginners, telling people to “be careful” isn’t enough. It works better to design the process so that human judgement stays involved.
Check a random sample. Even when AI is doing well, regularly review a random selection of its outputs in full, such as every tenth application, quote or report, not just the ones it flags.
Look at what the AI rejected. Errors of omission are the easiest to miss. Periodically review the items AI ranked lowest or filtered out, not only the top recommendations.
Make people commit first. For important decisions, ask the person to form their own view before seeing the AI’s recommendation, then compare.
Keep humans responsible for decisions that affect people. Hiring, credit, complaints and anything with legal or ethical consequences should involve genuine human review, not a rubber stamp. In South Africa, POPIA also gives people rights in relation to certain decisions based solely on automated processing of their personal information.
Track accuracy over time. Note when AI gets things wrong and share the examples. Visible mistakes keep healthy scepticism alive.
This connects to the other articles in this tab: Don’t Let AI Do Your Thinking for You and AI Sounds Sure of Itself.
[Infographic placement] The reliability trap. A line chart: as AI reliability rises (x-axis), human checking (y-axis) falls. A small red marker where rare AI errors slip through unchecked. Beneath, five safeguards: Random sampling · Review rejections · Decide first · Human responsibility · Track errors. Caption: Parasuraman and Manzey (2010), Human Factors; Dell’Acqua (2022), “Falling Asleep at the Wheel”.
Staying awake at the wheel
Choose one process where your business relies on an AI or automated tool to filter, rank or decide: screening enquiries, sorting applications, flagging invoices or prioritising leads. Add one simple safeguard this month, such as reviewing a random sample of everything the tool rejected. Note what you find. Even one surprising miss can reset a team’s level of attention.
A few questions can help:
- Which of our decisions now rely mainly on an AI recommendation?
- When did someone last review what the AI rejected or ranked low?
- Do people form their own view before seeing the AI’s answer on important decisions?
- Who is accountable when an automated recommendation turns out to be wrong?
Which brings us back to the HR consultancy and the outstanding candidate it never saw. The AI tool was genuinely helpful most of the time. The problem was that its reliability had quietly switched off the consultants’ own attention. A weekly review of a handful of lower-ranked CVs would have kept their judgement engaged, and might well have put that candidate in front of the client.
Sources
- Raja Parasuraman and Dietrich H. Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration”, Human Factors 52(3), 2010.
- Fabrizio Dell’Acqua, “Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment on HR Recruiters”, working paper, 2022.
- Protection of Personal Information Act 4 of 2013, section on automated decision-making (general reference).
