AI is a fast first drafter, not a final authority.
A property agent asks an AI assistant to summarise the rules on deposits for residential leases and to draft a short explainer for new landlords. The result is clear, confident and nicely formatted. It cites a specific section of an Act and gives a precise number of days within which deposits must be refunded.
She almost sends it to fifty clients. Then, out of habit, she checks the Act itself. The section number is wrong, and one of the time limits doesn’t match the legislation at all. The AI hadn’t flagged any uncertainty. It had simply produced something that sounded right.
This is one of the most important things to understand about today’s AI tools. They’re genuinely useful for drafting, summarising and brainstorming. But they can generate false information, invented sources and wrong figures with complete confidence. Researchers call this “hallucination”, and for businesses, it’s the main reason AI output always needs a human check.
AI is a fast first drafter, not a final authority.
How often AI gets it wrong
The most widely reported warning came from a US court case in 2023, Mata v. Avianca, in which lawyers submitted a legal brief that cited several court decisions which did not exist. They had been generated by ChatGPT. The judge fined the lawyers and their firm, and the case became a cautionary tale for professionals worldwide.
Researchers at Stanford University then studied the problem more systematically. In a 2024 study by Matthew Dahl, Varun Magesh, Mirac Suzgun and Daniel Ho, general-purpose AI chatbots answering questions about US court cases produced incorrect answers, or “hallucinations”, in a large share of cases, with rates ranging from roughly 58% to 88% depending on the model and question. A follow-up study found that even specialist legal AI tools, built specifically for lawyers and using legal databases, still produced incorrect or misleading information in a notable share of queries, roughly one in six or more.
AI models have improved since those studies, and error rates vary greatly by task. They tend to be more reliable at rewriting or summarising text you provide than at recalling specific facts, figures, citations or recent events. But no current AI tool is error-free, and the errors are often presented with exactly the same confidence as correct answers.
That combination, fluent writing with occasional invented facts, is what makes hallucinations risky. They’re easy to miss unless you know the subject or check the source.
Errors are presented with exactly the same confidence as correct answers.
Safeguards against confident errors
Small businesses don’t need to avoid AI. They need a few simple habits that catch mistakes before they reach customers.
Use AI for the right jobs. It’s generally strongest at drafting, rewriting, summarising material you give it, brainstorming options and structuring your thinking. It’s weakest at supplying facts, figures, legal or medical specifics, prices, dates and citations from memory.
Give it your sources. Instead of asking an AI to recall a regulation, paste in the relevant text and ask it to summarise or explain. This reduces, though doesn’t eliminate, invented details.
Check every fact that matters. Any number, quote, law, price, date or claim going to a customer should be verified against an original source by someone who understands the subject.
Ask for its sources, then check them. Some AI tools can show where information came from. Click through and confirm the source says what the AI claims.
Be extra careful in high-stakes areas. Legal, financial, medical, safety and contractual information deserves professional review, not just a quick glance.
Keep a person accountable. Whatever tool produced the first draft, a named person in the business should take responsibility for the final version.
INFOGRAPHIC GOES HERE
Where AI is stronger and weaker. Two columns. Generally stronger: drafting, rewriting your text, summarising documents you provide, brainstorming, structuring.
AI drafts → People check → People decide.
A checking habit
Agree a simple rule in your business: AI drafts, people decide. Before anything AI-assisted is sent, published or relied on, someone reads it critically and checks the facts that matter. For routine internal notes, a quick read may be enough. For anything going to clients or the public, or touching on money, law, health or safety, verify against original sources.
It also helps to share near-misses within the team. When someone spots an AI error, talk about it. Those stories build healthy scepticism faster than any policy.
A few questions can help:
- Which tasks do we use AI for, and are they the ones it does best?
- Do we check facts, figures and citations before AI-assisted work goes out?
- Who is accountable for the final version of AI-drafted content?
- Do we give the AI our own source material rather than relying on its memory?
Which brings us back to the property agent and her almost-sent explainer. AI saved her an hour of drafting, and that was genuinely valuable. Her habit of checking the actual legislation saved her from sending fifty clients incorrect advice. That’s the right partnership: let AI speed up the writing, and keep a knowledgeable person firmly responsible for the truth.
Let AI speed up the writing, and keep a knowledgeable person responsible for the truth.
One Like Research & Further Reading
- Mata v. Avianca, Inc., US District Court for the Southern District of New York, 2023 (sanctions over fabricated citations generated by ChatGPT).
- Matthew Dahl, Varun Magesh, Mirac Suzgun and Daniel E. Ho, “Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models”, Journal of Legal Analysis, 2024; and Varun Magesh et al., “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools”, Stanford RegLab/HAI, 2024.