Not long ago, a new website for a small business took weeks. There were meetings, mock-ups, revisions, a developer who went quiet for a fortnight and a launch date that drifted quietly into the next month. Today, a reasonably determined owner with an AI tool can produce a working site, a new logo, three versions of a brochure and a draft price list before lunch.
That’s genuinely exciting. It also creates a curious new problem. When making things was slow and expensive, businesses were forced to think carefully before they built. Now that making things is fast and cheap, it’s tempting to skip the thinking and go straight to building, then building again, and again, without ever finding out which version customers actually prefer.
Iteration, the habit of improving something through repeated small changes, has always been good advice. But the reason it matters has shifted. The hard part is no longer producing the next version. The hard part is learning whether it’s any better.
One Like insight: AI has made it cheaper to try things. It hasn’t made it any cheaper to find out whether they worked.
Building got cheap
The speed-up is real, and not only for software. In one early controlled experiment, researchers from Microsoft, GitHub and MIT asked developers to write a simple web server in JavaScript. Those given an AI coding assistant finished the task 55.8% faster than those working without one. That was a single, well-defined task, so it shouldn’t be stretched too far. But anyone who has used AI tools to draft a newsletter, mock up a flyer or rework a website will recognise the general direction.
For small businesses, this is mostly good news. Ideas that once needed a designer, a developer and a budget can now be roughed out in an afternoon. Owners who used to wait weeks to see an idea can now see five versions of it before the kettle boils.
The catch is psychological. When producing something takes effort, we tend to stop and ask whether it’s worth doing. When it takes almost none, that pause disappears. It becomes easy to confuse activity with progress: a new homepage this week, a different one next month, a fresh price list in between, each one feeling like an improvement because it’s new. Without some way of checking, a business can iterate busily in circles.
What the experimenters did differently
There’s a useful piece of research on what happens when businesses pair cheap changes with a habit of checking them. Rembrand Koning of Harvard Business School and Sharique Hasan and Aaron Chatterji of Duke University studied more than 35,000 technology start-ups founded between 2008 and 2013, tracking when they adopted A/B testing: showing different versions of a web page to different visitors to see which performs better.
Published in Management Science in 2022, their study found that relatively few start-ups used A/B testing at all. But among those that did, performance improved by 30% to 100% after a year, measured by how many people visited their websites. They also launched new products at a 9% to 18% higher rate.
The more interesting finding was about failure. Younger start-ups that tested were also more likely to fail faster. That sounds like bad news until you think about it. Testing told them sooner when an idea wasn’t working, so they could stop spending on it. In the researchers’ words, experimentation helped firms identify and scale promising ideas and fail faster when the signals were negative.
The researchers also found that small, unfunded start-ups seemed to have the most to gain from testing, yet used it the least. Hasan put the usual alternative bluntly, describing most small firms’ decision-making framework as “trust your gut.”
[Infographic placement] Two kinds of iteration. Two loops side by side. Left loop, labelled Busy iteration: Build → Launch → Build again → Launch again, with no checking step and a caption: “Feels like progress.” Right loop, labelled Learning iteration: Guess → Build something small → Measure → Decide (keep, change or stop) → Guess again. Caption: Koning, Hasan and Chatterji (2022): start-ups that tested grew 30–100% after a year, and the weaker ideas failed sooner.
Learning is the new bottleneck
Put the two findings together and the shape of the problem becomes clear. AI has made the “build” part of iteration fast. What it can’t speed up in the same way is the “learn” part, because learning depends on real customers doing real things, and they move at their own pace.
Most small businesses don’t have the website traffic for formal A/B testing, and that’s fine. The principle scales down well. What made testing valuable in the research had less to do with the software than with the discipline behind it: changing one thing, deciding in advance what success looks like, and then letting the result make the decision rather than gut feel.
For a small business, that might mean running the new price list for a month and comparing enquiries with the month before. It might mean sending half your customer list one email subject line and half another. It might mean offering two versions of a service to the next ten clients and noting which they choose. None of it needs special tools. All of it needs patience, and a willingness to be told you were wrong.
The trap to avoid is changing everything at once. If you redesign the website, rewrite the copy and adjust your prices in the same week, you might see a result, but you’ll have no idea which change caused it. Cheap building makes that temptation stronger, because changing everything costs almost nothing. Learning, though, needs a little restraint.
Highlight: AI lets you make ten versions. You still need customers to tell you which one works.
Iterating on purpose
So what does good iteration look like for a small business in an age of cheap building? It starts before the AI tool opens. Write down, in one sentence, what you expect the change to achieve and how you’ll know. “The new homepage will bring more booking enquiries” is a guess you can check. “The new homepage will look more modern” is a preference, which is fine, but it isn’t an experiment.
Then keep the change small enough to learn from, give it long enough to show a result, and decide in advance what you’ll do if it doesn’t work. As we explored in Design Thinking on a Small Business Budget, the cheapest test that answers your question is usually the right one.
A few questions can help:
- What exactly are we trying to improve, and how will we measure it?
- Are we changing one thing, or several at once?
- How long will we give this before we judge it?
- What result would make us drop the idea?
Which brings us back to the owner with the new website, logo and brochure before lunch. Nothing is wrong with making all of that quickly. The question is what happens next. If those versions go out into the world with a clear guess attached and a way of checking it, the speed becomes a real advantage. If they don’t, the business has simply found a faster way to keep guessing.
Sources
- Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”, arXiv, February 2023. Authors from Microsoft Research, GitHub and MIT.
- Rembrand Koning, Sharique Hasan and Aaron Chatterji, “Experimentation and Start-up Performance: Evidence from A/B Testing”, Management Science 68(9), 2022 (abstract; NBER working paper).
- Duke University Fuqua School of Business, news release on the study, via EurekAlert. Source of the Hasan quote.
