The owner of an online plant shop is certain that a new, beautifully designed home page will boost sales. It has bigger photos, a cleaner layout and a moving banner. Her designer agrees. Her friends love it. She launches it with excitement.
Sales fall by a tenth over the next month. When she tests the old page against the new one, showing each to half her visitors, the old page wins clearly. The moving banner pushed the “shop now” button below the fold on mobile phones, where most of her customers browse.
She isn’t unusual. Some of the world’s largest technology companies, which run thousands of controlled experiments a year, have found that most ideas, even ones experts are confident about, don’t improve the results they were meant to. That’s not a reason to stop trying new things. It’s a reason to test them.
One Like insight: Confidence in an idea is not evidence that it works.
What the experimenters found
In 2017, Ron Kohavi, who led experimentation at Microsoft, and Stefan Thomke of Harvard Business School published an article in Harvard Business Review called “The Surprising Power of Online Experiments”. They described how companies such as Microsoft, Amazon, Google and Booking.com run controlled experiments, often called A/B tests, in which different visitors are randomly shown different versions of a page, email or feature, and the results are compared.
Their most humbling finding concerned how often ideas succeed. At Microsoft, they reported, roughly a third of experiments improved the metrics they were designed to improve, a third made no meaningful difference, and a third made things worse. At search engines such as Bing and Google, the success rate of ideas was even lower.
The article also described how tiny changes could have unexpectedly large effects. In one example, a small change to how ad headlines were displayed in Bing, an idea that had sat untested for months because nobody thought it important, increased revenue by around 12%, worth over $100 million a year in the United States.
The lesson the authors drew was that intuition, even expert intuition, is a poor guide to what customers will actually respond to. The only reliable way to know is to test.
These companies have millions of visitors, which makes precise testing possible. Small businesses have far less traffic, so their tests will be rougher. But the underlying humility applies to everyone.
Simple testing for small businesses
You don’t need specialist software or millions of visitors to test ideas. A few practical approaches work well at a small scale.
Test one change at a time. Change the headline or the photo or the price, not all three at once, so you know what made the difference.
Use built-in tools. Many email platforms let you send two subject lines to small groups and automatically send the winner to everyone else. Many ad platforms let you run two versions of an ad side by side. Some website builders offer simple A/B testing.
Compare periods carefully. If you can’t split visitors, compare a change against a similar period before, ideally the same weeks last year or a period with similar seasonality, and be cautious about drawing firm conclusions.
Give tests long enough. With modest traffic, a test may need several weeks before results are meaningful. Don’t declare a winner after two days.
Decide what success means first. Choose the measure before you start: bookings, sales, enquiries or sign-ups, not just clicks or likes.
Keep a record. Note what you tested, what you expected and what happened. Over time, you’ll build real knowledge about your own customers. As we explored in Building Is Cheap Now. Learning Isn’t., learning is the scarce resource.
[Infographic placement] How often do ideas actually work? A simple pie chart from Microsoft’s experiments: about one-third improve results, one-third make no difference, one-third make things worse. Beside it, a five-step loop: Idea → Test one change → Wait long enough → Measure what matters → Keep, drop or refine. Caption: Kohavi and Thomke (2017), Harvard Business Review.
Building a testing habit
Pick one thing to test this month. Good starting points are email subject lines, the main headline on your home page, the photo on your most important product or service page, or the wording of your booking button. Write down what you expect to happen, run the test properly, and see whether you were right. Many business owners are surprised by how often their instincts are wrong, and how quickly testing pays off.
Then make it routine: one small test each month. It turns marketing from a series of guesses into a gradual accumulation of evidence about what your customers actually respond to.
A few questions can help:
- Which important decisions have we made recently based purely on opinion?
- What one change could we test this month?
- What result would tell us the change worked?
- Do we keep a record of what we’ve tested and learned?
Which brings us back to the plant shop and its beautiful new home page. The owner’s mistake wasn’t having a bold idea; it was trusting it without testing. By running the new design against the old one for a few weeks before switching fully, she would have spotted the hidden button problem early, fixed it, and perhaps ended up with a page that was both beautiful and better at selling plants.
Sources
- Ron Kohavi and Stefan Thomke, “The Surprising Power of Online Experiments”, Harvard Business Review, September–October 2017.
- Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020), general reference.
