Marketing should be judged by what it changes, not by what it’s standing next to when the sale happens.
A small chain of three dental practices pays for search ads every month. The ads dashboard looks excellent: hundreds of clicks, dozens of booked appointments, a strong return on spend. The biggest performer, by far, is the ad that appears when people search for the practice’s own name.
Then the practice manager asks an awkward question. If someone types the name of our practice into Google, weren’t they already looking for us? Would they have found us anyway, through the free listing right underneath the ad?
It’s one of the most important questions in measuring marketing: not “who clicked our ad before buying?”, but “who bought because of our ad?” Those can be very different numbers, and one of the largest experiments ever run on online advertising showed just how different.
Marketing should be judged by what it changes, not by what it’s standing next to when the sale happens.
What eBay discovered
In 2015, Thomas Blake and Chris Nosko of eBay Research Labs and Steven Tadelis of the University of California, Berkeley, published the results of large-scale experiments at eBay in Econometrica, one of the most respected journals in economics.
First, eBay stopped paying for search ads on queries that included its own name, such as “ebay shoes”, on certain search platforms. Traffic barely changed. People who had been clicking the paid ad simply clicked the free, organic listing instead. The researchers concluded that brand keyword ads have no measurable short-term benefits.
Then eBay switched off non-brand search ads, for terms like “used gibson les paul”, in a randomly chosen 30% of the United States for 60 days. The ads did influence new and infrequent customers. But most of the ad spending went on frequent eBay users who would have bought anyway. On average, the returns on paid search were negative, far lower than the dashboard figures had suggested.
The core problem is that people who are already about to buy are also the most likely to click an ad. So simply counting clicks or attributed sales can give advertising credit for sales it didn’t cause.
eBay is a very well-known brand, and the results may differ for smaller businesses that people don’t already know. For a little-known business, search ads can genuinely bring in customers who would never have found it otherwise. But the principle applies to everyone: measure what advertising adds, not just what it’s associated with.
People already about to buy are also the most likely to click an ad.
Measuring what marketing actually adds
Marketers call the true extra effect of an activity its incrementality: the sales that wouldn’t have happened without it. Small businesses can’t run experiments on the scale of eBay, but they can borrow the thinking.
Question your brand-name ads. If you pay for ads when people search for your business name, try pausing them for a few weeks, ideally when no competitor is bidding on your name, and watch whether total enquiries or bookings actually fall. Keep them if competitors are actively bidding on your name, since that changes the picture.
Run simple on-off tests. Switch an activity off, or on, in a defined period or area and compare results with a similar period or area where nothing changed. It’s not perfect, but it’s far better than relying only on clicks.
Look at total results, not just attributed ones. If an advertising channel claims 50 new customers a month but your total new customers don’t change when you pause it, it probably wasn’t creating most of them.
Focus spending on new customers. The eBay research found ads mattered most for people who didn’t already know the business. Target advertising at people who haven’t bought from you yet.
Ask customers. “How did you first hear about us?” is a simple, useful cross-check against what dashboards say.
INFOGRAPHIC GOES HERE
Did the ad cause the sale? Two paths for a customer searching for a business by name. Path 1: clicks the paid ad → books. Path 2 (ad switched off): clicks the free listing just below → books anyway.
Who clicked our ad? → Who bought because of our ad?
A healthy scepticism
None of this means advertising doesn’t work. Earlier articles in this series, such as Are You Loud Enough to Grow? and What Happens When You Stop Showing Up, show that visibility matters. The point is to measure carefully, because the platforms reporting your results also sell you the ads.
Choose one paid activity this quarter and test its real effect. Pause it, or reduce it sharply, for a defined period, and compare total enquiries and sales with a similar period before. If results barely change, redirect that money to activity that reaches genuinely new customers.
A few questions can help:
- Which of our ads target people who already know us and might find us anyway?
- Have we ever paused an activity to see what really changes?
- Do our total results move when our attributed results move?
- How much of our advertising reaches people who have never bought from us?
Which brings us back to the dental practices and their best-performing ad. The ad for their own name looked brilliant because it was catching patients who were already on their way. After a careful four-week pause showed bookings barely changed, the practice could move that budget to ads reaching people searching for a dentist in their area for the first time, which is where advertising can truly add something.
Measure what marketing adds, not just what it’s associated with.
One Like Research & Further Reading
- Thomas Blake, Chris Nosko and Steven Tadelis, “Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment”, Econometrica 83(1), 2015. NBER working paper version.