There were a number of questions and comments on yesterday’s post “conversion testing – how many actions?” and I wish I could respond to all of them.
Instead, I want to offer a practical alternative which plenty of people will take issue with, but I think it’s an important insight to put this all in perspective.
You see, in the early phase of a campaign, I certainly don’t wait for 100 actions. I rarely even wait for 30. Instead, (and this is only possible because I’m always involved in researching the market above and beyond adwords), if I see an ad pulling ahead THAT MAKES SENSE IN THE CONTEXT OF MY UNDERSTANDING OF THE MARKET … I’ll just go for it after 10 or 12 clicks. (There, I said it and lightening didn’t strike!)
This is INCREDIBLY RISKY from a purely statistical perspective, yet on a day to day basis I can get traffic flowing in a new market MUCH quicker than waiting for a 95% confidence level (e.g. 100 actions, or even 30). Moreover, my understanding of the market moves forward fast.
Do I accidentally delete winners when I do this? Yes, absolutely, positively, and definitely. (So use at your own risk)
But the time and money spent on the experiment is downgraded and I can move at the speed of the market. You see, the time and money you allocate to a test are ALSO a risk… leaving a test up too long has it’s costs too.
I start paying careful attention to statistical probabilities AFTER I’ve got something worth protecting (a “control worth beating”). The more it’s worth, the more successful history the ad has, the more protective I am, and the higher level of statistical confidence (or greater number of actions) I’ll require.
So my personal interpretation of Google’s recommended 100 actions is that WHEN YOU’VE GOT A CONTROL WORTH PROTECTING, what we previously thought was enough actions really isn’t, given the “noise” on the internet. (“Noise” = number of advertisers coming in and out of the market, changes in the economic news which are instantly available online, varying ad position and impression levels, changes in the organic rankings, public events, … even such things as internet worms and viruses which affect browsing behavior)
Bottom line?
When you’ve got something already making money, it might pay to wait longer than we’ve all traditionally thought. How much longer? Probably twice as long as you think.
But ultimately it depends upon how much you’re making, how much you already know about your market, and how big a bridge you’d jump off if you accidentally killed the winning ad 
{ 10 comments… read them below or add one }
Glenn,
This is the post that makes sense. I’m only starting to do PPC, and at the start, I have 10 to 20 ideas per campaign. It would take far too long to wait for 30 clicks for each one of those ideas, since 90% would turn out to be non-starters.
Rene
Hi,
I use very few keywords. I get results. That is important.
Thanks for the article.
regards
Prasaad
Hilarious…I don’t think I’d have the patience to do full test and would probably follow the same guidelines you state above if I know the market. A great way to catalyze your test I think would be to use Google trends and Google insights and maybe even eBay depending on your product.
That’s nifty: that’s pretty much ‘zactly what I’ve been up to! And it’s probably A Good Thing for an entrepreneur like myself to be doing things just like Glenn Livingston.
Thanks, Glenn, I’ll keep up the good work.
Here’s a quick success story from using this strategy …
I had a campaign that was already a solid success, a nice money machine that pulled in at least $2 for every buck invested. I never stopped split-testing. For over a year, I never saw anything but tests with ambiguous short-term results. Literally dozens of ads were tried and chucked relatively quickly because no ad was showing any obvious potency.
Then one week a new ad came along and simply kicked butt, immediately. The signal screamed out from the noise. Sure, it might have been a fluke, but it wasn’t. It just kept right on kicking butt. That ad reached out to real buyers in a way that no other ad I’d written ever had (and I still don’t know why).
Months later, it still pulls more clicks, and converts almost 100% better, than any other ad I’ve ever written. With the high CTR allowing lower bids, that campaign now spits out FIVE dollars for every dollar invested. 500% return. Seriously. Unbelievable.
The point: it was pretty obvious when a truly superior ad came along!
(And, not to throw the baby out with the bath water, I have also achieved successes by a series of thoroughly-tested incremental improvements. It depends!)
Thanks for the continuing information.
It’s an interesting approach.
I’ve found that even ads that were once controls can pull very marginally for the first few clicks and then go back to their original CTR after they’ve run for a while.
Thank you for agreeing with my point that obvious winners should be pushed along faster. As always I appreciate your point as well and possibly a strategy to look at would be when the ad starts to see some “burn out ” you can go back in and retest ads that you deleted with out a 30 or 100 test to see if you possibly did delete a hot ad prematurely.
As always thanks for the powerful marketing insights!
Cheap Charley
“I use very few keywords. I get results. That is important.
Thanks for the article.
regards
Prasaad”
Nice pointer
Good post, I used google adwords before and I know what you mean on this, I’m really trying to focus on mastering google adwords and all information I can get helps, thanks.
Terrance Charles
http://www.twitter.com/TerranceCharles
We work on thin click numbers (but very high ROI’s) on many ads. So we have to use small samples plus market knowledge to guide our testing. Glenn is right on about that.
We have an additional complicating factor that some marketeers also may need to consider. Our market is worldwide. Thus, the timespan of the test must be calibrated. Is one day adequate? Not if you are snagging the start of a weekend somewhere in the world. How about major holiday impacts, or events like Ramadan? (our offerings are workplace sensitive).
Great blog, Glenn. Thanks.
Good article. I think once you get the feel for it you can make good predictions based upon early results. You may miss a few but then moving with more speed you may also find more.