There were a number of questions and comments on yesterday’s post “conversion testing – how many actions?” and I wish I could respond to all of them.
Instead, I want to offer a practical alternative which plenty of people will take issue with, but I think it’s an important insight to put this all in perspective.
You see, in the early phase of a campaign, I certainly don’t wait for 100 actions. I rarely even wait for 30. Instead, (and this is only possible because I’m always involved in researching the market above and beyond adwords), if I see an ad pulling ahead THAT MAKES SENSE IN THE CONTEXT OF MY UNDERSTANDING OF THE MARKET … I’ll just go for it after 10 or 12 clicks. (There, I said it and lightening didn’t strike!)
This is INCREDIBLY RISKY from a purely statistical perspective, yet on a day to day basis I can get traffic flowing in a new market MUCH quicker than waiting for a 95% confidence level (e.g. 100 actions, or even 30). Moreover, my understanding of the market moves forward fast.
Do I accidentally delete winners when I do this? Yes, absolutely, positively, and definitely. (So use at your own risk)
But the time and money spent on the experiment is downgraded and I can move at the speed of the market. You see, the time and money you allocate to a test are ALSO a risk… leaving a test up too long has it’s costs too.
I start paying careful attention to statistical probabilities AFTER I’ve got something worth protecting (a “control worth beating”). The more it’s worth, the more successful history the ad has, the more protective I am, and the higher level of statistical confidence (or greater number of actions) I’ll require.
So my personal interpretation of Google’s recommended 100 actions is that WHEN YOU’VE GOT A CONTROL WORTH PROTECTING, what we previously thought was enough actions really isn’t, given the “noise” on the internet. (“Noise” = number of advertisers coming in and out of the market, changes in the economic news which are instantly available online, varying ad position and impression levels, changes in the organic rankings, public events, … even such things as internet worms and viruses which affect browsing behavior)
Bottom line?
When you’ve got something already making money, it might pay to wait longer than we’ve all traditionally thought. How much longer? Probably twice as long as you think.
But ultimately it depends upon how much you’re making, how much you already know about your market, and how big a bridge you’d jump off if you accidentally killed the winning ad 
I’m at The System seminar and I just heard Trevo Claiborne from the Google Website Optimizer team speak. He casually mentioned they recommend ONE HUNDRED conversions/action PER OPTION you are testing.
This is in direct contrast to the standard rules of thumb circling around the direct marketing world. (Most say 30, or 40 maximum is plenty).
Now, with my statistical background, rules of thumb have always bothered me in the first place. I’d rather see the appropriate statistical test applied, but I DO understand the practical need to move at the speed of business and use heurisitics and shortcuts.
But I was incredibly surprised to hear Trevor say ONE HUNDRED in contrast to the 30 we’re all used to hearing. So I asked whether this was based upon their observations across all their advertisers … I got a bit of a disclaimer, and then an answer I took for yes.
Here’s why I think this is a radical statement and the implications for us all.
First of all, Google knows.
I’m not doubting their observations.
They must SEE something we don’t. And I’m guessing what that is … what it MUST be, is that there is MUCH MORE NOISE on the internet than there is in traditional direct marketing environments.
This explains why so many clients report confusion after having declared a winner … then later find their profits haven’t really risen correspondingly.
So we probably should all lean in the direction of longer lasting, more statistically robust tests … which translates to MORE actions before making a decision.
On the other hand, the vast majority of clients I’ve spoken with don’t have the necessary traffic to accomplish this in any reasonable time frame.
So what’s the solution?
FEWER AND MORE WELL CHOSEN TESTING OPTIONS RUN LONGER. It really doesn’t make sense to throw everything against the wall. Do your research first and foremost to determine what MIGHT be likely to improve your conversions. This means surveying people as they’re exiting your page, installing live chats, monitoring competitive sites, reading repeatedly appearing sales copy in your market and identifying very strong candidates for conversion enhancement.
A plain old A-B test with well thought out inputs is probably more valuable than testing umpteen different options.
Of course, Taguchi starts to look more attractive in this situation (a method of compressing dozens of tests into one), but you have to remember that the same rules apply … we need MORE actions than we’re used to considering before we get really robust and stable results.
Which means your inputs for Taguchi testing are even more important.
Bottom line?
Google says there’s LOTS AND LOTS of noise online, and you have to test carefully and watch the results for a long time to be sure of them.
Believe them, and design your tests wisely since they’re not as “disposible” as you might have previously thought.
G
