My Only Guarantee: If You’re Not A/B Testing, Your Marketing Programs Aren’t As Good As They Could Be

There are not a lot of certainties for small business owners, but the one thing I can guarantee is that you’re not always right.  Neither are the marketing consultants you might talk to or the “best practices” articles you might read.  That’s why A/B testing is the one thing guaranteed to improve virtually every marketing program you have.

The concept behind A/B testing is pretty straightforward.  Try making one, and only one, change to your marketing campaign and see if it makes you more or less money.  For example, would more people respond to a restaurant promotion offering $20 off the total bill or a free bottle of wine?  We can guess or make assumptions all day long, but the only way to know for sure is to test the two offers head to head.  If you’re doing a direct mail promotion, this literally means splitting your mailing list into two groups and presenting one offer to one group and the other offer to the second group.  It’s very important that you only change one thing at a time, so keep every other aspect of the promotion identical, including when the mailings are sent (i.e. send them all at the same time).  Larger businesses with high volumes and big budgets can try something called multivariate testing where you change multiple things at the same time, but it’s far more complicated and expensive so I wouldn’t generally worry about that for most small business marketing programs.

There are a couple of other very important things to keep in mind when A/B testing your marketing programs.  First, you’re going to need to ensure you can track the results.  This includes measuring the difference in both the expenses and revenues.  For example, if you increase your promotion incentive from $10 to $20, you can’t just look at response rates to see which is better.  The $20 version needs enough additional responses to make up for the increased cost.  Also, you need to ensure there’s a way for you to tell which promotion generated the purchase.  For some channels like Search Engine Marketing (Google advertising), this is easy.  For other channels like a restaurant direct mail promotion, it may be a bit harder (but it’s still possible) to track the net profit generated for each test group.

The last thing I’ll touch on for now is the topic of sample size.  I’m not going to spend a lot of time talking about statistics, but it’s important to understand that your two test groups must be large enough to ensure that one group didn’t outperform the other purely by chance.  If you randomly mail different promotions to two groups of ten people and one person responds from Group A and two people respond from Group B, it will look like Group B was dramatically more successful.  However, because the groups were so small, you don’t really know if it was a better promotion or just random chance that made it more successful.    The more people you randomly include in each group, the more confident you can be that it was the change in your promotion that drove the increased response rate.

To summarize, here are the key things to remember for A/B testing:

  1. Only test one variable at a time
  2. You need to be able to track the results
  3. You need a large enough sample size
  4. You should start right away to improve the profitability of your marketing campaigns

I’ve discussed direct mail quite a bit, but A/B testing is also important for other marketing channels like search engine marketing and your general website.  I’ll have more on this topic in the coming posts.  In the meantime, thank you for reading.

Tags: , ,

Leave a Reply