FROM MY BEST SELLER: Make Analytics Great Again

A/B testing is a way of comparing two versions of the same variable, usually by testing a subject’s response to variable A against variable B and determining which of the two variables is more effective.
http://en.wikipedia.org/wiki/A/B_testing

Consider this: A company wants to increase the number of users who subscribe to their homepage newsletter. It’s a simple exercise in hypothesis (in this case, speculating) and testing. In this case, the company offers 50% of its users the old subscription page, and they provide the other 50% what they think will attract more subscriptions. They run the test, compare the statistics for the old page to the new page, and make their decisions based on that. A is the old page, and B is the proposed page. It’s that simple.

When to use: A/B testing is most appropriate when you want to test and compare two or more variations of the same element to determine which one works best for a specific metric or goal.

When not to use: You decide. It’s not a difficult thing to assess.

Conversely, don’t use A/B Testing when:
 Small sample size or low traffic volume.
 Lack of time to run the test or analyse the results.
 Minor, subtle changes that won’t have a noticeable impact.
 Absence of clear hypotheses or goals.
 Ethical or legal issues surrounding testing practices.
 Negative effects on user experience.
 Results may be misinterpreted or exaggerated.
 Misalignment with core business goals.
 Lack of tools or knowledge to run practical tests.