Someone meets you at a conference and says that you that you should do an A/B test on the landing pages of your new startup but you have no idea what it is and why you should bother. In this short post, I will try to quickly explain whats an A/B test and its sibling - the multivariate test, highligting the differences between the two and help you understand when you might want to choose one over the other.

A/B testing

  1. Used for testing two quite different versions of a design as a whole.
  2. Effective during the early phases of design when you are primarily brainstorming and need validation for your ideas.
  3. Example: You have two different landing pages and you want to know which ones of them has a higher engagement/conversion rate. You will need to define what you conversion rate it in terms of an end goal that you intend the user to complete. e.g. clicking on the buy button, downloading a white paper, saving items into a playlist etc.
  4. Works very well when you don’t have a lot of traffic and need quick feedback on your design because in order to perform this test you need to split your traffic between the two versions of your site.

For an A/B test, just ensure that you are trying to measure the effect of only one change. e.g. Dont change the color of a banner and the size of the button at the same time in your alternate version otherwise it will be difficult to tell which change caused the outcome.

Multivariate testing

  1. Used to identify the effect of variations to certain elements in a given design design.
  2. Effective during the the later stages of a testing when you have mostly settled on the major aspects of the overall design and are trying to tweak the smaller details of the design, e.g the size/color of a button/banner etc.
  3. Example: You have a landing page with an image, a buy button and a banner. You want to find out which variation of these three elements when combined together leads to a higher conversion rate.
  4. Just like A/B testing, you will need to split your traffic between the different versions of your multivariate test. You can create as many variations as you wish. However, the more variations you create, a higher volume of traffic is needed to correctly determine if a particular version reached your conversion goals.

You are not limited to testing exactly two versions of your application just because a test is named an A/B test. If you are testing 3 or 4 versions, it is usually called an A/B/C or A/B/C/D test and its completely fine to do so as long as you're not testing multiple variables.

Its always better to use these techniques to get early feedback on your designs rather than waiting for user sessions which are much harder, and not to mention - expensive, to conduct.