An A/B test is an experiment that compares two different versions of something and measures which one is more effective. These two different versions are often called the control and the test, or the control and the variant.

Example A/B Test:

An e-commerce company may want to test a new product page design.

Some people in the company hypothesize that the new product page design may lead to a higher percentage of users that make a purchase. Before they roll out the new product page design, they would run an A/B test to measure whether or not this is the case.

For the A/B test experiment, the e-commerce company may show 10% of their users the new product page (test or variant group) and the remaining 90% of their users would see the standard product page (control group). Over time, the company will measure the conversion rate on each page in order to understand which product page actually drives the most purchases before making any decision.

A/B tests can be run on much more than just page designs though. For example, you may want to test different versions of a search algorithm, recommendation engine or user onboarding flow. Typically, there will be a key set of metrics that you are measuring to understand the impact of the control and test variants.