Companies routinely turn to A/B testing when evaluating the effectiveness of their product changes. Also known as a randomized field experiment, A/B testing has been used extensively over the past decade to measure the causal impact of product changes or variants of services, and has proved to be an important success factor for businesses making decisions.

With increased adoption of A/B testing, proper analysis of experimental data is crucial to decision quality. Successful A/B tests must exhibit sensitivity — they must be capable of detecting effects that product changes generate. From a hypothesis-testing perspective, experimenters aim to have high statistical power, or the likelihood that the experiment will detect a nonzero effect when such an effect exists.

Read more: https://research.facebook.com/blog/2020/10/increasing-the-sensitivity-of-a-b-tests-by-utilizing-the-variance-estimates-of-experimental-units/