Increasing experimental power with variance reduction at the BBC

This article discusses how the Experimentation team have been accounting for pre-experiment variance in order to increase the statistical power of their experiments

In the Experimentation and Optimisation team at the BBC we have the privilege to be working with very large sample sizes, which means we can confidently determine how our variants have performed in a controlled experiment. As the BBC continues to iterate and optimise, changes can produce smaller effect sizes and determining whether changes are statistically significant or otherwise becomes more difficult. In this instance the demand for greater volumes of traffic also increases.

Despite the positive relationship that exists between sample size and experiment sensitivity, the large within-metric variance that is exhibited within our data is a constant, and thus often inhibits the detection of a statistically significant result.

For this reason, the Experimentation and Optimisation team at the BBC are implementing variance reduction methods in experiment analysis to detect significant findings for smaller effect sizes. One such method is CUPED which relies on the removal of variance within a metric that is attributed to pre-experiment information. CUPED is implemented on the notion that variance that pre-experiment data can explain in a metric is not correlated to…

Related posts:

Sebagai pamungkas dari para pendahulunya dalam series Reply yang rilis pada tahun 2015 lalu, Reply 1988 rupanya sangat membekas bagi para penonton drama korea dengan genre komedi satir. Karena…

So you like DatoCMS but you created your project with Contentful? No problem, we released contentful-to-datocms, a Node.js-based CLI tool that copies all content from a Contentful space to a DatoCMS…

Young adults today have the necessary tools and technology to reach thousands of individuals at once, as well as having access to numerous support groups based on their interests and needs and…