A one-table-join statistics trick can cut the number of users you need for a reliable test.
This is worth your time because it is a concrete, low-effort technique for any test you run on a product or a signup funnel, and it gets you a reliable answer faster for the same traffic. The problem: the number of users a test needs grows directly with how noisy the results are. Double the noise in a metric and you double the users you need, which can turn a two-week test into a four-week one.
Much of that noise has nothing to do with whatever you changed. Some customers spend $400 a month and others spend $4, and every test has to wade through that gap. Random assignment balances this out across many experiments, but never inside any single one, so results bounce and the confidence interval, the range your true answer probably falls in, stays wide. The usual fixes are blunt: run longer, which burns calendar time; accept a bigger minimum detectable effect, the smallest change you are willing to try to measure, which means missing smaller real wins; or cap outlier values, which just changes what you are measuring.
A technique from Microsoft researchers called CUPED (Controlled-experiment Using Pre-Experiment Data) fixes this for the cost of one extra data join. It builds an adjusted metric using each user's history from before the test started: take their in-experiment result, then subtract a weighted version of how far their pre-experiment value sat from the group average (written as Y minus theta times the gap between their past value and the group's average past value). Because the pre-experiment value was measured before anyone was assigned a group, the test cannot have influenced it, so the adjustment removes noise without changing the real answer. At a correlation of 0.7 between a user's past and future behavior, CUPED cuts variance, how spread out the numbers are, by about 51 percent, roughly halving the needed sample. At a lower correlation, the gain drops sharply. Stable, high-variance metrics like revenue per user benefit most; sparse yes-or-no metrics often barely move. The catch: CUPED needs user history, so new users and new metrics get nothing, and the pre-experiment value must be fixed before the test starts, not chosen afterward.
Etsy published a candid account of using this. In 2021 the team added CUPED to metrics like conversion rate and measured only about a 7 percent reduction in noise, shortening experiments by almost one day on average, far below the 50 percent some vendors advertise. The gap came down to the data: purchases happen rarely and correlate weakly across time, while visit-based metrics correlate much more strongly. Rather than give up, Etsy borrowed an idea from DoorDash called CUPAC (Control Using Predictions as Covariate) and replaced the single pre-period number with predictions from a more powerful statistical model, expanding from 3 input signals to more than 100. A model like that catches patterns a simple straight-line adjustment cannot.
The lesson is to measure before you promise. Before telling anyone this halves test time, replay CUPED on your last 20 finished experiments to see what each metric actually gains, and make sure your planning math uses the adjusted variance. Start with one high-variance metric, run simulated "A/A" tests, comparing a group against an identical copy of itself, which should show no real difference, to confirm the false positive rate stays in line with expectations, and check every pre-experiment input for leakage, meaning anything that could still change after someone is assigned a group. An AI model can do the grunt work: ask it to rank candidate pre-experiment inputs by how useful they would likely be, flag anything that could leak, list which metrics lack enough history for new users, and write and run the actual math rather than trusting its own arithmetic.