pipette
ESEspañol

On the Benefit of Blocking for Online Experiments with Skewed Data

Abbas Zaidi, Matthew Reimherr, Rina Friedberg, Fabian Martinez, Richard Mudd

Preprint

In the authors' words

This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%) due to the limits of discretization, the authors demonstrate it offers two crucial structural benefits over post-hoc statistical adjustments. First, fixed-weight blocking correctly targets the true average treatment effect, avoiding the severe bias introduced by efficiency-weighted alternatives. Second, blocking localizes extreme outliers, enabling targeted within-block winsorization that effectively controls noise without destroying the tail-end treatment effect.

Main resultLimitation the authors admit

Appeared: Thursday, October 8. arXiv. Preprint, not yet peer-reviewed.