The Winner’s Curse In Data-driven Decision Making: Evidence And Solutions

Abstract

Data-driven decision making involves estimating the value of each potential option and selecting the one with the highest estimated efficacy. This approach underpins a wide array of modern marketing, operations, and AI applications, including A/B testing, advertising and bidding, pricing, and personalized targeting. However, several papers have shown that the estimated effectiveness of the chosen options will be systematically over-optimistic (Smith and Winkler 2006, Efron 2011, Andrews et al. 2024), even when the estimated outcomes are themselves unbiased and efficient. Using simulations calibrated to realistic parameter values from recent marketing studies, we first demonstrate that the magnitude of the winner's curse is often high in relevant marketing contexts, and that the severity of the winner's curse depends on the true performance difference between options relative to the level of noise in the data, the number of alternatives under consideration, and the number of observations per tested condition. We further show that using machine learning methods to evaluate what treatment to give to individual consumers can lead to extremely high levels of winner's curse, especially if the machine learning functional form is very flexible. We propose a correction method based on a non-continuous bootstrap, and benchmark our method against several existing proposed solutions across many common marketing scenarios. We demonstrate that our bootstrap approach generally performs well, and usually outperforms the solutions that have been previously proposed in the literature.