Exploring Open Science Practices in Behavioural Public Policy Research

In their book `Nudge: Improving Decisions About Health, Wealth and Happiness', Thaler and Sunstein argue that choice architectures are promising public policy interventions. This research programme motivated the creation of so-called `nudge units' which aim to apply insights from behavioural science to improve public policy. We take a close look at a meta-analysis of the evidence gathered by two of the largest and most influential nudge units using statistical techniques to detect reporting biases. We find evidence suggestive of selective reporting on the full dataset, although this pattern is not robust in subgroup analyses. We therefore additionally evaluate the public pre-analysis plans from the Office of Evaluation Sciences. We find that the analysis plans and reporting usually lack sufficient detail to evaluate (unintentional) reporting biases. Nevertheless, we identify several instances of excellent practice. We additionally highlight several improvements that would enhance the effectiveness of the pre-analysis plans and reports as a means to combat reporting biases. We believe our findings and suggestions can further improve the evidence base on which policy decisions are made.