Lies, damned lies and statistics: The adverse incentive effects of the publication bias

In this project I investigate the use and misuse of p values in papers published in five (high-ranked) journals in experimental psychology. I use a data set of over 135 thousand p values from more than five thousand papers. I inspect (1) the way in which the p values are reported and (2) their distribution. The main findings are following: first, the authors choose the mode of reporting their results in an arbitrary and strategic way, that is, to make them seem more statistically significant than they really are (which is well known to improve the chances for publication). Specifically, they frequently report p values "just above" significance thresholds directly, whereas other values are reported by means of inequalities (e.g. "p<.1"), they round the p values down more eagerly than up and choose between the significance thresholds and between one- and two-sided tests after seeing the data. Further, about 8% of reported p values are inconsistent with their underlying statistics (e.g. F or t) and it appears that there are "too many" "just significant" values, suggesting use of data or model manipulation techniques to bring the p value to the right side of the threshold.