Big little lies: a compendium and simulation of p-hacking strategies, 2023, Stefan & Schönbrodt

[CRG]

Royal Society Open Science

Big little lies: a compendium and simulation of p-hacking strategies

Angelika M. Stefan and Felix D. Schönbrodt

Abstract

In many research fields, the widespread use of questionable research practices has jeopardized the credibility of scientific results. One of the most prominent questionable research practices is p-hacking.

Typically, p-hacking is defined as a compound of strategies targeted at rendering non-significant hypothesis testing results significant. However, a comprehensive overview of these p-hacking strategies is missing, and current meta-scientific research often ignores the heterogeneity of strategies.

Here, we compile a list of 12 p-hacking strategies based on an extensive literature review, identify factors that control their level of severity, and demonstrate their impact on false-positive rates using simulation studies.

We also use our simulation results to evaluate several approaches that have been proposed to mitigate the influence of questionable research practices. Our results show that investigating p-hacking at the level of strategies can provide a better understanding of the process of p-hacking, as well as a broader basis for developing effective countermeasures.

By making our analyses available through a Shiny app and R package, we facilitate future meta-scientific research aimed at investigating the ramifications of p-hacking across multiple strategies, and we hope to start a broader discussion about different manifestations of p-hacking in practice.

Full article: https://royalsocietypublishing.org/doi/full/10.1098/rsos.220346

 

[Hutan]

I haven't read much of the paper yet, but it looks really useful. Good reading for any peer reviewer or patient representative in a research team. There's content there that would be useful for people about conducting research well and being critical about scientific papers. Also good for anyone interested in putting systems in place to ensure better quality research.

Moreover, we would like to stress that this paper should in no way be interpreted as an instruction manual for ‘successful’ p-hacking.

but it could be.

For example, Simonsohn et al. ([9], p. 670) write ‘a researcher may run a regression with and without outliers, with and without a covariate, with one and then another dependent variable, and then only report the significant analyses in the paper’.

a review of randomized trials in Denmark found that 50% of efficacy and 65% of harm outcomes per trial were incompletely reported [51]. This suggests that selective reporting of dependent variables is a common practice, even in fields with high standards for study preregistration.