Large P-Values Cannot be Explained by Power Analysis, André, 2022

Excerpt:

Could the p-values in a paper be large because the researchers have conducted a careful power analysis?

The argument goes like this:

1. Researchers don’t want to be wasteful with their resources, so they do their best to predict the effect size that their experiment will yield.
2. Once they know this effect size, they perform a power analysis to determine how many observations they should collect.
3. Because they run exactly the number of participants they need (not more, not less) to detect a significant effect, they will tend to find p-values that are close to .05.

Another flavor of this argument is “small p-values means that researchers were wasteful, collected too many observations, and made their effect too significant”.

Are there any merits to this argument? Could large p-values be explained by thoughtful researchers collecting just enough participants to detect a carefully-estimated effect size? Let’s dive in!

[…]

Conclusion

Researchers cannot “aim” for p = .05, not even with a careful, perfectly accurate, power analysis.

In fact, the opposite is true: If their power analysis is accurate (i.e., gives them a good chance to detect their effect), they should get p-values that are much smaller, rather than close, to .05.

The most likely explanation for a set of multiple p-values hugging .05 is low statistical power, and/or p-hacking.

Blog post: https://quentinandre.net/post/large-pvalues-and-power-analysis/

https://twitter.com/user/status/1560395816283516928