Opinion: Reproducibility failures are essential to scientific inquiry, Redish et al, 2018

[Indigophoton]

Current fears of a “reproducibility crisis” have led researchers, sources of scientific funding, and the public to question both the efficacy and trustworthiness of science (1, 2).

Suggested policy changes have been focused on statistical problems, such as p-hacking, and issues of experimental design and execution (3, 4). However, “reproducibility” is a broad concept that includes a number of issues (5) (see also www.pnas.org/improving_reproducibility).

Furthermore, reproducibility failures occur even in fields such as mathematics or computer science that do not have statistical problems or issues with experimental design. Most importantly, these proposed policy changes ignore a core feature of the process of scientific inquiry that occurs after reproducibility failures: the integration of conflicting observations and ideas into a coherent theory.

Here we argue, using examples from mathematics and computer science, that current discussions of the reproducibility crisis overlook the essential role that failures of reproducibility play in scientific inquiry. This viewpoint that reproducibility is a key part of inquiry suggests several new perspectives and policies to promote good science.

First, science needs to be given the time necessary to reconcile conflicting results. It typically takes decades to probe the parameter space of a discovery to identify and characterize the fundamental variables.

Second, reproducibility failures are a critical part of this journey, and attention must be paid to the process of reconciling conflicting results.

Third, success or failure should not be based on the conclusions of one or a few studies; strategies such as those of theoretical and synthesis articles that integrate diverse perspectives should be encouraged.

We argue that the decades-long process of metabolizing reproducibility failures through theoretical integration is what leads to the reliable results across the sciences that have provided us with remarkable life-changing medical and engineering consequences.

http://www.pnas.org/content/115/20/5042

 

[alktipping]

I remember from chemistry at school that although chemistry teachers did the same experiments with different classes they could never be certain of getting the same results 100% of the time to many variables .

 

[BruceInOz]

I remember from chemistry at school that although chemistry teachers did the same experiments with different classes they could never be certain of getting the same results 100% of the time to many variables .

What this article is saying is that your chemistry teachers should view their inability to replicate as indicative that there is something about the process that they haven't fully understood. Some hidden variable that they thought was irrelevant but is in fact important. An unknown impurity in one of the chemicals or something that needed to be measured to greater accuracy or something. Science progresses by using the lack of reproducibility to ask more questions and uncover the hidden variables.

Great article. Well worth the read.