Variability in the analysis of a single neuroimaging dataset by many teams, 2019, Botvinik-Nezer et al

[InitialConditions]

I have just seen this new paper on Twitter. A timely reminder that imaging and the subsequent analysis is not an exact science.

https://www.biorxiv.org/content/10.1101/843193v1

Abstract
Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.

 

[Cheshire]

Merged thread - article about the paper

For decades, both the research and medical communities have relied on neuroimaging tools like functional magnetic resonance imaging (fMRI) to give them a window into the living human brain. Such scans have provided unprecedented insights into the brain’s structure and function – and the field, as a whole, has used this technique to better understand how the brain gives rise to thoughts, emotions, and actions. But as neuroimaging technology has advanced, so have the different analysis tools and the number of ways one can evaluate the resulting data. Now, the results of unique research project, the Neuroimaging Analysis, Replication, and Prediction Study (NARPS), suggest that different analyses can lead to strikingly different results from the same data set.

Poldrack and colleagues recruited 70 teams of scientists from around the world to analyze the same brain imaging data set of 108 study participants performing a well-known decision-making task called the mixed gambles task. The task requires participants to decide whether to accept or reject a particular gamble that will lead to gain or losing money. It is a task that has been widely used to demonstrate that human beings, in general, are much more sensitive to potential losses than gains when making decisions.

“When we first started reaching out to different labs to participate, we particularly focused on the neuroeconomics community, because this is a task that they are familiar with,” said Poldrack. “Once they agreed to take part, we basically just gave them the data set, nine set hypotheses asking if they saw activation in a particular brain area, and a three-month deadline. We told them to analyze the data however they normally would in their labs to address those hypotheses.”

https://dana.org/article/neuroimaging-many-analysts-differing-results/

 

[Hutan]

I think this is an important paper when helping people understand the very large potential for bias in fMRI studies.

As well as this one, I've referred to these:

A Machine Learning Approach to the Differentiation of Functional Magnetic Resonance Imaging Data of Chronic Fatigue Syndrome (CFS) From a Sedentary Control
A study of people with CFS and healthy controls of a similar size to that proposed here found 29 regions of interest on Day 1 and 28 regions on Day 2 after a physical and cognitive challenge. However, only 10 of the regions of interest were common to both days, demonstrating how important it is to manage activity prior to fMRI.

Time of day is associated with paradoxical reductions in global signal fluctuation and functional connectivity
fMRI also varies during the day; it has been suggested that variation in time of day could potentially account for between-study variation in results and failed replications.