Journal article
Variability in the analysis of a single neuroimaging dataset by many teams
Nature (London), Vol.582(7810), pp.84-88
06/04/2020
DOI: 10.1038/s41586-020-2314-9
PMCID: PMC7771346
PMID: 32483374
Abstract
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
Details
- Title: Subtitle
- Variability in the analysis of a single neuroimaging dataset by many teams
- Creators
- Rotem Botvinik-Nezer - Tel Aviv UniversityFelix Holzmeister - Universität InnsbruckColin Camerer - California Institute of TechnologyAnna Dreber - Stockholm School of EconomicsJuergen Huber - Universität InnsbruckMagnus Johannesson - Stockholm School of EconomicsMichael Kirchler - Universität InnsbruckRoni Iwanir - Tel Aviv UniversityJeanette Mumford - University of Wisconsin–MadisonR. Alison Adcock - Duke UniversityPaolo Avesani - University of TrentoBlazej Baczkowski - Max Planck SocietyAahana Bajracharya - Washington University in St. LouisLeah Bakst - Boston UniversitySheryl Ball - Virginia TechMarco Barilari - UCLouvainNadège Bault - University of PlymouthDerek Beaton - University of TorontoJulia Beitner - Goethe University FrankfurtRoland Benoit - Max Planck SocietyRuud Berkers - Max Planck SocietyJamil Bhanji - Rutgers, The State University of New JerseyBharat Biswal - New Jersey Institute of TechnologySebastian Bobadilla-Suarez - University College LondonTiago Bortolini - D’Or Institute for Research and EducationKatherine Bottenhorn - Florida International UniversityAlexander Bowring - University of OxfordSenne Braem - Vrije Universiteit BrusselHayley Brooks - University of DenverEmily Brudner - Rutgers, The State University of New JerseyCristian Calderon - Ghent UniversityJulia Camilleri - Heinrich Heine University DüsseldorfJaime Castrellon - Duke UniversityLuca Cecchetti - IMT School for Advanced Studies LuccaEdna Cieslik - Heinrich Heine University DüsseldorfZachary Cole - University of Nebraska–LincolnOlivier Collignon - UCLouvainRobert Cox - National Institutes of HealthWilliam Cunningham - University of TorontoStefan Czoschke - Goethe University FrankfurtKamalaker Dadi - Université Paris-SaclayCharles Davis - University of ConnecticutAlberto De Luca - University Medical Center UtrechtMauricio Delgado - Rutgers, The State University of New JerseyLysia Demetriou - University of OxfordJeffrey Dennison - Temple UniversityXin Di - New Jersey Institute of TechnologyErin Dickie - University of TorontoEkaterina Dobryakova - Kessler FoundationClaire Donnat - Stanford UniversityJuergen Dukart - Heinrich Heine University DüsseldorfNiall Duncan - Taipei Medical UniversityJoke Durnez - Stanford UniversityAmr Eed - Consejo Superior de Investigaciones CientíficasSimon Eickhoff - Heinrich Heine University DüsseldorfAndrew Erhart - University of DenverLaura Fontanesi - University of BaselG. Matthew Fricke - University of New MexicoShiguang Fu - Zhejiang University of TechnologyAdriana Galván - University of California, Los AngelesRemi Gau - UCLouvainSarah Genon - Heinrich Heine University DüsseldorfTristan Glatard - Concordia UniversityEnrico Glerean - Aalto UniversityJelle Goeman - Leiden UniversitySergej Golowin - Taipei Medical UniversityCarlos González-García - Ghent UniversityKrzysztof Gorgolewski - Stanford UniversityCheryl Grady - University of TorontoMikella Green - Duke UniversityJoão Guassi Moreira - University of California [Los Angeles]Olivia Guest - University College LondonShabnam Hakimi - Duke UniversityJ. Paul Hamilton - Linköping UniversityRoeland Hancock - University of ConnecticutGiacomo Handjaras - IMT School for Advanced Studies LuccaBronson Harry - The University of SydneyColin Hawco - University of TorontoPeer Herholz - McGill UniversityGabrielle Herman - University of TorontoStephan Heunis - Eindhoven University of TechnologyFelix Hoffstaedter - Heinrich Heine University DüsseldorfJeremy Hogeveen - University of New MexicoSusan Holmes - Stanford UniversityChuan-Peng Hu - Leibniz Institute for NeurobiologyScott Huettel - Duke UniversityMatthew Hughes - Swinburne University of TechnologyVittorio Iacovella - University of TrentoAlexandru Iordan - University of MichiganPeder Isager - Eindhoven University of TechnologyAyse Isik - Max Planck SocietyAndrew Jahn - University of MichiganMatthew Johnson - University of Nebraska–LincolnTom Johnstone - Swinburne University of TechnologyMichael Joseph - University of TorontoAnthony Juliano - Kessler FoundationJoseph Kable - University of PennsylvaniaMichalis Kassinopoulos - McGill UniversityCemal Koba - IMT School for Advanced Studies LuccaXiang-Zhen Kong - Max Planck Institute for Psycholinguistics
- Contributors
- Timothy R Koscik (Contributor) - University of Iowa, Psychiatry
- Resource Type
- Journal article
- Publication Details
- Nature (London), Vol.582(7810), pp.84-88
- DOI
- 10.1038/s41586-020-2314-9
- PMID
- 32483374
- PMCID
- PMC7771346
- NLM abbreviation
- Nature
- ISSN
- 0028-0836
- eISSN
- 1476-4687
- Publisher
- Nature Publishing Group
- Language
- English
- Date published
- 06/04/2020
- Academic Unit
- Psychiatry
- Record Identifier
- 9984280833002771
Metrics
11 Record Views