APPROACHES TO INCREASE REPLICABILITY IN NEUROSCIENCE – LESSONS LEARNED FROM CONSORTIA, MANY ANALYSTS AND COOPERATIVE DATA COLLECTION

Increasing reliability in clinical neuroimaging: Mega-analytic findings from the ENIGMA Transgender Persons Working Group

Sven Mueller

Ghent University, Belgium 

Abstract: One common problem with clinical MRI research is that sample sizes of tested populations are small. This can be due to either a small prevalence rate or a difficulty to recruit a particular population. Individual neuroimaging studies of transgender persons have rarely achieved samples sizes of larger than 30 individuals (per group). To increase reliability of findings, the ENIGMA Transgender Persons Working Group, a consortium of presently 10 different labs around the world, was created. Here, the analysis of 800 structural brain scans of roughly 200 transgender men, 200 transgender women, 200 cisgender men and 200 cisgender women will be presented. Finally, in the last third of the talk I will touch briefly on a second way to increase reliability in neuroimaging, namely by rescanning the same participants within a session. Data from such a study also in transgender persons will be presented.  

A novel attempt to improve replicability of EEG-Personality associations:  The CoScience Project 

Katharina Paul, Jan Wacker 

Department for Differential Psychology, University Hamburg, Hamburg, Germany 

Abstract: The issues at the centre of the replicability crisis, such as low statistical power and undisclosed flexibility in data analysis, are amplified in research aiming to link individual differences in EEG markers to variations in personality due to between-subjects research designs and high complexity of data processing. The CoScience Team, a collaboration of ten EEG-personality laboratories, employs the principles of cooperative forking paths analysis for the first time, aiming to address this unsatisfactory state of affairs by (1) significantly increasing statistical power through sharing the load of data collection, and by (2) eliminating undisclosed flexibility in data analysis. The latter is achieved through collaborative identification of both the most appropriate and all defensible pre-processing and analysis paths, and documentation of the resulting multiverse of millions of alternative analyses and results. These principles will be illustrated by data on the relationship between resting frontal alpha asymmetry and trait approach motivation. 

EEG ManyPipelines: One dataset many pipelines – how robust are EEG findings?

Elena Cesnaite   

Institute of Psychology, University of Münster, Germany  

Abstract: The challenges of reproducibility and replicability have recently come into spotlight, particularly since Open Science Collaboration published a paper evaluating state of the art in the field (Open Science Collaboration, 2015). Within the field of EEG, rich datasets and complex processing and analysis pipelines intentsify this issue further. This problem has been recognized by multiple recent projects in the field EEGManyLabs, EEGManyPipelines, COBIDAS, and ARTEM-IS, which will be presented in this talk.  

One of the challenges on the roud to reproducible and replicable science is transparency in the scientific record (Clayson et al., 2019, Šoškić et al., 2021). INCF Working Group on ARTEM-IS (Agreed Reporting Template for EEG Methodology - International Standard) aims to develop reporting tools for documenting methodological decisions of EEG studies and pipelines, as well as to engage the scientific community in developing and using such tools. 

ARTEM-IS: Improving Reproducibility and Replicability of EEG Research by Enhancing Transparency 

Anđela Šoškić^1,2,Vanja Ković², Suzy J. Styles³ 

¹Faculty Teacher Education Faculty, University of Belgrade , Belgrade, Serbia 
²Laboratory for Neurocognition and Applied Cognition, University of Belgrade Faculty of Philosophy , Belgrade, Serbia
³Nanyang Technological University, Psychology, Singapore; Nanyang Technological University,    
Centre for Research and Development on Learning (CRADLE), Singapore; A*Star Research Entities, Singapore Institute for Clinical Sciences (SICS), Singapore

Abstract: The challenges of reproducibility and replicability have recently come into  the spotlight, particularly since Open Science Collaboration published a paper evaluating state of the art  in the field (Open Science Collaboration, 2015). Within the field of EEG, rich datasets and complex  processing and analysis pipelines intensify this issue further. This problem has been recognized by  multiple recent projects in the field EEGManyLabs, EEGManyPipelines, COBIDAS, and ARTEM-IS, which  will be presented in this talk.   

One of the challenges on the road to reproducible and replicable science is transparency in the scientific  record (Clayson et al., 2019, Šoškić et al., 2021). INCF Working Group on ARTEM-IS (Agreed Reporting  Template for EEG Methodology - International Standard) aims to develop reporting tools for  documenting methodological decisions of EEG studies and pipelines, as well as to engage the scientific  community in developing and using such tools.