The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
A standardized framework to test event-based experiments
Open accessAbstract
The replication crisis in experimental psychology and neuroscience has received much attention recently. This has led to wide acceptance of measures to improve scientific practices, such as preregistration and registered reports. Less effort has been devoted to performing and reporting the results of systematic tests of the functioning of the experimental setup itself. Yet, inaccuracies in the performance of the experimental setup may affect the results of a study, lead to replication failures, and importantly, impede the ability to integrate results across studies. Prompted by challenges we experienced when deploying studies across six laboratories collecting electroencephalography (EEG)/magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and intracranial EEG (iEEG), here we describe a framework for both testing and reporting the performance of the experimental setup. In addition, 100 researchers were surveyed to provide a snapshot of current common practices and community standards concerning testing in published experiments’ setups. Most researchers reported testing their experimental setups. Almost none, however, published the tests performed or their results. Tests were diverse, targeting different aspects of the setup. Through simulations, we clearly demonstrate how even slight inaccuracies can impact the final results. We end with a standardized, open-source, step-by-step protocol for testing (visual) event-related experiments, shared via protocols.io. The protocol aims to provide researchers with a benchmark for future replications and insights into the research quality to help improve the reproducibility of results, accelerate multicenter studies, increase robustness, and enable integration across studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig. 3 Reporting practices of researchers who declared testing their last published study ( N = 96, since four respondents declared not testing their experiments at all). Outer circle: responses to “Did you report about the checks you performed and their results?” Red: respondents declaring not reporting the results of the tests ( N = 80). Light blue: respondents declaring reporting the results of some tests ( N = 15). Blue: one respondent who reported the results of all the performed tests. Inner circle: responses to “If you did not report the checks, why not?” Orange: because it was irrelevant ( N = 3 out of the “some” category, N = 40 out of the “no” category). Grayish-blue: because it wasn’t known where to report the results ( N = 11 [“some” category], N = 26 [“no” category], and 1 reporting the test). Tan: because it was irrelevant and not known where to report the results ( N = 1 [“some” category], N = 14 [“no” category])](https://figures.cochat.ai/paper-figures/8148ed69fac143ba8f576a2c2d57bcc7/a-standardized-framework-to-test-event-based-experiments-figure-3.jpg)





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“followed standardized32 and preregistered protocols12 to reduce setup differences and confirmatory bias22”
From Adversarial testing of global neuronal workspace and integrated information theories of consciousness (2025) · cites this paper as [32]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
2 retracted
RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M., Buttrick, N. R., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01749-9 · Cited by 97
RetractedGrave, J., Soares, S. C., Morais, S., Rodrigues, P., & Madeira, N. G. G. F. (2021) · Psychiatry Research · doi:10.1016/j.psychres.2021.114077 · Cited by 2
RetractedBrainard, D. H. (1997) · Spatial Vision · doi:10.1163/156856897x00357 · Cited by 19,702
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“Replication failures may stem from a number of factors (Open Science Collaboration, 2015), for instance, low standards of power calculations (Button et al., 2013; Ioannidis, 2005), the use of questionable statistical methods (e.g., Cumming, 2014; Wicherts et al., 2016), publication biases in favor…”
MethodPower failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“Replication failures may stem from a number of factors (Open Science Collaboration, 2015), for instance, low standards of power calculations (Button et al., 2013; Ioannidis, 2005), the use of questionable statistical methods (e.g., Cumming, 2014; Wicherts et al., 2016), publication biases in favor…”
MethodSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · cited 1×
“…power calculations (Button et al., 2013; Ioannidis, 2005), the use of questionable statistical methods (e.g., Cumming, 2014; Wicherts et al., 2016), publication biases in favor of positive findings (Fanelli, 2012), or scarce description of the methods (Poldrack et al., 2008; Simmons et al., 2011).”
Method(2005) · Cited by 5,977
PsychoPy—Psychophysics software in Python
Peirce, J. W. (2007) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2006.11.017 · Cited by 4,603
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Cited by
Cogitate Consortium, Ferrante, O., Gorska-Klimowska, U., Henin, S., Hirschhorn, R., Khalaf, A., Lepauvre, A., Liu, L., Richter, D., Vidal, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08888-1 · Cited by 136
“…a stringent and comprehensive tests of the theories in humans, (3) predefined large samples to increase statistical power, (4) followed standardized 32 and preregistered protocols 12 to reduce setup differences and confirmatory bias 22 (see sections 1 and 2 in Supplementary Information), and (5)…”
Self-citationConstant, M., Mandal, A., Asanowicz, D., Panek, B., Kotlewska, I., Yamaguchi, M., Gillmeister, H., Kerzel, D., Luque, D., Molinero, S., et al. (2025) · Cortex · doi:10.1016/j.cortex.2025.05.014 · Cited by 6
Seedat, A., Lepauvre, A., Jeschke, J., Górska, U., Armendáriz, M., Bendtz, K., Henin, S. E., Hirschhorn, R., Brown, T., Jensen, E., et al. (2025) · Scientific Data · doi:10.1038/s41597-025-04833-z · Cited by 9
“Before data collection, a procedure similar to that described in Lepauvre et al .”
Self-citationLepauvre, A. (2025) · Radboud University Press eBooks · doi:10.54195/9789465150550
Self-citationStockart, F. (2024) · theses.fr (ABES) · doi:10.70675/c4f62883zfadez4797z9c4ezc049226f96a4 · Cited by 1
Metadata
- Authors
- Lepauvre, A., Hirschhorn, R., Bendtz, K., Mudrik, L., & Melloni, L.
- Affiliations
- Radboud University Nijmegen (NL)
- Tel Aviv University (IL)
- Boston Children's Hospital (US)
- Canadian Institute for Advanced Research (CA)
- Venue
- Behavior Research Methods
- Published
- 2024-09-16
- Volume
- 56
- Issue
- 8
- Pages
- 8852-8868
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC11525435
- OpenAlex
- W4402581411
- Cited by
- 9
- References
- 39
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 1
- FWCI
- 3.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Behavior Research Methods (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion, public-domain)
- Funders
- Templeton World Charity Foundation (TWCF0389)
- Max Planck Institute for Empirical Aesthetics
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- replication crisis, reproducibility, preregistration, EEG/MEG, functional MRI, intracranial EEG, multicenter study, research quality, benchmarking, simulation study, reporting practices
- MeSH terms
- Electroencephalography, Humans, Magnetic Resonance Imaging, Neurosciences, Psychology, Experimental, Research Design, Reproducibility of Results, Magnetoencephalography
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A standardized framework to test event-based experiments. https://app.cochat.ai/papers/10.3758/s13428-024-02508-y
- The paper
- Lepauvre, A., Hirschhorn, R., Bendtz, K., Mudrik, L., & Melloni, L. (2024). A standardized framework to test event-based experiments. Behavior Research Methods. https://doi.org/10.3758/s13428-024-02508-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.