Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.

A standardized framework to test event-based experiments

Open access
Lepauvre, Alex, Hirschhorn, Rony, Bendtz, Katarina, Mudrik, Liad, Melloni, Lucia · 2024 · Behavior Research Methods

Abstract

The replication crisis in experimental psychology and neuroscience has received much attention recently. This has led to wide acceptance of measures to improve scientific practices, such as preregistration and registered reports. Less effort has been devoted to performing and reporting the results of systematic tests of the functioning of the experimental setup itself. Yet, inaccuracies in the performance of the experimental setup may affect the results of a study, lead to replication failures, and importantly, impede the ability to integrate results across studies. Prompted by challenges we experienced when deploying studies across six laboratories collecting electroencephalography (EEG)/magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and intracranial EEG (iEEG), here we describe a framework for both testing and reporting the performance of the experimental setup. In addition, 100 researchers were surveyed to provide a snapshot of current common practices and community standards concerning testing in published experiments’ setups. Most researchers reported testing their experimental setups. Almost none, however, published the tests performed or their results. Tests were diverse, targeting different aspects of the setup. Through simulations, we clearly demonstrate how even slight inaccuracies can impact the final results. We end with a standardized, open-source, step-by-step protocol for testing (visual) event-related experiments, shared via protocols.io. The protocol aims to provide researchers with a benchmark for future replications and insights into the research quality to help improve the reproducibility of results, accelerate multicenter studies, increase robustness, and enable integration across studies.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3758/s13428-024-02508-y

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 The type and frequency of pre-experimental tests, declared by researchers to have been conducted prior to their last published experiment ( N = 96). Y -axis: percentage of respondents. X -axis: aspect of the experimental environment tested (selected from a list of options; see Supplementary Material 1 ). The terms "event timing" and "event content" are defined in Box 1
Fig. 1 The type and frequency of pre-experimental tests, declared by researchers to have been conducted prior to their last published experiment ( N = 96). Y -axis: percentage of respondents. X -axis: aspect of the experimental environment tested (selected from a list of options; see Supplementary Material 1 ). The terms "event timing" and "event content" are defined in Box 1CC BY · PubMed Central
Fig. 2 The image illustrates the connections between the various components and devices used in an example experiment. Top left: A participant sits in an experimental environment (see Box 1). They face a computer screen displaying a green star. Underneath is an eye-tracking device (black) and a response box. An EEG cap and amplifier are displayed as an example of neural measurement. Top right: The screen connects to the experimental computer which runs the experimental software (see Box 1). Bottom right: An example peripheral device recording neural data, connected to both the experimental computer (top right) and the EEG amplifier (top left). Bottom left: An example peripheral device recording eye-tracking data connected to both the eye-tracking camera (top left) and the experimental computer (top right). Clocks in each box indicate the device’s internal clock. Dashed arrows depict connections between components of independent devices. Solid black arrows represent connections sending triggers from the experimental computer to peripherals used for synchronization between recording devices.
Fig. 2 The image illustrates the connections between the various components and devices used in an example experiment. Top left: A participant sits in an experimental environment (see Box 1). They face a computer screen displaying a green star. Underneath is an eye-tracking device (black) and a response box. An EEG cap and amplifier are displayed as an example of neural measurement. Top right: The screen connects to the experimental computer which runs the experimental software (see Box 1). Bottom right: An example peripheral device recording neural data, connected to both the experimental computer (top right) and the EEG amplifier (top left). Bottom left: An example peripheral device recording eye-tracking data connected to both the eye-tracking camera (top left) and the experimental computer (top right). Clocks in each box indicate the device’s internal clock. Dashed arrows depict connections between components of independent devices. Solid black arrows represent connections sending triggers from the experimental computer to peripherals used for synchronization between recording devices.CC BY · PubMed Central
Fig. 3 Reporting practices of researchers who declared testing their last published study ( N = 96, since four respondents declared not testing their experiments at all). Outer circle: responses to “Did you report about the checks you performed and their results?” Red: respondents declaring not reporting the results of the tests ( N = 80). Light blue: respondents declaring reporting the results of some tests ( N = 15). Blue: one respondent who reported the results of all the performed tests. Inner circle: responses to “If you did not report the checks, why not?” Orange: because it was irrelevant ( N = 3 out of the “some” category, N = 40 out of the “no” category). Grayish-blue: because it wasn’t known where to report the results ( N = 11 [“some” category], N = 26 [“no” category], and 1 reporting the test). Tan: because it was irrelevant and not known where to report the results ( N = 1 [“some” category], N = 14 [“no” category])
Fig. 3 Reporting practices of researchers who declared testing their last published study ( N = 96, since four respondents declared not testing their experiments at all). Outer circle: responses to “Did you report about the checks you performed and their results?” Red: respondents declaring not reporting the results of the tests ( N = 80). Light blue: respondents declaring reporting the results of some tests ( N = 15). Blue: one respondent who reported the results of all the performed tests. Inner circle: responses to “If you did not report the checks, why not?” Orange: because it was irrelevant ( N = 3 out of the “some” category, N = 40 out of the “no” category). Grayish-blue: because it wasn’t known where to report the results ( N = 11 [“some” category], N = 26 [“no” category], and 1 reporting the test). Tan: because it was irrelevant and not known where to report the results ( N = 1 [“some” category], N = 14 [“no” category])CC BY · PubMed Central
Fig. 4 Effect of label shuffle on the P1 (A) and reaction time (B) t-statistic. The heatmap represents the observed t-statistic as a function of the simulated effect size (x-axis) and proportion of trials for which the labels were shuffled (y-axis). The color bar is centered on 1.96. Values below significance are colored in shades of blue, while values above significance are colored in shades of orange.
Fig. 4 Effect of label shuffle on the P1 (A) and reaction time (B) t-statistic. The heatmap represents the observed t-statistic as a function of the simulated effect size (x-axis) and proportion of trials for which the labels were shuffled (y-axis). The color bar is centered on 1.96. Values below significance are colored in shades of blue, while values above significance are colored in shades of orange.CC BY · PubMed Central
Fig. 5 Effects of timing inaccuracies on t-statistic for ERPs (A, B) and reaction time (C, D). A, C 3D plot relating the proportion of jittered trials (x-axis), jitter duration (y-axis), and t-statistic (z-axis) as a function of effect size (θ, color bar) for P1 amplitude and reaction times, respectively. Gray hyperplane depicts significance threshold of t = 1.96. An example at θ = 0.2 for P1 amplitude (B) and reaction time (D). The color bar centered on 1.96, values below significance are colored in shades of blue, while values above significance are colored in shades of orange
Fig. 5 Effects of timing inaccuracies on t-statistic for ERPs (A, B) and reaction time (C, D). A, C 3D plot relating the proportion of jittered trials (x-axis), jitter duration (y-axis), and t-statistic (z-axis) as a function of effect size (θ, color bar) for P1 amplitude and reaction times, respectively. Gray hyperplane depicts significance threshold of t = 1.96. An example at θ = 0.2 for P1 amplitude (B) and reaction time (D). The color bar centered on 1.96, values below significance are colored in shades of blue, while values above significance are colored in shades of orangeCC BY · PubMed Central
Fig. 6 Flowchart of the implementation of the testing framework. (1) First, the experiment should be adjusted to present a photodiode square and record the sounds from a contact microphone measuring the sound produced by keyboard presses. In addition, a response sequence should be planned. (2) Then, a set of pre-run tests should be performed. The first consists of measuring the size of the stimuli in centimeters to compute the size of the observed stimuli in degrees of visual angles. In addition, several trials of the experiment should be run, manually annotating what was presented on the screen in each trial. These manual annotations should then be compared to the log file entries to ensure that the log file accurately records what was actually presented. (3) The experiment should then be run in full while recording the photodiode and microphone signals as well as the log file for offline analysis. (4) Then, the recorded photodiode signal should be compared to the log file to estimate event timing inaccuracies. The microphone signal should be compared to the log file response timestamping to assess the responses’ timestamping inaccuracies. In addition, the logged responses should be compared to the planned response sequence defined in step (1) to ensure that the log file accurately records the pressed buttons. Finally, the experimental design can be tested for correctness based on the log file and photodiode timing information
Fig. 6 Flowchart of the implementation of the testing framework. (1) First, the experiment should be adjusted to present a photodiode square and record the sounds from a contact microphone measuring the sound produced by keyboard presses. In addition, a response sequence should be planned. (2) Then, a set of pre-run tests should be performed. The first consists of measuring the size of the stimuli in centimeters to compute the size of the observed stimuli in degrees of visual angles. In addition, several trials of the experiment should be run, manually annotating what was presented on the screen in each trial. These manual annotations should then be compared to the log file entries to ensure that the log file accurately records what was actually presented. (3) The experiment should then be run in full while recording the photodiode and microphone signals as well as the log file for offline analysis. (4) Then, the recorded photodiode signal should be compared to the log file to estimate event timing inaccuracies. The microphone signal should be compared to the log file response timestamping to assess the responses’ timestamping inaccuracies. In addition, the logged responses should be compared to the planned response sequence defined in step (1) to ensure that the log file accurately records the pressed buttons. Finally, the experimental design can be tested for correctness based on the log file and photodiode timing informationCC BY · PubMed Central
Fig. 7 Depiction of visual presentation adjustment to enable testing of timing with a photodiode. A square at the corner of the screen should be flashed to white simultaneously with the onset of each event of interest and return to black thereafter. In this example, the square flashes to white at the onset of each visual stimulus (a colored star) as well as at the offset of each visual stimulus. Critically, the square should not remain white for the entire duration of the stimulus but only for a brief duration at the onset to enable the detection of transitions between each event of interest
Fig. 7 Depiction of visual presentation adjustment to enable testing of timing with a photodiode. A square at the corner of the screen should be flashed to white simultaneously with the onset of each event of interest and return to black thereafter. In this example, the square flashes to white at the onset of each visual stimulus (a colored star) as well as at the offset of each visual stimulus. Critically, the square should not remain white for the entire duration of the stimulus but only for a brief duration at the onset to enable the detection of transitions between each event of interestCC BY · PubMed Central
Fig. 8 Pipeline to compute the log file timestamping delays using the photodiode. The recorded signal must first be parsed (left panel) to extract the photodiode timestamps. Then, the extracted timestamps can be compared to the log file timestamping by investigating the difference between intervals of successive events as recorded by the photodiode and log file (right panel)
Fig. 8 Pipeline to compute the log file timestamping delays using the photodiode. The recorded signal must first be parsed (left panel) to extract the photodiode timestamps. Then, the extracted timestamps can be compared to the log file timestamping by investigating the difference between intervals of successive events as recorded by the photodiode and log file (right panel)CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

2 retracted

Show all 39 references

Cited by

See all 9

Metadata

Authors
Lepauvre, A., Hirschhorn, R., Bendtz, K., Mudrik, L., & Melloni, L.
Affiliations
  • Radboud University Nijmegen (NL)
  • Tel Aviv University (IL)
  • Boston Children's Hospital (US)
  • Canadian Institute for Advanced Research (CA)
Venue
Behavior Research Methods
Published
2024-09-16
Volume
56
Issue
8
Pages
8852-8868
ISSN
  • 1554-351X (ISSN-L)
  • 1554-3528
OpenAlex
W4402581411
Cited by
9
References
39
Citations by year
2026: 4 · 2025: 4 · 2024: 1
FWCI
3.36
Type
journal article
Language
en
Open access
Licence
Locations
  • Behavior Research Methods (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion, public-domain)
Funders
  • Templeton World Charity Foundation (TWCF0389)
  • Max Planck Institute for Empirical Aesthetics
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
replication crisis, reproducibility, preregistration, EEG/MEG, functional MRI, intracranial EEG, multicenter study, research quality, benchmarking, simulation study, reporting practices
MeSH terms
Electroencephalography, Humans, Magnetic Resonance Imaging, Neurosciences, Psychology, Experimental, Research Design, Reproducibility of Results, Magnetoencephalography

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A standardized framework to test event-based experiments. https://app.cochat.ai/papers/10.3758/s13428-024-02508-y
The paper
Lepauvre, A., Hirschhorn, R., Bendtz, K., Mudrik, L., & Melloni, L. (2024). A standardized framework to test event-based experiments. Behavior Research Methods. https://doi.org/10.3758/s13428-024-02508-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.