The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration.
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
A growing body of research has suggested that horizontal saccadic eye movements facilitate the retrieval of episodic memories in free recall and recognition memory tasks. Nevertheless, a minority of studies have failed to replicate this effect. This article attempts to resolve the inconsistent results by introducing a novel variant of proponent-skeptic collaboration. The proposed approach combines the features of adversarial collaboration and purely confirmatory preregistered research. Prior to data collection, the adversaries reached consensus on an optimal research design, formulated their expectations, and agreed to submit the findings to an academic journal regardless of the outcome. To increase transparency and secure the purely confirmatory nature of the investigation, the 2 parties set up a publicly available adversarial collaboration agreement that detailed the proposed design and all foreseeable aspects of the data analysis. As anticipated by the skeptics, a series of Bayesian hypothesis tests indicated that horizontal eye movements did not improve free recall performance. The skeptics suggested that the nonreplication may partly reflect the use of suboptimal and questionable research practices in earlier eye movement studies. The proponents countered this suggestion and used a p curve analysis to argue that the effect of horizontal eye movements on explicit memory did not merely reflect selective reporting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 4×
“Although the proposed recommendations vary considerably in focus, they all emphasize the importance of increasing the transparency of scientific communication (Ioannidis, 2005; Koole & Lakens, 2012; Pashler & Harris, 2012; Wagenmakers et al., 2011; Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012).”
InfluentialBayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688 · cited 1×
“This test relies on the default Jeffreys–Zellner–Siow prior setting, the standard choice for model selection in regression models (Liang, Paulo, Molina, Clyde, & Berger, 2008) and in the t test (Rouder et al., 2009; Wagenmakers et al., 2011, 2012).”
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 1×
“…& Savage, 1963; Jef-freys, 1961; Kass & Raftery, 1995; Rouder, Morey, Speckman, & Province, 2012; Rouder, Speckman, Sun, Morey, & Iverson, 2009; Wagenmakers, 2007; Wagenmakers, Lodewyckx, Kuriyal, & Gras-man, 2010; Wagenmakers et al., 2011, 2012; Wetzels, Raaijmakers, Jakab, & Wagenmakers,…”
Self-citationMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Group sequential methods in the design and analysis of clinical trials
POCOCK, S. J. (1977) · Biometrika · doi:10.1093/biomet/64.2.191 · Cited by 1,831 · cited 1×
“…sequential testing is, however, more flexible because—as opposed to its frequentist counterpart—it does not require researchers to specify the total duration of the data collection period (e.g., Reboussin, DeMets, Kim, & Lan, 2000) or the number of interim analyzes in advance (e.g., Pocock, 1977).”
Default Bayes factors for ANOVA designs
Rouder, J. N., Morey, R. D., Speckman, P. L., & Province, J. M. (2012) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2012.08.001 · Cited by 1,800 · cited 1×
“…on hypothesis testing using the Bayes factor (e.g., Berger & Mortera, 1999; Edwards, Lindman, & Savage, 1963; Jef-freys, 1961; Kass & Raftery, 1995; Rouder, Morey, Speckman, & Province, 2012; Rouder, Speckman, Sun, Morey, & Iverson, 2009; Wagenmakers, 2007; Wagenmakers, Lodewyckx, Kuriyal, &…”
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · cited 1×
“The steps involved in the selection of p values that met these selection criteria are documented in the recommended p -curve disclosure table (cf. Simonsohn et al., 2014) available in the supplemental materials at http://osf.io/pXT3M/.”
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Self-citationNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158 · cited 1×
“Or it may be due to experimenter bias (Rosenthal & Rubin, 1978).”
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · cited 6×
“This test relies on the default Jeffreys–Zellner–Siow prior setting, the standard choice for model selection in regression models (Liang, Paulo, Molina, Clyde, & Berger, 2008) and in the t test (Rouder et al., 2009; Wagenmakers et al., 2011, 2012).”
InfluentialSelf-citationConsequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 2×
“That is, the beneficial effect of horizontal eye movements on free recall may seem more established than it actually is, due to publication bias and the file-drawer problem (Greenwald, 1975; Rosenthal, 1979).”
Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests
Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611406923 · Cited by 945
Self-citationRevised standards for statistical evidence
Johnson, V. E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1313476110 · Cited by 808 · cited 2×
“A well-known problem of frequentist hypothesis testing is that p values overstate evidence against H 0 (Berger & Delampady, 1987; Edwards et al., 1963; Johnson, 2013; Sellke, Bayarri, & Berger, 2001).”
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800 · cited 1×
“…disclose prior to data collection which dependent variables they intend to measure and which statistical analyses they intend to conduct (see also Bakker, van Dijk, & Wicherts, 2012; Chambers, 2013; Chambers & Mu-nafo, 2013; de Groot, 1956/2014; Goldacre, 2009; Ioannidis, 2005; Koole & Lakens,…”
Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method
Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R. (2010) · Cognitive Psychology · doi:10.1016/j.cogpsych.2009.12.001 · Cited by 740
Self-citationRegistered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · cited 1×
“…under scrutiny: Direct replications can identify false positives, may facilitate the identification of factors that moderate the (size of the) effect, and can generate skepticism about the existence of seemingly well-established scientific findings (Nosek & Lakens, 2014; Pashler & Harris, 2012).”
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
Behavioral Priming: It’s all in the Mind, but whose Mind?
Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0029081 · Cited by 670 · cited 1×
“At one end of the spectrum, failures to replicate key studies in the psychological literature (e.g., Doyen, Klein, Pichon, & Cleeremans, 2012; Shanks et al., 2013) have prompted hostility and finger-pointing between research groups.”
Berger, J. O., & Delampady, M. (1987) · Statistical Science · doi:10.1214/ss/1177013238 · Cited by 664 · cited 1×
“A well-known problem of frequentist hypothesis testing is that p values overstate evidence against H 0 (Berger & Delampady, 1987; Edwards et al., 1963; Johnson, 2013; Sellke, Bayarri, & Berger, 2001).”
Berger, J. O., & Wolpert, R. L. (1988) · doi:10.1214/lnms/1215466210 · Cited by 616
Roese, N. J., & Vohs, K. D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612454303 · Cited by 457 · cited 1×
“…the probability of the investigators falling prey to various cognitive biases, such as hindsight bias (i.e., judging an event as more predictable after it has occurred; Roese & Vohs, 2012) and confirmation bias (i.e., favoring information that confirms one’s own hypotheses; Nickerson, 1998).”
Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration
Mellers, B., Hertwig, R., & Kahneman, D. (2001) · Psychological Science · doi:10.1111/1467-9280.00350 · Cited by 448 · cited 5×
“We propose the following guidelines for preregistered proponent-skeptic collaborations (see also Hofstee, 1984; Mellers et al., 2001, for suggestions on adversarial collaborations).”
InfluentialPerforming high‐powered studies efficiently with sequential analyses
Lakens, D. (2014) · European Journal of Social Psychology · doi:10.1002/ejsp.2023 · Cited by 414
Testing competing models of loss aversion: an adversarial collaboration
Bateman, I., Kahneman, D., Munro, A., Starmer, C., & Sugden, R. (2005) · Journal of Public Economics · doi:10.1016/j.jpubeco.2004.06.013 · Cited by 208 · cited 1×
“Although the past two decades have witnessed a number of successful adversarial collaborations in various disciplines (e.g., Bateman, Kahneman, Munro, Starmer, & Sugden, 2005; Cadsby, Croson, Marks, & Maynes, 2008; Gilovich, Medvec, & Kahne-man, 1998; Mellers et al., 2001; Schlitz, Wiseman, Watt, &…”
Experiences of collaborative research
Kahneman, D. (2003) · American Psychologist · doi:10.1037/0003-066x.58.9.723 · Cited by 149 · cited 3×
“Here we advocate a hybrid variant of scientific conflict resolution that combines the features of adversarial collaboration (Kahneman, 2003) and preregistered confirmatory research (Wagenmakers et al.”
Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.4.752 · Cited by 143 · cited 2×
“…& Savage, 1963; Jef-freys, 1961; Kass & Raftery, 1995; Rouder, Morey, Speckman, & Province, 2012; Rouder, Speckman, Sun, Morey, & Iverson, 2009; Wagenmakers, 2007; Wagenmakers, Lodewyckx, Kuriyal, & Gras-man, 2010; Wagenmakers et al., 2011, 2012; Wetzels, Raaijmakers, Jakab, & Wagenmakers, 2009).”
Self-citation
Cited by
Bayesian inference for psychology. Part II: Example applications with JASP
Wagenmakers, E.-J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., Gronau, Q. F., Dropmann, D., Boutin, B., et al. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1323-7 · Cited by 1,715
Self-citationVan Dam, N. T., van Vugt, M. K., Vago, D. R., Schmalzl, L., Saron, C. D., Olendzki, A., Meissner, T., Lazar, S. W., Kerr, C. E., Gorchov, J., et al. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617709589 · Cited by 1,506
The JASP guidelines for conducting and reporting a Bayesian analysis
van Doorn, J., van den Bergh, D., Böhm, U., Dablander, F., Derks, K., Draws, T., Etz, A., Evans, N. J., Gronau, Q. F., Haaf, J. M., et al. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01798-5 · Cited by 1,114
Self-citationJASP: Graphical Statistical Software for Common Statistical Designs
Love, J., Selker, R., Marsman, M., Jamil, T., Dropmann, D., Verhagen, J., Ly, A., Gronau, Q. F., Smíra, M., Epskamp, S., et al. (2019) · Journal of Statistical Software · doi:10.18637/jss.v088.i02 · Cited by 1,010
Self-citationSequential hypothesis testing with Bayes factors: Efficiently testing mean differences.
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017) · Psychological Methods · doi:10.1037/met0000061 · Cited by 567
Self-citation
Metadata
- Authors
- Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H. A., van der Molen, M. W., & Wagenmakers, E.-J.
- Venue
- Journal of Experimental Psychology: General
- Published
- 2015-02
- Volume
- 144
- Issue
- 1
- Pages
- e1-e15
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W2047792536
- Cited by
- 114
- References
- 32
- FWCI
- 9.03
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.