Contextual sensitivity in scientific reproducibility
Open accessThe DOI matches its Crossref record; 98 of the 98 references it cites were found in a registry or an index.
Abstract
Significance: Scientific progress requires that findings can be reproduced by other scientists. However, there is widespread debate in psychology (and other fields) about how to interpret failed replications. Many have argued that contextual factors might account for several of these failed replications. We analyzed 100 replication attempts in psychology and found that the extent to which the research topic was likely to be contextually sensitive (varying in time, culture, or location) was associated with replication success. This relationship remained a significant predictor of replication success even after adjusting for characteristics of the original and replication studies that previously had been associated with replication success (e.g., effect size, statistical power). We offer recommendations for psychologists and other scientists interested in reproducibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23 -25).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [24]
Experimental — accuracy not yet measuredIn the citing paper
“These methodologies have several advantages (40, 41) but can be computationally expensive, create a barrier for adoption for behavioral scientists who lack the required technical expertise, and are most likely context-dependent which can undercut replicability (42).”
From Troll and divide: the language of online polarization (2022) · cites this paper as [42]
In the citing paper
“Therefore, many researchers have begun to push for increased focus on reproducible and open practices in science31,53–55.”
From Guidelines for wrist-worn consumer wearable assessment of heart rate in biobehavioral research (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
The Handbook of Social Psychology
Gravatt, A. E., Lindzey, G., & Aronson, F. (1969) · The Family Coordinator · doi:10.2307/582242 · Cited by 12,517
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“Scientists have speculated that a lack of reproducibility in psychology, as well as in other fields, is the result of a wide range of questionable research practices, including a file-drawer problem (21, 22), low statistical power (23–25), researcher’s degrees of freedom (26), presenting post hoc hypotheses as a prior…”
BackgroundEstimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1×
“Lakens D (2016) The statistical conclusions in Gilbert et al (2016) are completely invalid.”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 3×
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
BackgroundResultWEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,076
Principles of topological psychology.
Lewin, K. (1936) · McGraw-Hill eBooks · doi:10.1037/10019-000 · Cited by 3,927
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 3×
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
BackgroundResultGreenstein, F. I., & Polsby, N. W. (1975) · Addison-Wesley Longman Publishing Co., Inc. eBooks · Cited by 2,344
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 5×
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
BackgroundResultInfluentialThe statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Advances in Behavioral Economics
(2004) · Princeton University Press eBooks · doi:10.1515/9781400829118 · Cited by 1,696
A comprehensive review of genetic association studies
Hirschhorn, J. N., Lohmueller, K., Byrne, E., & Hirschhorn, K. (2002) · Genetics in Medicine · doi:10.1097/00125817-200203000-00002 · Cited by 1,674 · cited 2×
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
ResultOne Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
Personality at the crossroads : current issues in interactional psychology
Magnusson, D., & Endler, N. S. (1977) · Lawrence Erlbaum Associates eBooks · Cited by 1,460
Experiencing physical warmth promotes interpersonal warmth
Williams, L. E., & Bargh, J. A. (2008) · Science · doi:10.1126/science.1162548 · Cited by 1,429
Stereotype Susceptibility: Identity Salience and Shifts in Quantitative Performance
Shih, M., Pittinsky, T. L., & Ambady, N. (1999) · Psychological Science · doi:10.1111/1467-9280.00111 · Cited by 1,366
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
The Boundaryless Career: A New Employment Principle for a New Organizational Era
Crocitto, M., Arthur, M. B., & Rousseau, D. M. (1998) · The Academy of Management Review · doi:10.2307/259107 · Cited by 1,233
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Location, location, location: contextualizing organizational research*
Rousseau, D. M., & Fried, Y. (2001) · Journal of Organizational Behavior · doi:10.1002/job.78 · Cited by 1,092
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055 · cited 2×
“Furthermore, they found sample (United States vs. international) and setting (online vs. in-lab) differences for nearly one-third (10/32) of their moderation tests, seven of which were among the largest effects (i.e., anchoring, allowed–forbidden).”
BackgroundIs psychology suffering from a replication crisis? What does “failure to replicate” really mean?
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015) · American Psychologist · doi:10.1037/a0039400 · Cited by 1,005
The Oxford Handbook of Analytical Sociology
(2011) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199215362.001.0001 · Cited by 987
Verbal overshadowing of visual memories: Some things are better left unsaid
Schooler, J. W., & Engstler-Schooler, T. Y. (1990) · Cognitive Psychology · doi:10.1016/0010-0285(90)90003-m · Cited by 794
Purdie-Vaughns, V., Steele, C. M., Davies, P. G., Ditlmann, R., & Crosby, J. R. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.4.615 · Cited by 786
The Replication Recipe: What makes for a convincing replication?
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2013.10.005 · Cited by 705
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697
(2010) · doi:10.1002/9780470561119 · Cited by 649
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630
Stereotype Susceptibility in Children: Effects of Identity Activation on Quantitative Performance
Ambady, N., Shih, M., Kim, A., & Pittinsky, T. L. (2001) · Psychological Science · doi:10.1111/1467-9280.00371 · Cited by 629
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · cited 1×
“Scientists have speculated that a lack of reproducibility in psychology, as well as in other fields, is the result of a wide range of questionable research practices, including a file-drawer problem (21, 22), low statistical power (23–25), researcher’s degrees of freedom (26), presenting post hoc hypotheses as a prior…”
BackgroundThe alleged crisis and the illusion of exact replication
Stroebe, W., & Strack, F. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514450 · Cited by 597
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al-Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 502
Temperature and aggression: Ubiquitous effects of heat on occurrence of human violence.
Anderson, C. A. (1989) · Psychological Bulletin · doi:10.1037//0033-2909.106.1.74 · Cited by 483
Comment on “Estimating the reproducibility of psychological science.”
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016) · Science · doi:10.1126/science.aad7243 · Cited by 474 · cited 1×
“For instance, a recent critique of the Reproducibility Project alleged that several replication studies differed significantly from the original studies, undercutting any inferences about lack of reproducibility in psychology (56).”
BackgroundThe Uses of experiment: studies in the natural sciences
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5743 · Cited by 457
Simmons, J., Nelson, L., & Simonsohn, U. (2011) · PsycEXTRA Dataset · doi:10.1037/e519702015-014 · Cited by 454
The Oxford Handbook of Political Science
(2011) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199604456.001.0001 · Cited by 426
Yong, E. (2012) · Nature · doi:10.1038/485298a · Cited by 405
Social psychological contributions to the decade of the brain: Doctrine of multilevel analysis.
Cacioppo, J. T., & Berntson, G. G. (1992) · American Psychologist · doi:10.1037//0003-066x.47.8.1019 · Cited by 380
The Oxford handbook of political science
(2010) · Choice Reviews Online · doi:10.5860/choice.47-5883 · Cited by 375
Judging near and distant virtue and vice
Eyal, T., Liberman, N., & Trope, Y. (2008) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2008.03.012 · Cited by 373
Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Kaplan, R. M., & Irvin, V. L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132382 · Cited by 356 · cited 1×
“gov registry in 2000, which forced researchers to preregister their methods and outcome measures, the percentage of large heart-disease clinical trials reporting significant positive results plummeted from 57% to a mere 8% (7).”
MethodThe (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · cited 1×
“published data on potential drug targets? Nat Rev Drug Discov 10(9):712.”
BackgroundErrors Are Aversive: Defensive Motivation and the Error-Related Negativity
Hajcak, G., & Foti, D. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02053.x · Cited by 350
Using prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331
Priming, Replication, and the Hardest Science
Cesario, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613513470 · Cited by 320
Colzato, L. S., Bajo, M. T., van den Wildenberg, W., Paolieri, D., Nieuwenhuis, S., La Heij, W., & Hommel, B. (2008) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.34.2.302 · Cited by 300
A Bayesian Perspective on the Reproducibility Project: Psychology
Etz, A., & Vandekerckhove, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149794 · Cited by 295 · cited 1×
“According to a Bayesian reanalysis of the Reproducibility Project, one pair of authors argued that “the apparent discrepancy between the original set of results and the outcome of the Reproducibility Project can be explained adequately by the combination of deleterious publication practices and weak standards of evide…”
Optimal Time of Day and the Magnitude of Age Differences in Memory
May, C. P., Hasher, L., & Stoltzfus, E. R. (1993) · Psychological Science · doi:10.1111/j.1467-9280.1993.tb00573.x · Cited by 293
Advances in Behavioral Economics
(2011) · Princeton University Press eBooks · doi:10.2307/j.ctvcm4j8j · Cited by 282
Rewarding Replications: A Sure and Simple Way to Improve Psychological Science
Koole, S. L., & Lakens, D. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612462586 · Cited by 281
Replication in Psychological Science
Lindsay, D. S. (2015) · Psychological Science · doi:10.1177/0956797615616374 · Cited by 280
Failure to Replicate a Genetic Association May Provide Important Clues About Genetic Architecture
Greene, C. S., Penrod, N. M., Williams, S. M., & Moore, J. H. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005639 · Cited by 266
Lilienfeld, S. O. (2012) · American Psychologist · doi:10.1037/a0023963 · Cited by 245
Alter, A. L., & Oppenheimer, D. M. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02062.x · Cited by 242
The Weirdest People in the World?Preprint
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1601785 · Cited by 240
Voodoo Correlations Are Everywhere—Not Only in Neuroscience
Fiedler, K. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611400237 · Cited by 235
Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not'
Chang, A. C., & Li, P. (2015) · doi:10.2139/ssrn.2669564 · Cited by 217 · cited 4×
“Unsuccessful attempts to replicate findings in genetics (1), pharmacology (2), oncology (3), biology (4), and economics (5) have given credence to previous speculation that most published research findings are false (6).”
BackgroundResultInfluentialIs Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say "Usually Not"
Chang, A. C., & Li, P. (2015) · Finance and Economics Discussion Series · doi:10.17016/feds.2015.083 · Cited by 211
Enactment and the Boundary less Career: Organizing as We Work
Weick, K. E. (1996) · The Boundaryless Career · doi:10.1093/oso/9780195100143.003.0003 · Cited by 196
Response to Comment on “Estimating the reproducibility of psychological science”
Anderson, C. J., Bahník, Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., et al. (2016) · Science · doi:10.1126/science.aad9163 · Cited by 191
Harmon-Jones, E., Harmon-Jones, C., Fearn, M., Sigelman, J. D., & Johnson, P. (2008) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.94.1.1 · Cited by 165
Multidimensional visual statistical learning.
Turk-Browne, N. B., Isola, P. J., Scholl, B. J., & Treat, T. A. (2008) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.34.2.399 · Cited by 163
Over half of psychology studies fail reproducibility test
Baker, M. (2015) · Nature · doi:10.1038/nature.2015.18248 · Cited by 162
Racial "experimenter effects" in experimentation, testing, interviewing, and psychotherapy,.
Sattler, J. M. (1970) · Psychological Bulletin · doi:10.1037/h0028511 · Cited by 157
Low Hopes, High Expectations: Expectancy Effects and the Replicability of Behavioral Experiments
Klein, O., Doyen, S., Leys, C., Magalhães de Saldanha da Gama, P. A., Miller, S., Questienne, L., & Cleeremans, A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463704 · Cited by 144
Personality research: Components of variance attributable to the person and the situation.
Sarason, I. G., Smith, R. E., & Diener, E. (1975) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.32.2.199 · Cited by 136
Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells
Reaves, M. L., Sinha, S., Rabinowitz, J. D., Kruglyak, L., & Redfield, R. J. (2012) · Science · doi:10.1126/science.1219861 · Cited by 125 · cited 2×
“findings or mistaken assumptions? Pers Soc Psychol Bull 7(4):593–595.”
BackgroundResultMany psychology papers fail replication test
Bohannon, J. (2015) · Science · doi:10.1126/science.349.6251.910 · Cited by 122
Gelman, A. (2015) · Journal of Management · doi:10.1177/0149206314525208 · Cited by 115
Contemporary psychology: A tour inside the onion.
Bevan, W. (1991) · American Psychologist · doi:10.1037/0003-066x.46.5.475 · Cited by 110
Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.001 · Cited by 108
Leek, J. T., & Peng, R. D. (2015) · Science · doi:10.1126/science.aaa6146 · Cited by 107
Ferguson, C. (2015) · American Psychologist · doi:10.1037/a0039405 · Cited by 99
A new etiquette for replication.
Kahneman, D. J. (2014) · Cited by 96
Statistical Conclusion Validity: Some Common Threats and Simple Remedies
García-Pérez, M. A. (2012) · Frontiers in Psychology · doi:10.3389/fpsyg.2012.00325 · Cited by 92
The Replication Recipe: What Makes for a Convincing Replication?Preprint
Brandt, M., IJzerman, H., Dijksterhuis, A., et al. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2283856 · Cited by 83
Article Commentary: On the Persistence of Low Power in Psychological Science
Vankov, I., Bowers, J., & Munafò, M. R. (2014) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.885986 · Cited by 82
The ultimate sampling dilemma in experience-based decision making.
Fiedler, K. (2008) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.34.1.186 · Cited by 80
Monin, B., Oppenheimer, D. M., Ferguson, M. J., Carter, T. J., Hassin, R. R., Crisp, R. J., Miles, E., Husnu, S., Schwarz, N., Strack, F., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000202 · Cited by 76
A mouse’s house may ruin experiments
Reardon, S. (2016) · Nature · doi:10.1038/nature.2016.19335 · Cited by 68
Dunn, D. S., & Fisher, D. (2010) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2010.29.8.945 · Cited by 61
The Value Heuristic in Judgments of Relative Frequency
Dai, X., Wertenbroch, K., & Brendl, C. M. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02039.x · Cited by 51
An Additional Future for Psychological Science
McGuire, W. J. (2013) · Perspectives on Psychological Science · doi:10.1177/1745691613491270 · Cited by 39
Introduction to the Special Section on Research Practices
Spellman, B. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465075 · Cited by 24
Government: Plan for ecosystem services
Joppa, L., Boyd, J. W., Duke, C. S., Hampton, S. E., Jackson, S. T., Jacobs, K. L., Kassam, K.-A., Mooney, H. A., Ogden, L. A., Ruckelshaus, M. H., et al. (2016) · Science · doi:10.1126/science.351.6277.1037-a · Cited by 17
Effect of Initial Conditions on Reproducibility of Scientific Research
Djulbegović, B., & Hozo, I. (2014) · Acta Informatica Medica · doi:10.5455/aim.2014.22.156-159 · Cited by 9
Replication Failures in Personality and Social Psychology
Touhey, J. C. (1981) · Personality and Social Psychology Bulletin · doi:10.1177/014616728174012 · Cited by 9
Contemporary psychology: A tour inside the onion.
Bevan, W. (1991) · American Psychologist · doi:10.1037//0003-066x.46.5.475
An action-based model of dissonance: (633982013-122)
Harmon-Jones, E. (2007) · PsycEXTRA Dataset · doi:10.1037/e633982013-122
Cited by
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944
“…to replicate when there have been no empirical tests of whether such moderators are operative (Baumeister & Vohs, 2016; Crisp, Miles, & Husnu, 2014; Gilbert, King, Pettigrew, & Wilson, 2016; Ramscar, Shaoul, & Baayen, 2015; Schwarz & Clore, 2016; Stroebe & Strack, 2014; Van Bavel et al., 2016).”
Transparency and replicability in qualitative research: the case of interviews with elite informants
Aguinis, H., & Solarino, A. M. (2019) · Strategic Management Journal · doi:10.1002/smj.3015 · Cited by 791
Replicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis
Shrout, P. E., & Rodgers, J. L. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011845 · Cited by 725
“…2014), and some have even suggested specific target attention to measurement error (Asendorpf et al. 2013) and to the quality of peer review (Coyne 2016, Nosek & Bar-Anan F o r p e r on a Bar-Anan 2012) and for flexibility regarding the traditional significance level of p < 0.05 (Vazire 2016).”
BackgroundInfluentialConstruct Validation in Social and Personality Research: Current Practice and Recommendations
Flake, J. K., Pek, J., & Hehman, E. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693063 · Cited by 697
Metadata
- Authors
- Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A.
- Affiliations
- Department of Psychology, New York University, New York, NY 10003
- New York University (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2016-05-23
- Volume
- 113
- Issue
- 23
- Pages
- 6454-6459
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4988618
- OpenAlex
- W2397691833
- Cited by
- 398
- References
- 92
- Citations by year
- 2026: 17 · 2025: 23 · 2024: 22 · 2023: 29 · 2022: 47 · 2021: 51 · 2020: 49 · 2019: 48 · 2018: 43 · 2017: 43 · 2016: 26
- FWCI
- 75.91
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- National Science Foundation (1555131)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › General Psychology › Academic and Historical Perspectives in Psychology
- Keywords
- scientific reproducibility, replication success, context sensitivity, replication study, statistical power, degrees of freedom, effect size
- MeSH terms
- Humans, Psychology, Publishing, Research, Science, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.