The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Protecting against researcher bias in secondary data analysis: challenges and potential solutions
Open accessAbstract
Analysis of secondary data sources (such as cohort studies, survey data, and administrative records) has the potential to provide answers to science and society’s most pressing questions. However, researcher biases can lead to questionable research practices in secondary data analysis, which can distort the evidence base. While pre-registration can help to protect against researcher biases, it presents challenges for secondary data analysis. In this article, we describe these challenges and propose novel solutions and alternative approaches. Proposed solutions include approaches to (1) address bias linked to prior knowledge of the data, (2) enable pre-registration of non-hypothesis-driven research, (3) help ensure that pre-registered analyses will be appropriate for the data, and (4) address difficulties arising from reduced analytic flexibility in pre-registration. For each solution, we provide guidance on implementation for researchers and data guardians. The adoption of these practices can help to protect against researcher bias in secondary data analysis, to improve the robustness of research based on existing data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, there is clear evidence from both MR and the natural experiment of an increase in the school leaving age that an increase in the number of years in education has a causal protective effect on health behaviours such as smoking.156–160”
From Mendelian randomization (2022) · cites this paper as [158]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M., Buttrick, N. R., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01749-9 · Cited by 97
RetractedConfirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Lakens, D. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617697177 · Cited by 2,161
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
Equivalence Testing for Psychological Research: A Tutorial
Lakens, D., Scheel, A. M., & Isager, P. M. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918770963 · Cited by 1,560 · cited 1×
“, Bayesian methods [58] or equivalence tests, which determine whether an observed effect is surprisingly small [52, 59, 60].”
MethodPublication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,512
The association between adolescent well-being and digital technology use
Orben, A., & Przybylski, A. K. (2019) · Nature Human Behaviour · doi:10.1038/s41562-018-0506-1 · Cited by 1,500 · cited 4×
“First, researchers now have increasing access to high-dimensional datasets that offer a multitude of ways to analyse the same data [6].”
BackgroundInfluentialNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Problems of reporting genetic associations with complex outcomes
Colhoun, H. M., McKeigue, P., & Smith, G. D. (2003) · The Lancet · doi:10.1016/s0140-6736(03)12715-8 · Cited by 1,203
Variability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097
Rubin, D. B. (2007) · Statistics in Medicine · doi:10.1002/sim.2739 · Cited by 1,047
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042
Duncan, L. E., & Keller, M. C. (2011) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2011.11020191 · Cited by 1,032
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 2×
“While this is a positive outcome for research integrity, null results are less likely to be published [13, 41, 42] and cited [11], which could disadvantage researchers’ careers.”
BackgroundResultBayes factor design analysis: Planning for compelling evidence
Schönbrodt, F. D., & Wagenmakers, E. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1230-y · Cited by 810
Accounting for missing data in statistical analyses: multiple imputation is not always the answer
Hughes, R. A., Heron, J., Sterne, J. A. C., & Tilling, K. (2019) · International Journal of Epidemiology · doi:10.1093/ije/dyz032 · Cited by 802
Schäfer, T., & Schwarz, M. A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00813 · Cited by 774
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. C., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 773
Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials
Mathieu, S. (2009) · JAMA · doi:10.1001/jama.2009.1242 · Cited by 730
Border, R., Johnson, E. C., Evans, L. M., Smolen, A., Berley, N., Sullivan, P. F., & Keller, M. C. (2019) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.18070881 · Cited by 629
Registered Reports: A new publishing initiative at Cortex
Chambers, C. (2012) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 608
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
Using genetic data to strengthen causal inference in observational research
Pingault, J.-B., O’Reilly, P. F., Schoeler, T., Ploubidis, G. B., Rijsdijk, F., & Dudbridge, F. (2018) · Nature Reviews Genetics · doi:10.1038/s41576-018-0020-3 · Cited by 444
Self-citationAn Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M., & Lakens, D. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 417
Why Hypothesis Testers Should Spend Less Time Testing Hypotheses
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620966795 · Cited by 382 · cited 1×
“Furthermore, without a strong theoretical background, hypotheses will be arbitrary and could lead to unhelpful inferences [32, 33], and so should be avoided in novel areas of research.”
BackgroundLikelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Kaplan, R. M., & Irvin, V. L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132382 · Cited by 356
Preregistration Is Hard, And Worthwhile
Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., Veer, A. V. '., & Vazire, S. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.009 · Cited by 326
How scientists fool themselves – and how they can stop
Nuzzo, R. (2015) · Nature · doi:10.1038/526182a · Cited by 296
Gopalakrishna, G., ter Riet, G., Vink, G., Stoop, I., Wicherts, J. M., & Bouter, L. M. (2022) · PLOS ONE · doi:10.1371/journal.pone.0263023 · Cited by 296
Improving inferences about null effects with Bayes factors and equivalence tests.
Lakens, D., McLatchie, N., Isager, P. M., Scheel, A. M., & Dienes, Z. (2020) · The Journals of Gerontology: Series B · doi:10.1093/geronb/gby065 · Cited by 279
Recommendations for Increasing the Transparency of Analysis of Preexisting Data Sets
Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919848684 · Cited by 233
Blind analysis: Hide results to seek the truth
MacCoun, R., & Perlmutter, S. (2015) · Nature · doi:10.1038/526187a · Cited by 218
Invited Commentary: The Need for Cognitive Science in Methodology
Greenland, S. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx259 · Cited by 212
Specification Curve: Descriptive and Inferential Statistics on All Reasonable SpecificationsPreprint
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2694998 · Cited by 200
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021) · Royal Society Open Science · doi:10.1098/rsos.211037 · Cited by 197
Quintana, D. S. (2020) · eLife · doi:10.7554/elife.53275 · Cited by 150
Is Preregistration Worthwhile?
Szollosi, A., Kellen, D., Navarro, D., Shiffrin, R. M., van Rooij, I., Zandt, T. V., & Donkin, C. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.11.009 · Cited by 146
Probing Birth-Order Effects on Narrow Traits Using Specification-Curve Analysis
Rohrer, J. M., Egloff, B., & Schmukle, S. C. (2017) · Psychological Science · doi:10.1177/0956797617723726 · Cited by 145
Giudice, M. D., & Gangestad, S. W. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920954925 · Cited by 140
Preregistration of secondary data analysis: A template and tutorial
van den Akker, O. R., Weston, S. J., Campbell, L., Chopik, B., Damian, R. I., Davis-Kean, P. E., Hall, A., Kosie, J. E., Kruse, E. T., Olsen, J., et al. (2021) · Meta-Psychology · doi:10.15626/mp.2020.2625 · Cited by 137
Bishop, D. (2019) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021819886519 · Cited by 115
de Vries, Y. A., Roest, A. M., de Jonge, P., Cuijpers, P. J., Munafo, M. R., & Bastiaansen, J. A. (2018) · Psychological Medicine · doi:10.1017/s0033291718001873 · Cited by 108
Exploratory reports: A new article type for Cortex
McIntosh, R. D. (2017) · Cortex · doi:10.1016/j.cortex.2017.07.014 · Cited by 90
Dataset decay and the problem of sequential analyses on open datasets
Thompson, W. H., Wright, J., Bissett, P. G., & Poldrack, R. A. (2020) · eLife · doi:10.7554/elife.53498 · Cited by 70
Split-Sample Strategies for Avoiding False Discoveries
Anderson, M. L., & Magruder, J. R. (2017) · National Bureau of Economic Research · doi:10.3386/w23544 · Cited by 65
The past, present, and future of Registered ReportsPreprint
Chambers, C., & Tzavella, L. (2020) · doi:10.31222/osf.io/43298 · Cited by 63
Does preregistration improve the credibility of research findings?
Rubin, M. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.4.p376 · Cited by 61
specr: Conducting and Visualizing Specification Curve Analyses
Masur, P. K., & Scharkow, M. (2020) · doi:10.32614/cran.package.specr · Cited by 58
High Replicability of Newly-Discovered Social-behavioral Findings is AchievablePreprint
Protzko, J., Krosnick, J. A., Nelson, L. D., Nosek, B. A., Axt, J. R., Berent, M. K., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., et al. (2020) · doi:10.31234/osf.io/n2a9x · Cited by 48
Practical Considerations for Navigating Registered Reports
Kiyonaga, A., & Scimeca, J. M. (2019) · Trends in Neurosciences · doi:10.1016/j.tins.2019.07.003 · Cited by 46
Preregistration of secondary data analysis: A template and tutorialPreprint
van den Akker, O. R., Weston, S. J., Campbell, L., et al. (2019) · doi:10.31234/osf.io/hvfmr · Cited by 46
A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses
Seibold, H., Czerny, S., Decke, S., Dieterle, R., Eder, T., Fohr, S., Hahn, N., Hartmann, R., Heindl, C., Kopper, P., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0251194 · Cited by 37
Gopalakrishna, G., ter Riet, G., Vink, G., et al. (2021) · doi:10.31222/osf.io/vk9yt · Cited by 37
Blind Analysis as a Correction for Confirmatory Bias in Physics and in Psychology
MacCoun, R. J., & Perlmutter, S. (2017) · Wiley eBooks · doi:10.1002/9781119095910.ch15 · Cited by 34
Preregistration of Epidemiologic Studies
Vandenbroucke, J. P. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181e942b8 · Cited by 32
Lash, T. L. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181e9bba6 · Cited by 29
Greenland, S. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym174 · Cited by 18
What should a preregistration contain?Preprint
McPhetres, J. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/cj5mh · Cited by 16
Goldstein, B. A. (2020) · Clinical Trials · doi:10.1177/1740774520931211 · Cited by 15
Enabling Confirmatory Secondary Data Analysis by Logging Data Checkout
Scott, K. M., & Struhl, M. K. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918815849 · Cited by 13
Lawlor, D. A. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym168 · Cited by 13
Flagged first, then by citations
Research works printed but not linked: 2
- Small DS, Firth D, Keele L, et al. Protocol for a study of the effect of surface mining in central appalachia on adverse birth outcomes. arXiv.org. 2020
- Deshpande SK, Hasegawa RB, Weiss J, Small DS. Protocol for an observational study on the effects of playing football in adolescence on mental health in early adulthood. arXiv preprint 2018
Cited by
Sorted by citations
Sanderson, E., Glymour, M. M., Holmes, M. V., Kang, H., Morrison, J., Munafò, M. R., Palmer, T., Schooling, C. M., Wallace, C., Zhao, Q., et al. (2022) · Nature Reviews Methods Primers · doi:10.1038/s43586-021-00092-5 · Cited by 2,229
Skrivankova, V. W., Richmond, R. C., Woolf, B. A. R., Davies, N. M., Swanson, S. A., VanderWeele, T. J., Timpson, N. J., Higgins, J. P. T., Dimou, N., Langenberg, C., et al. (2021) · BMJ · doi:10.1136/bmj.n2233 · Cited by 1,648
Guidelines for performing Mendelian randomization investigations
Burgess, S., Davey Smith, G., Davies, N. M., Dudbridge, F., Gill, D., Glymour, M. M., Hartwig, F. P., Holmes, M. V., Minelli, C., Relton, C. L., et al. (2020) · Wellcome Open Research · doi:10.12688/wellcomeopenres.15555.2 · Cited by 1,273
Guidelines for performing Mendelian randomization investigations: update for summer 2023
Burgess, S., Smith, G. D., Davies, N. M., Dudbridge, F., Gill, D., Glymour, M. M., Hartwig, F. P., Kutalik, Z., Holmes, M. V., Minelli, C., et al. (2023) · Wellcome Open Research · doi:10.12688/wellcomeopenres.15555.3 · Cited by 747
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 205
Metadata
- Authors
- Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R.
- Affiliations
- King's College London (GB)
- University College London (GB)
- University of Bristol (GB)
- Corresponding author
- Baldwin, Jessie R.
- Venue
- European Journal of Epidemiology
- Published
- 2022-01
- Volume
- 37
- Issue
- 1
- Pages
- 1-10
- ISSN
- 0393-2990 (ISSN-L)
- 1573-7284
- PMC
- PMC8791887
- OpenAlex
- W4205384179
- Cited by
- 178
- References
- 63
- Citations by year
- 2026: 55 · 2025: 50 · 2024: 31 · 2023: 27 · 2022: 10
- FWCI
- 17.81
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- European Journal of Epidemiology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Explore Bristol Research (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Funders
- Wellcome Trust (215917/Z/19/Z)
- Medical Research Foundation (MRF-160-0002-ELP-PINGA)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- researcher bias, secondary analysis, preregistration, questionable research practices, research robustness
- MeSH terms
- Humans, Surveys and Questionnaires, Cohort Studies, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
