The DOI matches its Crossref record; 75 of the 76 references it cites were found in a registry or an index.
Reproducible and replicable pain research: a critical review
Open accessAbstract
Recently, the degree to which scientific publications provide a reliable source of information has come under intense scrutiny. Reports suggest that a substantial amount of published literature is likely to be biased,28 distorted,27,53 and nonreproducible.7,50 This cuts across basic,10,55 preclinical,1,16 and clinical research.4,50 Although controversial, some have claimed that nonreproducible preclinical research consumes $28 billion/year (USD)16 and that 85% of biomedical research resources are wasted on biased research.35 The response from the scientific and policy community has been to identify common practices that contribute to the problem and develop methods to counteract them.47 This matter is very relevant to the pain field.1 Understanding the causes of nonreproducible and nonreplicable research and its ultimate impact on how we prevent and treat pain should assist pain researchers to improve the reproducibility and replicability of their work. The distinction between reproducibility and replicability is presented in Figure 1 and defined elsewhere.51 This article aims to: (1) outline the drivers of nonreproducible and nonreplicable research using examples from pain sciences and broader research fields; and (2) provide an overview of potential solutions and practices that could improve the reproducibility and replicability of pain research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“we submitted our protocol prior to data collection and remained transparent in all reporting.56”
Citations
Cites
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
Hoffmann, T. C., Glasziou, P. P., Boutron, I., Milne, R., Perera, R., Moher, D., Altman, D. G., Barbour, V., Macdonald, H., Johnston, M., et al. (2014) · BMJ · doi:10.1136/bmj.g1687 · Cited by 10,548
Self-citationWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 5×
“3 Researcher degrees of freedom The term ‘researcher degrees of freedom’ refers to the decisions made by a researcher from project conception to dissemination, all of which may shift the outcome and impact of the project, usually towards false positive discoveries [10].”
BackgroundInfluentialSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Cohen, J. (1992) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep10768783 · Cited by 5,870
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“Reports suggest that a substantial amount of published literature is likely to be biased [28], distorted [27,53], and non-reproducible [7,50].”
BackgroundA manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Guidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437
Scanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055
Biomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860 · cited 2×
“5 prestige) are selected and passed from advisors to trainees, and across peers [68].”
BackgroundDo Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
How Open Science Helps Researchers Succeed
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., et al. (2016) · eLife · doi:10.7554/elife.16800 · Cited by 798
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754
Registered Reports: A Method to Increase the Credibility of Published Results
Nosek, B. A., & Lakens, D. (2014) · Social Psychology · doi:10.1027/1864-9335/a000192 · Cited by 716 · cited 3×
“Registered reports and preregistration do not impede exploratory research, but they facilitate the distinction between exploratory and confirmatory research up front by giving researchers the opportunity to declare the nature of the work [49].”
BackgroundKidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700
Edwards, M. A., & Roy, S. (2017) · Environmental Engineering Science · doi:10.1089/ees.2016.0223 · Cited by 688
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614
Statistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials
Moher, D. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020048013 · Cited by 608
Simera, I., Moher, D., Hirst, A., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-24 · Cited by 590
Promoting Transparency in Social Science Research
Miguel, E. A., Camerer, C. F., Casey, K., Cohen, J., Esterling, K., Gerber, A. S., Glennerster, R., GREEN, D. P., Humphreys, M., Imbens, G. W., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 559
Valle, A., Massaro, D., Castelli, I., Intra, F. S., Lombardi, E., Bracaglia, E. A., & Marchetti, A. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01213 · Cited by 448
Publish or perish: Where are we heading?
Rawat, S., & Meena, S. (2014) · Europe PMC (PubMed Central) · Cited by 389
Boutron, I., Altman, D. G., Hopewell, S., Vera-Badillo, F., Tannock, I., & Ravaud, P. (2014) · Journal of Clinical Oncology · doi:10.1200/jco.2014.56.7503 · Cited by 368
Excess Significance Bias in the Literature on Brain Volume Abnormalities
Ioannidis, J. P. A. (2011) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.28 · Cited by 299
Reproducible Epidemiologic Research
Peng, R. D., Dominici, F., & Zeger, S. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj093 · Cited by 292
The Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review
Morey, R. D., Chambers, C., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., Lewandowsky, S., Morey, C. C., Newman, D. P., Schönbrodt, F. D., et al. (2016) · Royal Society Open Science · doi:10.1098/rsos.150547 · Cited by 291
Perceptual confidence neglects decision-incongruent evidence in the brain
Peters, M. A. K., Thesen, T., Ko, Y., Maniscalco, B., Carlson, C., Davidson, M., Doyle, W., Kuzniecky, R., Devinsky, O., Halgren, E., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0139 · Cited by 265
Adverse Events in Randomized Trials
Ioannidis, J. P. A. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.313 · Cited by 241
Hopewell, S., Ravaud, P., Baron, G., & Boutron, I. (2012) · BMJ · doi:10.1136/bmj.e4178 · Cited by 235
knitr: A Comprehensive Tool for Reproducible Research in R
Xie, Y. (2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315373461-1 · Cited by 227
Quality of reporting in systematic reviews of adverse events: systematic review
Zorzela, L., Golder, S., Liu, Y., Pilkington, K., Hartling, L., Joffe, A., Loke, Y., & Vohra, S. (2014) · BMJ · doi:10.1136/bmj.f7668 · Cited by 217
Low statistical power in biomedical science: a review of three human research domains
Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F., & Munafò, M. R. (2017) · Royal Society Open Science · doi:10.1098/rsos.160254 · Cited by 208
Karran, E. L., McAuley, J. H., Traeger, A. C., Hillier, S., Grabherr, L., Russek, L. N., & Moseley, G. L. (2017) · BMC Medicine · doi:10.1186/s12916-016-0774-4 · Cited by 181
Self-citationTools and techniques for computational reproducibility
Piccolo, S., & Frampton, M. (2016) · GigaScience · doi:10.1186/s13742-016-0135-4 · Cited by 179
Data Authorship as an Incentive to Data Sharing
Bierer, B. E., Crosas, M., & Pierce, H. H. (2017) · New England Journal of Medicine · doi:10.1056/nejmsb1616595 · Cited by 177
A purely confirmatory replication study of structural brain-behavior correlations
Boekel, W., Wagenmakers, E., Belay, L., Verhagen, J., Brown, S. D., & Forstmann, B. U. (2015) · Cortex · doi:10.1016/j.cortex.2014.11.019 · Cited by 175
Registered Reports: Realigning incentives in scientific publishing
Chambers, C., Dienes, Z. P., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015) · Cortex · doi:10.1016/j.cortex.2015.03.022 · Cited by 171
Schroeder, L., Petrou, S., Patel, N. V., Hollowell, J., Puddicombe, D., Redshaw, M., Brocklehurst, P., & on behalf of the Birthplace in England Collaborative Group (2012) · BMJ · doi:10.1136/bmj.e2292 · Cited by 158
Taichman, D. B., Sahni, P., Pinborg, A., Peiperl, L., Laine, C., James, A., Hong, S.-T., Haileamlak, A., Gollogly, L., Godlee, F., et al. (2017) · New England Journal of Medicine · doi:10.1056/nejme1705439 · Cited by 156
Lazarus, C., Haneef, R., Ravaud, P., & Boutron, I. (2015) · BMC Medical Research Methodology · doi:10.1186/s12874-015-0079-x · Cited by 137
Bauchner, H., Golub, R. M., & Fontanarosa, P. (2016) · JAMA · doi:10.1001/jama.2016.2420 · Cited by 136
Watzlawick, R., Rind, J., Sena, E. S., Brommer, B., Zhang, T., Kopp, M. A., Dirnagl, U., Macleod, M. R., Howells, D. W., & Schwab, J. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002468 · Cited by 121
Ensuring transparency and minimization of methodologic bias in preclinical pain research
Andrews, N. A., Latrémolière, A., Basbaum, A. I., Mogil, J. S., Porreca, F., Rice, A. S. C., Woolf, C. J., Currie, G. L., Dworkin, R. H., Eisenach, J. C., et al. (2015) · Pain · doi:10.1097/j.pain.0000000000000458 · Cited by 98
Holden, S., Rathleff, M. S., Jensen, M. B., & Barton, C. J. (2017) · British Journal of Sports Medicine · doi:10.1136/bjsports-2017-097547 · Cited by 92
A statistical definition for reproducibility and replicabilityPreprint
Patil, P., Peng, R. D., & Leek, J. T. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/066803 · Cited by 90
Data Sharing from Clinical Trials — A Research Funder’s Perspective
Kiley, R., Peatfield, T., Hansen, J., & Reddington, F. (2017) · New England Journal of Medicine · doi:10.1056/nejmsb1708278 · Cited by 85 · cited 1×
“The value and ethical obligations of sharing raw data is well accepted by funders, journals, researchers, and consumers [32,71].”
BackgroundNo publication without confirmation
Mogil, J. S., & Macleod, M. R. (2017) · Nature · doi:10.1038/542409a · Cited by 76
Buck, S. (2015) · Science · doi:10.1126/science.aac8041 · Cited by 72
Héroux, M. E., Taylor, J. L., & Gandevia, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144151 · Cited by 62
Spatially-defined motor deficits in people with unilateral complex regional pain syndrome
Reid, E., BRAITHWAITE, F., Wallwork, S. B., Harvie, D. S., Chalmers, J., Spence, C. J., Gallace, A., & Moseley, G. L. (2017) · Cortex · doi:10.1016/j.cortex.2017.06.024 · Cited by 53
Self-citationPerneger, T., & Combescure, C. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.04.003 · Cited by 51
Quality of Pain Intensity Assessment Reporting: ACTTION Systematic Review and Recommendations
Smith, S. M., Hunsinger, M., McKeown, A., Parkhurst, M. L., Allen, R. R., Kopko, S., Lu, Y., Wilson, H., Burke, L. B., Desjardins, P. J., et al. (2015) · Journal of Pain · doi:10.1016/j.jpain.2015.01.004 · Cited by 43
Rice, A. S. C., Morland, R., Huang, W., Currie, G. L., Sena, E. S., & Macleod, M. R. (2013) · Scandinavian Journal of Pain · doi:10.1016/j.sjpain.2013.02.002 · Cited by 42
Reporting of Sample Size Calculations in Analgesic Clinical Trials: ACTTION Systematic Review
McKeown, A., Gewandter, J. S., McDermott, M., Pawlowski, J. R., Poli, J. J., Rothstein, D., Farrar, J. T., Gilron, I., Katz, N. P., Lin, A. H., et al. (2014) · Journal of Pain · doi:10.1016/j.jpain.2014.11.010 · Cited by 37
Smith, S. M., Wang, A. T., Pereira, A. C., Chang, D. R., McKeown, A., Greene, K. A., Rowbotham, M. C., Burke, L. B., COPLAN, P. M., Gilron, I., et al. (2013) · Pain · doi:10.1016/j.pain.2013.08.011 · Cited by 34
Taichman, D. B., Backus, J., Baethge, C., Bauchner, H., de Leeuw, P. W., Drazen, J. M., Fletcher, J., Frizelle, F., Groves, T., Haileamlak, A., et al. (2016) · BMJ · doi:10.1136/bmj.i255 · Cited by 29
Bridging the Data-Sharing Divide — Seeing the Devil in the Details, Not the Other Camp
Rosenbaum, L. R. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1704482 · Cited by 28
Shanahan, D. R., Sousa, I. L. D., & Marshall, D. M. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0044-9 · Cited by 25
Neurosurgical Randomized Controlled Trials—Distance Travelled
Azad, T. D., Veeravagu, A., Mittal, V., Esparza, R., Johnson, E., Ioannidis, J. P. A., & Grant, G. A. (2017) · Neurosurgery · doi:10.1093/neuros/nyx319 · Cited by 25
Sensory experience modifies feature map relationships in visual cortex
Cloherty, S. L., Hughes, N. J., Hietanen, M. A., Bhagavatula, P. S., Goodhill, G. J., & Ibbotson, M. R. (2016) · eLife · doi:10.7554/elife.13911 · Cited by 24
McDonald, L., Schultze, A., Simpson, A., Graham, S., Wasiak, R., & Ramagopalan, S. V. (2017) · F1000Research · doi:10.12688/f1000research.12673.1 · Cited by 21
The case for publishing ‘negative’ clinical trials
Rowbotham, M. C. (2009) · Pain · doi:10.1016/j.pain.2009.09.026 · Cited by 19
Antibiotics for back pain: hope or hype?
McCartney, M. (2013) · BMJ · doi:10.1136/bmj.f3122 · Cited by 15
Registering clinical trials is necessary for ethical, scientific and economic reasons.
Antes, G. (2004) · Europe PMC (PubMed Central) · doi:10.1590/s0042-96862004000500003 · Cited by 11
Herbert, R., & Elkins, M. R. (2017) · Journal of physiotherapy · doi:10.1016/j.jphys.2017.05.011 · Cited by 11
Trial Registration to Prevent Duplicate Publication
Antes, G., & Dickersin, K. (2004) · JAMA · doi:10.1001/jama.291.20.2432-c · Cited by 9
Lack of data sharing in observational studies
McDonald, L., Schultze, A., Simpson, A., Graham, S., Wasiak, R., & Ramagopalan, S. V. (2017) · BMJ · doi:10.1136/bmj.j4866 · Cited by 8
Publishing the best basic and applied pain science: open science and PAIN
Keefe, F. J., Ballantyne, J. C., Blyth, F., Coghill, R. C., Dickenson, A. H., Dionne, C. E., Eccleston, C., Finnerup, N. B., Kuner, R., Seminowicz, D. A., et al. (2018) · Pain · doi:10.1097/j.pain.0000000000001166 · Cited by 6
Robertson, M. (2017) · BMC Biology · doi:10.1186/s12915-017-0394-2 · Cited by 3
Research works printed but not linked: 2
- Gewandter, 2017, Checklist for the preparation and review of pain clinical trial publications, PAIN Rep, 0, 1
- Moseley, 2013, Finding the love between scientists and clinicians—a response to Dr Butler on noijam, BodyinMind
Cited by
Sorted by citations
Vægter, H. B., & Jones, M. D. (2020) · PAIN Reports · doi:10.1097/pr9.0000000000000823 · Cited by 223
Siddall, B., Ram, A., Jones, M. D., Booth, J., Perriman, D., & Summers, S. J. (2022) · Pain · doi:10.1097/j.pain.0000000000002308 · Cited by 174
Talbot, K., Madden, V. J., Jones, S., & Moseley, G. L. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2019.03.033 · Cited by 171
Self-citationWewege, M. A., & Jones, M. D. (2020) · Journal of Pain · doi:10.1016/j.jpain.2020.04.003 · Cited by 133
Karran, E. L., Cashin, A. G., Barker, T., Boyd, M. A., Chiarotto, A., Dewidar, O., Mohabir, V., Petkovic, J., Sharma, S., Tejani, S., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.09.017 · Cited by 96
Self-citation
Metadata
- Authors
- Lee, H., Lamb, S. E., Bagg, M. K., Toomey, E., Cashin, A. G., & Moseley, G. L.
- Affiliations
- University of Oxford (GB)
- UNSW Sydney (AU)
- Ollscoil na Gaillimhe – University of Galway (IE)
- University of South Australia (AU)
- Corresponding author
- Lee, Hopin
- Venue
- Pain
- Published
- 2018-04-23
- Volume
- 159
- Issue
- 9
- Pages
- 1683-1689
- ISSN
- 0304-3959 (ISSN-L)
- 1872-6623
- OpenAlex
- W2802823940
- Cited by
- 150
- References
- 76
- Citations by year
- 2026: 10 · 2025: 24 · 2024: 12 · 2023: 10 · 2022: 24 · 2021: 32 · 2020: 26 · 2019: 9 · 2018: 3
- FWCI
- 24.80
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- cc-by-nc-nd
- Locations
- Pain (publishedVersion)
- PubMed (publishedVersion)
- ARAN (University of Galway Research Repository) (Ollscoil na Gaillimhe – University of Galway) (acceptedVersion, cc-by-nc-nd)
- ARAN (University of Galway Research Repository) (Ollscoil na Gaillimhe – University of Galway) (submittedVersion, cc-by)
- Funders
- National Institute for Health and Care Research (NF-SI-0611-10020)
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Keywords
- reproducibility, replicability, pain research, research bias, research waste, replication crisis, research practices, preclinical research, research integrity
- MeSH terms
- Animals, Humans, Pain, Research Design, Statistics as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review