Meta-research: Evaluation and Improvement of Research Methods and Practices
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
As the scientific enterprise has grown in size and diversity, we need empirical evidence on the research process to test and apply interventions that make it more efficient and its results more reliable. Meta-research is an evolving scientific discipline that aims to evaluate and improve research practices. It includes thematic areas of methods, reporting, reproducibility, evaluation, and incentives (how to do, report, verify, correct, and reward science). Much work is already done in this growing field, but efforts to-date are fragmented. We provide a map of ongoing efforts and discuss plans for connecting the multiple meta-research efforts across science worldwide.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The first is that the validity of these concerns is being assessed scientifically by a global metaresearch program, with results that have been briefly overviewed above (44).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Self-citationPolicy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
Self-citationBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Self-citationRescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Self-citationVan Noorden, R., Maher, B., & Nuzzo, R. (2014) · Nature · doi:10.1038/514550a · Cited by 508
Estimates of the Continuously Publishing Core in the Scientific Workforce
Ioannidis, J. P. A., Boyack, K. W., & Klavans, R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0101698 · Cited by 160 · cited 1×
“Several million new research papers are published annually, and the number of publishing authors in 1996–2011 exceeded, according to one estimate, 15 million [2].”
BackgroundSelf-citationReproducibility of Results in Preclinical Studies: A Perspective From the Bone Field
Manolagas, S. C., & Kronenberg, H. M. (2014) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.2293 · Cited by 44
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
“The field of metascience (or metaresearch) is growing rapidly, with over 2,000 relevant publications accruing annually 16 .”
Self-citationScanning the horizon: towards transparent and reproducible neuroimaging research
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J.-B., Vul, E., & Yarkoni, T. (2017) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.167 · Cited by 1,410
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
Quintana, D. S., Alvares, G. A., & Heathers, J. A. J. (2016) · Translational Psychiatry · doi:10.1038/tp.2016.73 · Cited by 420
“42,43 Given the absence of reporting standards speci fi c to psychiatric research, it is therefore unsurprising that critical study details are inconsis-tently communicated.”
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Self-citationReproducibility of animal research in light of biological variation
Voelkl, B., Altman, N. S., Forsman, A., Forstmeier, W., Gurevitch, J., Jaric, I., Karp, N. A., Kas, M. J., Schielzeth, H., Van de Casteele, T., et al. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0313-3 · Cited by 390
Meta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364
“Nonetheless, concerns that papers reporting false or exaggerated findings might be widespread and growing have inspired an expanding literature of research on research (aka meta-research), which points to a postulated core set of bias patterns and factors that might increase the risk for researchers to engage in biasg…”
BackgroundSelf-citationA standardized citation metrics author database annotated for scientific field
Ioannidis, J. P. A., Baas, J., Klavans, R., & Boyack, K. W. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000384 · Cited by 356
Self-citationMeta-research: Why research on research matters
Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2005468 · Cited by 314
Self-citationHardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
Research Weaving: Visualizing the Future of Research Synthesis
Nakagawa, S., Samarasinghe, G., Haddaway, N. R., Westgate, M. J., O’Dea, R. E., Noble, D. W. A., & Lagisz, M. (2019) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2018.11.007 · Cited by 261
Considering aspects of the 3Rs principles within experimental animal biology
Sneddon, L. U., Halsey, L. G., & Bury, N. R. (2017) · Journal of Experimental Biology · doi:10.1242/jeb.147058 · Cited by 254
“Many authors have discussed how to combat irreproducibility (Freedman et al., 2015; Ioannidis et al., 2015; McNutt, 2014; Nosek et al., 2015; Woolston, 2014; http://validation.scienceexchange.com/ #/reproducibility-initiative).”
BackgroundReproducibility of preclinical animal research improves with heterogeneity of study samples
Voelkl, B., Vogt, L., Sena, E. S., & Würbel, H. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2003693 · Cited by 262
Task-Based Learner Production: A Substantive and Methodological Review
Plonsky, L., & Kim, Y. (2016) · Annual Review of Applied Linguistics · doi:10.1017/s0267190516000015 · Cited by 238
“Ishikawa, T. (2008). The effect of task demands of intentional reasoning on L2 speech performance. The Journal of Asia TEFL, 5, 29–63. Jackson, D. O., & Suethanapornkul, S. (2013). The cognition hypothesis: A synthesis and meta-analysis of research on second language task complexity. Language Learning, 63, 330–367. Ke…”
BackgroundInfluentialThe h-index is no longer an effective correlate of scientific reputation
Koltun, V., & Hafner, D. (2021) · PLOS ONE · doi:10.1371/journal.pone.0253397 · Cited by 200
“Our findings can lead to more effective distribution of resources and thus accelerate scientific discovery [21].”
BackgroundStandardized mean differences cause funnel plot distortion in publication bias assessments
Zwetsloot, P.-P., Van Der Naald, M., Sena, E. S., Howells, D. W., IntHout, J., De Groot, J. A., Chamuleau, S. A., MacLeod, M. R., & Wever, K. E. (2017) · eLife · doi:10.7554/elife.24260 · Cited by 212
Hardwicke, T. E., Wallach, J. D., Kidwell, M. C., Bendixen, T., Crüwell, S., & Ioannidis, J. P. A. (2020) · Royal Society Open Science · doi:10.1098/rsos.190806 · Cited by 217
“A cascade of reform initiatives have been launched in an effort to improve transparency [21,22], and close monitoring and evaluation of specific reform initiatives will help to identify efforts that are the most successful and highlight areas for improvement [25,26].”
BackgroundSelf-citationBouter, L. M., Tijdink, J., Axelsen, N., Martinson, B. C., & ter Riet, G. (2016) · Research Integrity and Peer Review · doi:10.1186/s41073-016-0024-5 · Cited by 211
“Finally, we would like to emphasize that all attempts to fight sloppy science and worse should ideally be accompanied by sound evaluation to assess their effects [25, 26].”
BackgroundGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 233
“Strategies for improvement Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis [46, 53].”
Self-citationHafliðadóttir, S. H., Juhl, C. B., Nielsen, S. M., Henriksen, M., Harris, I. A., Bliddal, H., & Christensen, R. (2021) · Trials · doi:10.1186/s13063-021-05454-8 · Cited by 173
“Only trials included in the latest update of the Cochrane review “Placebo interventions for all clinical conditions” by Hróbjartsson and Gøtzsche [14] were considered eligible, and therefore, no new literature search was performed [15, 16].”
BackgroundShemilt, I., Khan, N., Park, S., & Thomas, J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0315-4 · Cited by 168
“In the context of this evidence base, the current ‘case study’ can be viewed as an ‘n of 1’ study that contributes a single SWAR dataset for potential incorporation into a methodology review on this topic [6, 12].”
MethodYılmaz, K. (2021) · MANAS Sosyal Araştırmalar Dergisi · doi:10.33206/mjss.791537 · Cited by 222
More than 3Rs: The importance of scientific validity for harm-benefit analysis of animal research
Würbel, H. (2017) · Lab Animal · doi:10.1038/laban.1220 · Cited by 152
What incentives increase data sharing in health and medical research? A systematic review
Rowhani-Farid, A., Allen, M. J., & Barnett, A. G. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0028-9 · Cited by 146
CoViD-19 Immunopathology & Immunotherapy
Chiappelli, F., Khakshooy, A., & Greenberg, G. (2020) · Bioinformation · doi:10.6026/97320630016219 · Cited by 132
Exposure-wide epidemiology: revisiting Bradford Hill
Ioannidis, J. P. A. (2015) · Statistics in Medicine · doi:10.1002/sim.6825 · Cited by 133
“I think that we need more empirical meta-research studies [39,40] to continue to monitor evidence on these pivotal issues, because things may change in the future.”
BackgroundSelf-citationThe NIH Open Citation Collection: A public access, broad coverage resource
Hutchins, B. I., Baker, K. L., Davis, M. T., Diwersy, M., Haque, E., Harriman, R. M., Hoppe, T. A., Leicht, S. A., Meyer, P., & Santangelo, G. M. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000385 · Cited by 124
“Meta-research, frequently drawing on the historical citation record, seeks to apply scientific methods to further accelerate research by identifying potential improvements to research practices and organization [4,5].”
BackgroundQuantitative research methods and study quality in learner corpus research
Paquot, M., & Plonsky, L. (2017) · International Journal of Learner Corpus Research · doi:10.1075/ijlcr.3.1.03paq · Cited by 129
Devezer, B., Nardin, L. G., Baumgaertner, B. O., & Buzbas, E. O. (2019) · PLoS ONE · doi:10.1371/journal.pone.0216125 · Cited by 127
“Our work contributes to meta-research [12] by introducing a theoretical framework in which we can observe the effects of parameters of the process of scientific discovery on a variety of indicators of scientific performance.”
BackgroundRath, A., Salamon, V., Peixoto, S., Hivert, V., Laville, M., Segrestin, B., Neugebauer, E. A. M., Eikermann, M., Bertelè, V., Garattini, S. K., et al. (2017) · Trials · doi:10.1186/s13063-017-2287-7 · Cited by 111
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“Meta-research (research-on-research) is a burgeoning discipline that leverages theoretical, observational, and experimental approaches to investigate quality, bias, and efficiency as research unfolds in a complex and evolving scientific ecosystem (Ioannidis et al. 2015, Ioannidis 2018a).”
Self-citationHeterogeneity in meta-analysis: a comprehensive overview
Stogiannis, D., Siannis, F., & Androulakis, E. (2023) · The International Journal of Biostatistics · doi:10.1515/ijb-2022-0070 · Cited by 111
Specific barriers to the conduct of randomised clinical trials on medical devices
Neugebauer, E. A. M., Rath, A., Antoine, S., Eikermann, M., Seidel, D., Koenen, C., Jacobs, E. M. G., Pieper, D., Laville, M., Pitel, S., et al. (2017) · Trials · doi:10.1186/s13063-017-2168-0 · Cited by 103
Did awarding badges increase data sharing inBMJ Open? A randomized controlled trial
Rowhani-Farid, A., Aldcroft, A., & Barnett, A. G. (2020) · Royal Society Open Science · doi:10.1098/rsos.191818 · Cited by 94
“Journal data sharing policies have a crucial role to play in promoting data sharing [6].”
BackgroundMetascience as a Scientific Social Movement
Peterson, D., & Panofsky, A. L. (2023) · Minerva · doi:10.1007/s11024-023-09490-3 · Cited by 119
“For instance, in their manifestos and programmatic articles (Begley and Ioannidis 2014; Ioannidis et al. 2015; Munafò et al. 2017; Nosek & Bar-Anan 2012; Nosek et al. 2012), the word “crisis” hardly appears.”
BackgroundSampling and Generalizability in Lx Research: A Second-Order Synthesis
Plonsky, L. (2023) · Languages · doi:10.3390/languages8010075 · Cited by 100
“Stemming from the literature on synthetic methods in this and related fields (e.g., Aguinis et al. 2023; Ioannidis et al. 2015; Plonsky and Gass 2011), this line of inquiry systematically examines different facets of study quality and other methodological features as a means to provide empirically…”
Maniadis, Z., Tufano, F., & List, J. A. (2017) · The Economic Journal · doi:10.1111/ecoj.12527 · Cited by 105
“In addition to proposing replicability strategies employed in other disciplines, it is important to examine whether it is possible to employ tools from the new discipline of meta-research (Ioannidis et al., 2015) to”
Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology
Nuijten, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017) · Collabra: Psychology · doi:10.1525/collabra.102 · Cited by 79
“Research that aims to do so, such as this paper, adds to a growing body of literature on “meta-science” (Ioannidis, Fanelli, Dunne, & Goodman, 2015; Munafò et al., 2017).”
BackgroundMartínez-Bravo, M. C., Sádaba, C., & Serrano-Puche, J. (2020) · El Profesional de la Informacion · doi:10.3145/epi.2020.jul.28 · Cited by 89
““Meta-research involves taking a panoramic view of science (...) This emphasis on the larger picture is typical of many meta-research investigations” ( Ioannidis et al. , 2015).”
BackgroundFlis, I. (2019) · Theory & Psychology · doi:10.1177/0959354319835322 · Cited by 74
Scanning the Horizon: Towards transparent and reproducible neuroimaging researchPreprint
Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafo, M. R., Nichols, T. E., Poline, J., Vul, E., & Yarkoni, T. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/059188 · Cited by 74
Promote scientific integrity via journal peer review data
Lee, C. J., & Moher, D. (2017) · Science · doi:10.1126/science.aan4141 · Cited by 78
Metapesquisa no campo da política educacional: elementos conceituais e metodológicos
Mainardes, J. (2018) · Educar em Revista · doi:10.1590/0104-4060.59762 · Cited by 71
Lozada-Martínez, I. D., Lozada‐Martinez, L. M., & Fiorillo-Moreno, O. (2024) · The Journal of the Royal College of Physicians of Edinburgh · doi:10.1177/14782715241227991 · Cited by 83
“Unquestionably, even in the meta-research proposed by Ioannidis et al., 6 when examining the quality, reporting, incentives, reproducibility and evaluation of research, this would be a crucial factor to consider.”
BackgroundLeichtmann, B., Nitsch, V., & Mara, M. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.838116 · Cited by 66
“Metascience is thus not to be confused with meta-analysis that summarizes effects in the literature on a certain research question quantitatively, but metascience means the science about the science and “ involves taking a bird ’ s eye view of science ” (Ioannidis et al., 2015).”
BackgroundA study on the prevalence of human values in software engineering publications, 2015 -- 2018
Perera, H., Hussain, W., Whittle, J., Nurwidyantoro, A., Mougouei, D., Shams, R. A., & Oliver, G. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380393 · Cited by 63
“n the call for papers of top-tier SE conferences and journals included in the Journal Citation Reports and the instructions to authors of relevant journals and published works. Also, Ioannidis et al. [18] categorized the meta-research discipline into five main thematic fields corresponding to how to conduct, report, v…”
BackgroundHardwicke, T. E., Mathur, M. B., MacDonald, K. E., Nilsonne, G., Banks, G. C., Kidwell, M. C., Mohr, A. H., Clayton, E., Yoon, E. J., Tessler, M. H., et al. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/39cfb · Cited by 63
A data-driven framework for mapping domains of human neurobiology
Beam, E., Potts, C., Poldrack, R. A., & Etkin, A. (2021) · Nature Neuroscience · doi:10.1038/s41593-021-00948-9 · Cited by 68
“Moreover, they should build on efforts to reduce quantifiable biases that are underway in basic science, 5 biomedical informatics, 46 and computational linguistics.”
Liu, Y., Chen, S., Cheng, H., Yu, M., Ran, X., Mo, A., Tang, Y., & Huang, Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642698 · Cited by 61
“…have different specific topics of interest during RQ co-creation, the higher-level research thinking and skills, as discussed in existing meta-research works [25, 67, 77], can be beneficial in general when shared across users, especially for novice researchers or researchers new to certain fields.”
Thomas Willis Lecture: Is Translational Stroke Research Broken, and if So, How Can We Fix It?
Dirnagl, U. (2016) · Stroke · doi:10.1161/strokeaha.116.013244 · Cited by 61
Resting-state brain and spinal cord networks in humans are functionally integrated
Vahdat, S., Khatibi, A., Lungu, O., Finsterbusch, J., Büchel, C., Cohen‐Adad, J., Marchand‐Pauvert, V., & Doyon, J. (2020) · PLoS Biology · doi:10.1371/journal.pbio.3000789 · Cited by 61
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · JBI Evidence Synthesis · doi:10.11124/jbies-23-00139 · Cited by 63
Self-citationThe Weak Spots in Contemporary Science (and How to Fix Them)
Wicherts, J. M. (2017) · Animals · doi:10.3390/ani7120090 · Cited by 59
“The literature on these problems is very large and growing rapidly [127], and so my review of problems is not exhaustive.”
BackgroundReplication crisis or an opportunity to improve scientific production?
Navarro, M. D. F., Pascual‐Llobell, J., Pascual‐Soler, M., Perezgonzalez, J. D., & Berríos-Riquelme, J. (2020) · European Journal of Education · doi:10.1111/ejed.12417 · Cited by 62
Lozada-Martínez, I. D., Rodado, D. N., Guevara, D. M., Cruz-Soto, H. S., Sanchez-Echeverry, M. P., & Liscano, Y. (2025) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2025.1497280 · Cited by 58
“Meta-research in medical sciences: a tool to reduce the scientific fracture and monitor scientific activity Meta-research is defined as the research of research itself (Ioannidis, 2018; Ioannidis et al., 2015; Ioannidis, 2016).”
InfluentialThe influence of the team in conducting a systematic review
Uttley, L., & Montgomery, P. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0548-x · Cited by 54
“Systematic reviews benefit from team working, and co-production is an essential part of high-quality research synthesis and healthcare decision [1, 2].”
BackgroundSystematic review and meta-analyses of studies analysing instructions to authors from 1987 to 2017.
Malički, M., Jerončić, A., Aalbersberg, I. J., Bouter, L., & ter Riet, G. (2021) · Nature Communications · doi:10.1038/s41467-021-26027-y · Cited by 53
Systematic overview of hepatitis C infection in the Middle East and North Africa
Chaabna, K., Cheema, S., Abraham, A., Alrouh, H., Lowenfels, A. B., Maisonneuve, P., & Mamtani, R. (2018) · World Journal of Gastroenterology · doi:10.3748/wjg.v24.i27.3038 · Cited by 53
Why is Getting Rid of P -Values So Hard? Musings on Science and Statistics
Goodman, S. N. (2019) · The American Statistician · doi:10.1080/00031305.2018.1558111 · Cited by 53
Self-citationPersonal Motivations and Systemic Incentives: Scientists on Questionable Research Practices
Bruton, S. V., Medlin, M. M., Brown, M., & Sacco, D. F. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00182-9 · Cited by 55
“2014; Holm and Hofmann 2018) and research methods (Begley and Ioannidis 2015; Munafò et al.”
MethodPerez‐Brena, N. J., Duncan, J. C., Bámaca‐Colbert, M. Y., & Perez, R. (2022) · Journal of Family Theory & Review · doi:10.1111/jftr.12446 · Cited by 56
Lozada-Martínez, I. D., Hernández-Páez, D. A., Moreno, O. F., Picón‐Jaimes, Y. A., & Bermúdez, V. (2025) · Publications · doi:10.3390/publications13010007 · Cited by 56
“This discipline focuses on five key areas: how to conduct, communicate, verify, evaluate, and reward scientific research (Ioannidis et al., 2015).”
Quantifying research waste in ecology
Purgar, M., Klanjšček, T., & Čulina, A. (2022) · Nature Ecology & Evolution · doi:10.1038/s41559-022-01820-0 · Cited by 54
“We also preformed meta-regressions to obtain mean estimates from the meta-studies (1) with a narrow coverage (degree of generality 1) and those with”
BackgroundMethodResultInfluentialAltered brain activity in unipolar depression unveiled using connectomics
Cash, R. F. H., Müller, V. I., Fitzgerald, P. B., Eickhoff, S. B., & Zalesky, A. (2023) · Nature Mental Health · doi:10.1038/s44220-023-00038-8 · Cited by 56
Mapping the community: use of research evidence in policy and practice
Farley-Ripple, E. N., Oliver, K. A., & Boaz, A. (2020) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00571-2 · Cited by 51
Access to Clinical Trials for Adolescents and Young Adults With Cancer: A Meta-Research Analysis
de Rojas, T., Neven, A., Terada, M., García-Abós, M., Moreno, L., Gaspar, N., & Péron, J. (2019) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkz057 · Cited by 46
“Nonetheless, this arguably subjective selection of tumor types is unlikely to interfere with the aim of this meta-research study, a study of research itself: to describe research practices in order to improve them (43,44).”
BackgroundIrreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy
Flier, J. S. (2016) · Molecular Metabolism · doi:10.1016/j.molmet.2016.11.006 · Cited by 45
Saldanha, I. J., Smith, B. T., Ntzani, E., Jap, J., Balk, E. M., & Lau, J. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1250-y · Cited by 44
Lozada-Martínez, I. D., Hernández-Páez, D. A., Zárate, Y. E. J., & Delgado, P. (2025) · Revista da Associação Médica Brasileira · doi:10.1590/1806-9282.20241612 · Cited by 44
“Some authors have raised concerns about the considerable volume of papers being published in journals within short timeframes, prompting questions about the adequacy of peer-review processes 6 .”
Badges for sharing data and code at Biostatistics: an observational study
Rowhani-Farid, A., & Barnett, A. G. (2018) · F1000Research · doi:10.12688/f1000research.13477.2 · Cited by 43
Reward-driven changes in striatal pathway competition shape evidence evaluation in decision-making
Dunovan, K., Vich, C., Clapp, M., Verstynen, T., & Rubin, J. E. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006998 · Cited by 47
A manifesto for reproducible science.
Munafo, M. R., Nosek, B. A., Bishop, D., Button, K. S., Chambers, C., du Sert, N. P., Simonsohn, U., Wagenmakers, E., Ware, J. J., & Ioannidis, J. P. A. (2024) · American Psychological Association eBooks · doi:10.1037/0000409-041 · Cited by 42
Self-citationTransparent, Open, and Reproducible Prevention Science
Grant, S., Wendt, K. E., Leadbeater, B. J. R., Supplee, L. H., Mayo‐Wilson, E., Gardner, F. E. M., & Bradshaw, C. P. (2022) · Prevention Science · doi:10.1007/s11121-022-01336-w · Cited by 46
Mapping the past, present and future research landscape of paternal effects
Rutkowska, J., Lagisz, M., Bonduriansky, R., & Nakagawa, S. (2020) · BMC Biology · doi:10.1186/s12915-020-00892-3 · Cited by 40
“research) could therefore help to identify gaps, biases and clusters in order to facilitate future investigation of paternal effects [26, 27].”
BackgroundCatalá-López, F., Aleixandre-Benavent, R., Caulley, L., Hutton, B., Tabarés-Seisdedos, R., Moher, D., & Alonso-Arroyo, A. (2020) · Trials · doi:10.1186/s13063-019-3944-9 · Cited by 38
Are Leadership and Management Essential for Good Research? An Interview Study of Genetic Researchers
Antes, A. L., Mart, A., & DuBois, J. M. (2016) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/1556264616668775 · Cited by 45
“Research issues that span all scientific research do not have clear funding or publication outlets (Ioannidis, Fanelli, Dunne, & Goodman, 2015).”
Global trends in disruptive technological change: social and policy implications for education
Moravec, J. W., & Martínez-Bravo, M. C. (2023) · On the Horizon The International Journal of Learning Futures · doi:10.1108/oth-02-2023-0007 · Cited by 33
A theory and methodology to quantify knowledge
Fanelli, D. (2019) · Royal Society Open Science · doi:10.1098/rsos.181055 · Cited by 44
“Growing resources are invested, for example, in ensuring reproducibility [1], but there is little agreement on how reproducibility ought to be predicted, measured and understood in different fields [31,75].”
BackgroundSelf-citationLozada-Martínez, I. D., Ealo-Cardona, C. I., Marrugo-Ortiz, A. C., Picón‐Jaimes, Y. A., Cabrera-Vargas, L. F., & Narvaez‐Rojas, A. R. (2023) · International Journal of Surgery · doi:10.1097/js9.0000000000000422 · Cited by 42
Understanding Replication in a Way That Is True to Science
Haig, B. D. (2021) · Review of General Psychology · doi:10.1177/10892680211046514 · Cited by 37
Guidance to best tools and practices for systematic reviews1
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Journal of Pediatric Rehabilitation Medicine · doi:10.3233/prm-230019 · Cited by 48
“Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis [46, 53].”
Self-citationMonsarrat, P., & Vergnes, J. (2017) · GigaScience · doi:10.1093/gigascience/gix121 · Cited by 36
“We suggest that biomedical researchers should be skilled in meta-research in order to take a bird’s eye view of science [34].”
BackgroundMeta-Research Evidence for Evaluating Therapies
Fuller, J. (2018) · Philosophy of Science · doi:10.1086/699689 · Cited by 40
“This is the field of meta-research” (Ioannidis et al. 2015, 1–2).”
MethodNew Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research
Mertens, W., & Recker, J. (2023) · Technology, work and globalization · doi:10.1007/978-3-031-38719-7_13 · Cited by 36
Jaric, I., Voelkl, B., Clerc, M., Schmid, M. W., Novak, J., Rosso, M., Rufener, R., von Kortzfleisch, V. T., Richter, S. H., Buettner, M., et al. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001837 · Cited by 32
Hendriks, B., Simons, A., & Reinhart, M. (2019) · Minerva · doi:10.1007/s11024-019-09367-4 · Cited by 35
Hidalgo, E. S., & Morell, M. F. (2019) · Humanities and Social Sciences Communications · doi:10.1057/s41599-019-0364-0 · Cited by 35
“And also, from a meta-research perspective (Ioannidis et al., 2015), how such an approach could be applied to research processes.”
BackgroundEvidence appraisal: a scoping review, conceptual framework, and research agenda
Goldstein, A., Venker, E., & Weng, C. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx050 · Cited by 30
Science of science: A multidisciplinary field studying science
Krauß, A. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e36066 · Cited by 31
“…by integrating the subfields mentioned above and others that study science and have remained isolated to date – from methodology of science [30,38,39], economics of science [25 – 28, 40] and A central argument here is that to understand science comprehensively we need to integrate evidence of…”
Zhong, J., Liu, X., Lu, J. A., Yang, J., Zhang, G., Mao, S., Chen, H., Yin, Q., Cen, Q., Jiang, R., et al. (2025) · European Radiology · doi:10.1007/s00330-024-11331-0 · Cited by 32
“This is a meta-research study that focuses on the sample size determination of the published literature on radio-mics models [47, 48].”
Bert, B., Dörendahl, A., Leich, N., Vietze, J., Steinfath, M., Chmielewska, J., Hensel, A., Grune, B., & Schönfelder, G. (2017) · PLoS Biology · doi:10.1371/journal.pbio.2003217 · Cited by 29
Lunny, C., Reid, E. K., Neelakant, T., Chen, A., Zhang, J. H., Shinger, G., Stevens, A., Tasnim, S., Sadeghipouya, S., Adams, S., et al. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1542 · Cited by 31
“This is a meta-research study, which aims to evaluate research practices.(21) We followed systematic review guidance for the searching, study selection, and data extraction stages of our study.”
MethodGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · British Journal of Pharmacology · doi:10.1111/bph.16100 · Cited by 33
“Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis (Ioannidis et al., 2015; Negrini et al., 2021).”
BackgroundSelf-citationAI-driven research in pure mathematics and theoretical physics
He, Y. (2024) · Nature Reviews Physics · doi:10.1038/s42254-024-00740-1 · Cited by 28
“• Meta-mathematics: meta-science has been used to refer to the science of scientific research 89 .”
The Research Reproducibility Crisis and Economics of Science
Maniadis, Z., & Tufano, F. (2017) · The Economic Journal · doi:10.1111/ecoj.12526 · Cited by 31
“On the other hand, by applying (whenever disciplinary differences allow) empirical methods from the emerging discipline of metaresearch (Ioannidis et al., 2015) economists can systematically assess how the field of economics fares in terms of reproducibility.”
The illusion of science in comparative cognitionPreprint
Farrar, B. G., & Ostojić, L. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/hduyx · Cited by 31
Psychology Exceptionalism and the Multiple Discovery of the Replication Crisis
Nelson, N. C., Chung, J., Ichikawa, K., & Malik, M. M. (2021) · Review of General Psychology · doi:10.1177/10892680211046508 · Cited by 27
“Many of the figures who are central to replication discussions, such as John Ioannidis, have championed the idea of a meta discipline devoted to using scientific methods to study other disciplines (Ioannidis et al., 2015). Metascience is a discipline that appears to operate according to what David Ribes and colleagues…”
BackgroundInfluentialPesquisa e metapesquisa sobre comunicação na América Latina
Navarro, R. F. (2019) · Matrizes · doi:10.11606/issn.1982-8160.v13i1p27-48 · Cited by 37
Otterbring, T., & Folwarczny, M. (2024) · Food Quality and Preference · doi:10.1016/j.foodqual.2024.105175 · Cited by 27
Jones, L., Barnett, A., & Vagenas, D. (2024) · medRxiv · doi:10.1371/journal.pone.0299617 · Cited by 23
“Unfortunately, when statistical methods are used poorly, they can provide misleading results, leading to wasted resources and patients receiving ineffective or even harmful treatments [4,5].”
Metadata
- Authors
- Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N.
- Affiliations
- Stanford University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-10-02
- Volume
- 13
- Issue
- 10
- Pages
- e1002264
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4592065
- OpenAlex
- W1767553509
- Cited by
- 495
- References
- 10
- Citations by year
- 2026: 36 · 2025: 48 · 2024: 134 · 2023: 50 · 2022: 37 · 2021: 40 · 2020: 39 · 2019: 28 · 2018: 29 · 2017: 30 · 2016: 22 · 2015: 1
- FWCI
- 24.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Laura and John Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Keywords
- meta-research, research practices, research methodology, reproducibility, research reporting, research evaluation, research incentives, research reliability, research efficiency
- MeSH terms
- Animals, Humans, Meta-Analysis as Topic, Guidelines as Topic, Evidence-Based Medicine, Biomedical Research, Empirical Research, Internationality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.