Meta-research: Evaluation and Improvement of Research Methods and Practices
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
As the scientific enterprise has grown in size and diversity, we need empirical evidence on the research process to test and apply interventions that make it more efficient and its results more reliable. Meta-research is an evolving scientific discipline that aims to evaluate and improve research practices. It includes thematic areas of methods, reporting, reproducibility, evaluation, and incentives (how to do, report, verify, correct, and reward science). Much work is already done in this growing field, but efforts to-date are fragmented. We provide a map of ongoing efforts and discuss plans for connecting the multiple meta-research efforts across science worldwide.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The first is that the validity of these concerns is being assessed scientifically by a global metaresearch program, with results that have been briefly overviewed above (44).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Self-citationPolicy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
Self-citationBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Self-citationRescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Self-citationVan Noorden, R., Maher, B., & Nuzzo, R. (2014) · Nature · doi:10.1038/514550a · Cited by 508
Estimates of the Continuously Publishing Core in the Scientific Workforce
Ioannidis, J. P. A., Boyack, K. W., & Klavans, R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0101698 · Cited by 160 · cited 1×
“Several million new research papers are published annually, and the number of publishing authors in 1996–2011 exceeded, according to one estimate, 15 million [2].”
BackgroundSelf-citationReproducibility of Results in Preclinical Studies: A Perspective From the Bone Field
Manolagas, S. C., & Kronenberg, H. M. (2014) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.2293 · Cited by 44
Cited by
Items 101–200 of 545
Technological advances in preclinical meta-research
Bannach‐Brown, A., Hair, K., Bahor, Z., Soliman, N., Macleod, M. R., & Liao, J. (2021) · BMJ Open Science · doi:10.1136/bmjos-2020-100131 · Cited by 28
Güneş, A., & Kaban, A. L. (2025) · Higher Education Quarterly · doi:10.1111/hequ.70057 · Cited by 25
Ortloff, A.-M., Martius, F., Meier, M., Sans-Raimbault, T., Geierhaas, L., & Smith, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713671 · Cited by 30
“Meta research is the study of research, which encompasses investigating research methods, reporting practice, reproducibility of research, evaluation methods, and how to incentivize good science [59].”
Zhong, J., Xing, Y., Lu, J. A., Zhang, G., Mao, S., Chen, H., Yin, Q., Cen, Q., Jiang, R., Hu, Y., et al. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02117-x · Cited by 28
“Our study is a meta-research study, and corresponding reporting guideline is currently not available [36–41].”
MethodValuing Healthcare Improvement: Implicit Norms, Explicit Normativity, and Human Agency
Carter, S. M. (2017) · Health Care Analysis · doi:10.1007/s10728-017-0350-x · Cited by 25
Opening science: The rebirth of a scholarly journal
Waltman, L., Larivière, V., Milojević, S., & Sugimoto, C. R. (2020) · Quantitative Science Studies · doi:10.1162/qss_e_00025 · Cited by 24
Guidance to Best Tools and Practices for Systematic Reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · JBJS Reviews · doi:10.2106/jbjs.rvw.23.00077 · Cited by 28
Self-citationGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · BMC Infectious Diseases · doi:10.1186/s12879-023-08304-x · Cited by 27
“Strategies for improvement Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis [46, 53].”
Self-citationFive aspects of research waste in biomedicine: A scoping review
Rosengaard, L. O., Andersen, M. Z., Rosenberg, J., & Fonnes, S. (2024) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12616 · Cited by 28
How our approaches to assessing benefits and harms can be improved
Sena, E., & Currie, G. (2019) · Animal Welfare · doi:10.7120/09627286.28.1.107 · Cited by 23
“This approach applied to assess methods, reporting, evaluation, reproducibility and incentives, allows us to reach a rigorous understanding of what makes research reliable, and how it can be most effectively improved (Ioannidis et al 2015).”
MethodCountering medical nihilism by reconnecting facts and values
Upshur, R. E. G., & Goldenberg, M. J. (2020) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2020.08.005 · Cited by 25
Reckoning With Our Crisis: An Agenda for the Field of Social and Personality Psychology
Schiavone, S. R., & Vazire, S. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221101060 · Cited by 25
“Below, we review some of the most relevant tools and approaches that can be used to examine the state of social and personality psychology (for a review of the broader need and value of metascience in the sciences, see Fidler & Wilcox, 2021; Hardwicke et al., 2020; Ioannidis et al., 2015).”
MethodResearch Misconduct as White-Collar Crime
Faria, R. (2018) · Springer eBooks · doi:10.1007/978-3-319-73435-4 · Cited by 22
Prolific non-research authors in high impact scientific journals: meta-research study
Ioannidis, J. P. A. (2023) · Scientometrics · doi:10.1007/s11192-023-04687-5 · Cited by 22
“This would require better training of science writers in rigorous meta-research methods (Ioannidis et al., 2015) and/or pairing with meta-researchers in co-authorship.”
BackgroundMethodSelf-citationPapaefthimiou, M., Kontou, P. I., Bagos, P. G., & Braliou, G. G. (2024) · Antioxidants · doi:10.3390/antiox13060692 · Cited by 24
“Metascience has been developed as a response to the “replication crisis” as part of a growing awareness of the problem [72,73].”
BackgroundClinical research in cancer palliative care: a metaresearch analysis
Vinches, M., Neven, A., Fenwarth, L., Terada, M., Rossi, G., Kelly, S. M., Péron, J., Thomaso, M., Grøenvold, M., & de Rojas, T. (2020) · BMJ Supportive & Palliative Care · doi:10.1136/bmjspcare-2019-002086 · Cited by 21
Santo, T. D., Rice, D. B., Amiri, L. S. N., Tasleem, A., Li, K., Boruff, J. T., Geoffroy, M., Benedetti, A., & Thombs, B. D. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.05.017 · Cited by 21
What meta‐research has taught us about research and changes to research practices
Ioannidis, J. P. A. (2024) · Journal of Economic Surveys · doi:10.1111/joes.12666 · Cited by 25
Self-citationPast, Present, and Future of Citation Practices in HCI
Oppenlaender, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713556 · Cited by 20
“This is commonly referred to as meta-research [15].”
Pesquisa participativa reconectando diversidade: democracia de saberes para a sustentabilidade
Giatti, L. L., Gutberlet, J., de Toledo, R. F., & dos Santos, F. N. P. (2021) · Estudos Avançados · doi:10.1590/s0103-4014.2021.35103.013 · Cited by 20
Educating for Responsible Research Practice in Biomedical Sciences: Towards Learning Goals
Gerrits, E. M., Bredenoord, A. L., & van Mil, M. H. W. (2021) · Science & Education · doi:10.1007/s11191-021-00295-y · Cited by 23
Data sharing platforms: instruments to inform and shape science policy on data sharing?
Devriendt, T., Shabani, M., Lekadir, K., & Borry, P. (2022) · Scientometrics · doi:10.1007/s11192-022-04361-2 · Cited by 20
“Meta-research (“research-on-research”) engages in studying the process of knowledge production itself through the use of scientific methodology (Ioannidis et al., 2015).”
BackgroundFraud, individuals, and networks: A biopsychosocial model of scientific frauds
Leistedt, S., & Linkowski, P. (2016) · Science & Justice · doi:10.1016/j.scijus.2016.01.002 · Cited by 20
“This ismuchharder to prove due to a lack of consistency in defining “authorship” or “substantial contribution” [14,15].”
BackgroundThe design, experiment, analyse, and reproduce principle for experimentation in virtual reality
Grübel, J. (2023) · Frontiers in Virtual Reality · doi:10.3389/frvir.2023.1069423 · Cited by 20
“Reproducibility has become the Achilles ’ heel of modern science, with several experiments not being able to continuously produce the same results (Ioannidis et al., 2015; Camerer et al., 2018) and consequently putting the claims made based on the experiments performed on shaky grounds…”
BackgroundTheory and methods of the multiverse: an application for panel-based models
Cantone, G. G., & Tomaselli, V. (2023) · Quality & Quantity · doi:10.1007/s11135-023-01698-5 · Cited by 21
“If statistically significant results are over-published, there are incentives to employ weak methods to fish for false (but significant) discoveries in data analysis (Ioannidis et al. 2015).”
Bradshaw, A. C., Hughes, N., García, D. V., Chokoshvili, D., Owens, A. P., Hansen, C., Emmert, K., Maetzler, W., Killin, L. O. J., Barnes, R., et al. (2023) · Frontiers in Neurology · doi:10.3389/fneur.2023.1187095 · Cited by 18
“Data sharing is the process of making data available to people other than the data generators, collectors, custodians or stewards, forming a cornerstone of Open Science, wherein data is easily accessible, comprehensible, reproducible, replicable, and verifiable (1).”
BackgroundMethodInfluentialOpen Access, Scholarly Communication, and Open Science in Psychology: An Overview for Researchers
Mullen, L. B. (2024) · SAGE Open · doi:10.1177/21582440231205390 · Cited by 23
Ng, J. Y., Lee, M. S., Liu, J., Steel, A., Wieland, L. S., Witt, C. M., Moher, D., & Cramer, H. (2024) · Integrative Medicine Research · doi:10.1016/j.imr.2024.101068 · Cited by 18
“14 , 16”
BackgroundInfluentialHow failure to falsify in high-volume science contributes to the replication crisis
Rajtmajer, S. M., Errington, T. M., & Hillary, F. G. (2022) · eLife · doi:10.7554/elife.78830 · Cited by 25
“Metascience uses the tools of science to describe and evaluate science on a macro scale and to motivate reforms in scientific practice (Munafò et al., 2017; Ioannidis et al., 2015; Gurevitch et al., 2018).”
MethodUnderstanding the importance of quality control and quality assurance in preclinical PET/CT imaging
McDougald, W., & Mannheim, J. G. (2022) · EJNMMI Physics · doi:10.1186/s40658-022-00503-w · Cited by 22
“estimated that 85% of invested effort and resources in biomedical research were wasted due to a variety of diverse inefficiencies [66].”
BackgroundShould Psychology Journals Adopt Specialized Statistical Review?
Hardwicke, T. E., Frank, M. C., Vazire, S., & Goodman, S. N. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919858428 · Cited by 23
Self-citationMeta-Research: Broadening the Scope of PLOS Biology
Kousta, S. T., Ferguson, C., & Ganley, E. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002334 · Cited by 18
“The emerging field of meta-research aims to characterize existing standards and identify improved practices, in the hope of raising awareness and ultimately improving the quality and reliability of scientific research [7].”
BackgroundData collection in the global south: practical, methodological, and philosophical considerations
Brown, S., Saxena, D. K., & Wall, P. J. (2023) · Information Technology for Development · doi:10.1080/02681102.2023.2185581 · Cited by 22
“We view this work as an example of meta-research (Hardwicke et al., 2020; Ioannidis et al., 2015) in which the authors take a second look at their completed studies to evaluate and improve research practices.”
Lozada-Martínez, I. D., Hernández-Páez, D. A., Velasco, I. P., Guevara, D. M., & Liscano, Y. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13185441 · Cited by 20
“In general, meta-research studies that evaluate the methodological quality, reporting, assessment, reproducibility, and incentives [27] ensuring the falsifiability of surgical evidence in older persons are scarce.”
BackgroundEvaluating solutions to the problem of false positives
Gall, T., & Maniadis, Z. (2018) · Research Policy · doi:10.1016/j.respol.2017.12.005 · Cited by 17
Kulo, V. A., & Cestone, C. M. (2023) · Medical Science Educator · doi:10.1007/s40670-023-01893-x · Cited by 18
Chapter 4. Psychoneuroendocrinology
Sforzini, L., Weston, F. I., & Pariante, C. M. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as04 · Cited by 16
Replicability Crisis and Scientific Reforms: Overlooked Issues and Unmet Challenges
Andreoletti, M. (2020) · International Studies in the Philosophy of Science · doi:10.1080/02698595.2021.1943292 · Cited by 17
“This has led to the explosion of meta-research, a ‘science of science’ that promises to improve scientific progress and discovery, by focusing on evaluating and eventually improving research practices (Ioannidis et al. 2015).”
BackgroundCostas, R., Corona-Sobrino, C., & Robinson‐García, N. (2024) · Edward Elgar Publishing eBooks · doi:10.4337/9781839105722.00026 · Cited by 17
A categorization scheme for software engineering conference papers and its application
Bertolino, A., Calabrò, A., Lonetti, F., Marchetti, E., & Miranda, B. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.11.048 · Cited by 17
Is “the time ripe” for quantitative research on misconduct in science?
Zuckerman, H. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00065 · Cited by 21
Dirnagl, U. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04467-5 · Cited by 17
Meta-research in pharmacy: Time for a look in the mirror
Harpe, S. E. (2021) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2021.04.006 · Cited by 15
Many nonnormalities, one simulation: Do different data generation algorithms affect study results?
Fairchild, A. J., Yin, Y., Baraldi, A. N., Astivia, O. L. O., & Shi, D. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02364-w · Cited by 18
D’Isanto, T., Esposito, G., Altavilla, G., D’Elia, F., & Raiola, G. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1176632 · Cited by 16
Envisioning data sharing for the biocomputing community
Riccardi, E., Pantano, S., & Potestio, R. (2019) · Interface Focus · doi:10.1098/rsfs.2019.0005 · Cited by 17
Lianou, D. T., & Fthenakis, G. C. (2020) · Pathogens · doi:10.3390/pathogens9070585 · Cited by 15
“Meta-science can be defined as the use of research methods to study and appraise research itself and the areas where improvements can be made; meta-science studies can deal with methods, reporting, reproducibility, evaluation, and incentives, which are related with performance, communication, verification, evaluation,…”
BackgroundThe value of bioethical research: A qualitative literature analysis of researchers’ statements
Mertz, M., Fischer, T., & Salloch, S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0220438 · Cited by 16
Meta‐research publications in dentistry: a review
Faggion, C. M., Listl, S., & Smits, K. P. J. (2021) · European Journal Of Oral Sciences · doi:10.1111/eos.12748 · Cited by 14
Advancing Meta‐Analytic Methods in L2 Research
Plonsky, L., Hu, Y., Sudina, E., & Oswald, F. L. (2023) · Wiley eBooks · doi:10.1002/9781394259670.ch14 · Cited by 20
A framework for open policy analysis
de la Guardia, F. H., Grant, S., & Miguel, E. A. (2020) · Science and Public Policy · doi:10.1093/scipol/scaa067 · Cited by 19
Introduction to the special issue on Introducing bibliometrics in applied linguistics
Plonsky, L. (2023) · Studies in Second Language Learning and Teaching · doi:10.14746/ssllt.40213 · Cited by 14
Oyserman, D. (2023) · Psychological Bulletin · doi:10.1037/bul0000376 · Cited by 14
Meta-HCI: First Workshop on Meta-Research in HCI
Oppenlaender, J., Malacria, S., Fang, X., van Berkel, N., Chevalier, F., Yatani, K., & Hosio, S. J. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706723 · Cited by 15
“Self-reflective [85] and meta-scientific contributions [30, 47] are critically important to advance the HCI field as a whole.”
Against Dataism and for Data Sharing of Big Biomedical and Clinical Data with Research Parasites
Emmert‐Streib, F., Dehmer, M., & Yli-Harja, O. P. (2016) · Frontiers in Genetics · doi:10.3389/fgene.2016.00154 · Cited by 13
Valdez, D., Vorland, C. J., Brown, A. W., Mayo-Wilson, E., Otten, J. M., Ball, R. J., Grant, S., Levy, R., Svetina, D., & Allison, D. B. (2020) · F1000Research · doi:10.12688/f1000research.26594.1 · Cited by 16
“The science concerning these norms is a specific branch of meta-science, or “research on research”, led by scientists who promote these values through the education of early career scientists, identifying areas of concern for scientific validity, and postulating paths toward stronger, more credible science (Ioannidis…”
BackgroundBarnett, A. G., & Glasziou, P. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053377 · Cited by 14
Experiments as Code and its application to VR studies in human-building interaction
Aguilar, L., Gath-Morad, M., Grübel, J., Ermatinger, J., Zhao, H., Wehrli, S., Sumner, R. W., Zhang, C., Helbing, D., & Hölscher, C. (2024) · Scientific Reports · doi:10.1038/s41598-024-60791-3 · Cited by 20
“The relatively new field of meta-science (Ioannidis et al., 2015) and initiatives across all sciences (Stojnic et al., 2021; Swiss Data Science Center, 2021; Aguilar et al., 2021; Orchard et al., 2003; Welter et al., 2014) aim to improve experimental research (Baum-gaertner et al., 2018) and enable…”
Carrasco‐Campos, Á., Lapiedra, E. S., & Nicolás, M. M. (2018) · adComunica revista científica de estrategias tendencias e innovación en comunicación · doi:10.6035/2174-0992.2018.15.4 · Cited by 13
Cragg, J. J., Kramer, J. L. K., Borisoff, J., Patrick, D. M., & Ramer, M. S. (2018) · European Journal of Clinical Investigation · doi:10.1111/eci.13045 · Cited by 13
Methods used to assess outcome consistency in clinical studies: A literature-based evaluation
Rogozińska, E., Gargon, E., Olmedo‐Requena, R., Asour, A., Cooper, N. A. M., Vale, C. L., & Hooft, J. V. ’. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235485 · Cited by 14
“In recent years, there has been a growing body of literature focusing on the methodological aspects of medical research [29].”
BackgroundMethodGlobal environmental change mediated response of wetland plants: Evidence from past decades
Sati, P., Chandola, V., Chandra, S., Trivedi, V. L., Purohit, V. K., & Nautiyal, M. C. (2025) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2025.178668 · Cited by 12
Lawson, D. O., Wang, M. K., Kim, K. S., Eikelboom, R., Rodrigues, M., Trapsa, D., Thabane, L., & Moher, D. (2022) · Trials · doi:10.1186/s13063-022-06624-y · Cited by 12
“Meta-science—also called research-on-research, metaresearch, and many other names—has been described as the scientific discipline that seeks to evaluate the practice of research [43].”
BackgroundSports Metaresearch: An Emerging Discipline of Sport Science and Medicine
Warmenhoven, J., Menaspà, P., Borg, D. N., Vazire, S., White, N. M., Sainani, K. L., Nimphius, S., Coutts, A. J., & Impellizzeri, F. M. (2025) · Sports Medicine · doi:10.1007/s40279-025-02181-x · Cited by 11
“Table 1 The five major areas of metaresearch described by Ioannides et al. [32], with a non-exhaustive list of contemporary examples from the sport science or sports medicine literature”
Meta-research studies should improve and evaluate their own data sharing practices
Cristea, I. A., Naudet, F., & Caquelin, L. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.05.007 · Cited by 11
Guidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.14295 · Cited by 11
“Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis (Ioannidis et al., 2015; Negrini et al., 2021).”
BackgroundSelf-citationDevelopment of science and education in the Western Balkan countries: competitiveness with the EU
Živković, Ž., & Panić, M. (2020) · Scientometrics · doi:10.1007/s11192-020-03554-x · Cited by 11
Introducing our series: research synthesis and meta-research in biology
Nakagawa, S., Koricheva, J., Macleod, M. R., & Viechtbauer, W. (2020) · BMC Biology · doi:10.1186/s12915-020-0755-0 · Cited by 12
Open Science in Applied Linguistics: An Introduction to MetasciencePreprint
Al‐Hoorie, A. H., & Hiver, P. V. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/76dak · Cited by 12
Analyzing Cyber Security Research Practices through a Meta-Research Framework
Pochat, V. L., & Joosen, W. (2023) · Cyber Security Experimentation and Test Workshop (CSET) · doi:10.1145/3607505.3607523 · Cited by 10
Zhong, J., Xing, Y., Hu, Y., Lu, J. A., Yang, J., Zhang, G., Mao, S., Chen, H., Yin, Q., Cen, Q., et al. (2024) · Insights into Imaging · doi:10.1186/s13244-024-01769-7 · Cited by 11
“We performed a cross-sectional meta-research study [40 – 44].”
MethodIoannidis, J. P. A. (2019) · Journal of Pharmacokinetics and Pharmacodynamics · doi:10.1007/s10928-019-09621-y · Cited by 10
Self-citationReproducibility failure in biomedical research: problems and solutions
Haven, T. L., & Ioannidis, J. P. A. (2026) · Annual Review of Medicine · doi:10.1146/annurev-med-050124-050859 · Cited by 9
Self-citationReaping the Bounty of Publicly Available Clinical Trial Consent Forms.
Lynch, H. F., Largent, E. A., & Zarin, D. A. (2017) · PubMed · Cited by 9
Zwetsloot, P.-P. M., Kouwenberg, L. H. J. A., Sena, E. S., Eding, J., den Ruijter, H. M., Sluijter, J. P. G., Pasterkamp, G., Doevendans, P. A., Hoefer, I. E., Chamuleau, S. A. J., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-14294-z · Cited by 10
Meta-research: bird’s eye views of primary care research
Tatsioni, A., & Ioannidis, J. P. A. (2020) · Family Practice · doi:10.1093/fampra/cmaa025 · Cited by 10
Self-citationWambier, L. M., Gonçalves, A. D. R., Wambier, D. S., Reis, A., & Chibinski, A. C. R. (2022) · Brazilian Oral Research · doi:10.1590/1807-3107bor-2022.vol36.0017 · Cited by 8
Daraio, C. C., Leo, S. D., & Ferazzoli, F. (2025) · International Transactions in Operational Research · doi:10.1111/itor.70054 · Cited by 9
Zyoud, S. H. (2024) · Regional Environmental Change · doi:10.1007/s10113-024-02246-z · Cited by 9
“…a source of several meta-research studies, a growing scientific discipline intended to assess and improve research procedures (Bartoš et al. 2024; Ioannidis et al. 2015; Raynaud et al. 2024), and in assessing research trends across numerous scientific disciplines (Fu et al. 2023; Zyoud 2022,…”
Arguments against efficiency in science
Peterson, D., & Panofsky, A. L. (2021) · Social Science Information · doi:10.1177/05390184211021383 · Cited by 8
Trustworthiness of Science in Debate: Challenges, Responses, and Implications
Develaki, M. (2021) · Science & Education · doi:10.1007/s11191-021-00300-4 · Cited by 8
Kokinda, E., Moster, M., Dominic, J., & Rodeghero, P. (2023) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier58687.2023.00016 · Cited by 8
Recognizing the value of meta-research and making it easier to find
Stevens, E. R., & Laynor, G. (2023) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2023.1758 · Cited by 8
“To promote robust science, meta-research uses an interdisciplinary approach to examine research practices with the same scientific rigor given to other areas of scientific inquiry [4].”
BackgroundNudging transparent behavioural science and policy
Maynard, O., & Munafo, M. R. (2018) · Behavioural Public Policy · doi:10.1017/bpp.2018.10 · Cited by 9
“The field of meta-science (Ioannidis et al., 2015) is rapidly developing strategies to improve the transparency and robustness of research (Ioannidis, 2014; Munafò et al.”
BackgroundOrtloff, A.-M., Grohs, J. A., Lenau, S., & Smith, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714022 · Cited by 7
“Metascience analyzes research methods and current practice and aims to improve them [36].”
Cereda, F. (2025) · FORMAZIONE & INSEGNAMENTO. Rivista internazionale di Scienze dell'educazione e della formazione · doi:10.7346/-fei-xxiii-01-25_19 · Cited by 7
Reflecting on Research Practices
Pochat, V. L. (2024) · Communications of the ACM · doi:10.1145/3651965 · Cited by 7
Moraes, M. A., Arabe, L. B., Resende, B. L., Codo, B. C., Reis, A. L. D. A. L., & Souza, B. R. (2025) · Hormones and Behavior · doi:10.1016/j.yhbeh.2025.105707 · Cited by 7
Lozada-Martínez, I. D., Arias, L. M. D., Rojas, D. M. P., & Torres, I. L. R. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14238552 · Cited by 7
“The lack of systematic self-assessment means that promising discoveries from laboratories or small clinical trials may fail when applied to real-world patients [25].”
Predicting the Dynamics of Research Impact
Manolopoulos, Y., & Vergoulis, T. (2021) · Springer eBooks · doi:10.1007/978-3-030-86668-6 · Cited by 7
Ethics: What is the Research Scenario in the Brazilian Symposium WebMedia?
Carvalho, L. P., Suzano, J. A., Santoro, F. M., Oliveira, J., & Pimentel, M. D. G. C. (2022) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3539637.3557932 · Cited by 7
“The practice involving the supposed technique and the academic-scientific practice in question are subjects of ethical or moral study [39].”
BackgroundMethodZhang, P. (2023) · Frontiers in Neurology · doi:10.3389/fneur.2023.1221209 · Cited by 7
“Our approach in this article can be described as meta-scientific in nature (26).”
Testing the consistency of performance scores reported for binary classification problems
Fazekas, A., & Kovács, G. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111993 · Cited by 8
Mertz, M. (2017) · Springer eBooks · doi:10.1007/978-3-662-55377-0_6 · Cited by 7
Current Issues on Research Conducted to Improve Women’s Health
Siristatidis, C. S., Karageorgiou, V., & Vogiatzi, P. (2021) · Healthcare · doi:10.3390/healthcare9010092 · Cited by 7
“New foundations should be placed in research by focusing on the improvement of quality, such as NIH and PCORI [60].”
BackgroundA Case Study Protocol for Meta-Research into Digital Practices in the Humanities
Maryl, M., Dallas, C., Edmond, J., Labov, J., Kelpšienė, I., Doran, M., Kołodziejska, M., & Grabowska, K. (2020) · Digital humanities quarterly · doi:10.63744/ed2v2wz8mc3j · Cited by 7
“…or, more precisely, what it takes for researchers to consider themselves digital humanists; 5. it accounts for the holistic, contextual and situational dimensions of human culture retrospective studies to planned, prospective inquiry on scientific practice [Enserink 2018] [Ioannidis et al. 2015].”
BackgroundWissenschaftsdidaktik II: Einzelne Disziplinen
Reinmann, G., & Rhein, R. (2023) · transcript Verlag eBooks · doi:10.14361/9783839462959 · Cited by 18
Chapter 50. Treatment of Anxiety Disorders
Choi, S., & Stein, M. B. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as50 · Cited by 7
Educating educators on research on research
Ioannidis, J. P. A. (2021) · Perspectives on Medical Education · doi:10.1007/s40037-021-00662-z · Cited by 7
Self-citation
Metadata
- Authors
- Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N.
- Affiliations
- Stanford University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-10-02
- Volume
- 13
- Issue
- 10
- Pages
- e1002264
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4592065
- OpenAlex
- W1767553509
- Cited by
- 495
- References
- 10
- Citations by year
- 2026: 36 · 2025: 48 · 2024: 134 · 2023: 50 · 2022: 37 · 2021: 40 · 2020: 39 · 2019: 28 · 2018: 29 · 2017: 30 · 2016: 22 · 2015: 1
- FWCI
- 24.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Laura and John Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Keywords
- meta-research, research practices, research methodology, reproducibility, research reporting, research evaluation, research incentives, research reliability, research efficiency
- MeSH terms
- Animals, Humans, Meta-Analysis as Topic, Guidelines as Topic, Evidence-Based Medicine, Biomedical Research, Empirical Research, Internationality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.