Meta-research: Evaluation and Improvement of Research Methods and Practices
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
As the scientific enterprise has grown in size and diversity, we need empirical evidence on the research process to test and apply interventions that make it more efficient and its results more reliable. Meta-research is an evolving scientific discipline that aims to evaluate and improve research practices. It includes thematic areas of methods, reporting, reproducibility, evaluation, and incentives (how to do, report, verify, correct, and reward science). Much work is already done in this growing field, but efforts to-date are fragmented. We provide a map of ongoing efforts and discuss plans for connecting the multiple meta-research efforts across science worldwide.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The first is that the validity of these concerns is being assessed scientifically by a global metaresearch program, with results that have been briefly overviewed above (44).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Self-citationPolicy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
Self-citationBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Self-citationRescuing US biomedical research from its systemic flaws
Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1404402111 · Cited by 817
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Self-citationVan Noorden, R., Maher, B., & Nuzzo, R. (2014) · Nature · doi:10.1038/514550a · Cited by 508
Estimates of the Continuously Publishing Core in the Scientific Workforce
Ioannidis, J. P. A., Boyack, K. W., & Klavans, R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0101698 · Cited by 160 · cited 1×
“Several million new research papers are published annually, and the number of publishing authors in 1996–2011 exceeded, according to one estimate, 15 million [2].”
BackgroundSelf-citationReproducibility of Results in Preclinical Studies: A Perspective From the Bone Field
Manolagas, S. C., & Kronenberg, H. M. (2014) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.2293 · Cited by 44
Cited by
Items 201–300 of 545
Duranté, E. K., Ribeiro, A., Gaspard-Boulinc, L., Boutron, I., Henry, C., Petit, A., Houenou, J., Lemogne, C., & Chevance, A. (2025) · The British Journal of Psychiatry · doi:10.1192/bjp.2024.292 · Cited by 8
The American Psychiatric Association Publishing Textbook of Psychopharmacology
Schatzberg, A. F., & Nemeroff, C. B. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523 · Cited by 10
Charting the neuroscience of interpersonal trust: A bibliographic literature review
Wu, Y. H., & Krüeger, F. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105930 · Cited by 6
Nielsen, S. M., Storgaard, H., Ellingsen, T., Shea, B., Wells, G. A., Welch, V. A., Furst, D. E., de Wit, M. P. T., Voshaar, M., Juhl, C. B., et al. (2020) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2020-217237 · Cited by 9
Béchard, B., Kimmerle, J., Lawarée, J., Bédard, P.-O., Straus, S. E., & Ouimet, M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19106234 · Cited by 7
“Systematic reviews, which are part of the methodological arsenal of meta-research [3,4], provide an analysis of the limitations or uncertainty of relevant studies by drawing on a growing library of methodological tools [5,6].”
BackgroundNew Guidelines for Null Hypothesis Significance Testing in Hypothetico-Deductive IS Research
scientist, B. S.-E. D., Mertens, W., & Recker, J. (2020) · Journal of the Association for Information Systems · doi:10.17705/1jais.00629 · Cited by 40
“They are also backed by an increasing amount of evidence regarding their effectiveness (e.g., Munafò et al., 2017; Starbuck, 2016) that has been accrued through metaresearch that examines scientific practices and develops and tests alternatives (e.g., Ioannidis et al., 2015).”
BackgroundInformation science’s contributions towards emerging open evaluation practices
Pinto, L. G., & Ochôa, P. (2018) · Performance Measurement and Metrics · doi:10.1108/pmm-05-2018-0015 · Cited by 6
Porta, M. S., & Bolúmar, F. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0181-3 · Cited by 7
Veldkamp, C. L. S. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/g8cjq · Cited by 6
Predictors of citation rates and the problem of citation bias: a scoping review
Nørgaard, B., Lie, K. E., & Lund, H. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.112057 · Cited by 6
Amarilyo, G., Fürst, D. E., Woo, J. M. P., Li, W., Bliddal, H., Christensen, R., & Tarp, S. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147556 · Cited by 6
Verhagen, A. P., Stubbs, P. W., Mehta, P., Kennedy, D. S., Nasser, A. M., de Oliveira, C. Q., Pate, J. W., Skinner, I., & McCambridge, A. B. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-054875 · Cited by 5
Bała, M. M., Peričić, T. P., Žuljević, M. F., Bralić, N., Zając, J., Motaze, N. V., Rohwer, A. C., Gajdzica, M., & Young, T. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112647 · Cited by 6
“42 43 Reporting guidelines aim to improve research and reporting and there are numerous strategies available to enhance adherence to reporting guidelines that include different stakeholders, like journals, funding bodies and ethics boards.”
Evaluating strategies to recruit health researchers to participate in online survey research
Stevens, E. R., Cleland, C. M., Shunk, A., & Shahawy, O. E. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02275-6 · Cited by 6
“The median number of years of research experience was 12 [1, 45], 60.9% had a PhD, 33.8% a MD, and 3.4% had an MD/PhD.”
BackgroundInfluentialTsionkis, G., Andronidou, E. M., Kontou, P. I., Tamposis, I., Tegopoulos, K., Pergantas, P., Grigoriou, M. E., Skavdis, G., Bagos, P. G., & Braliou, G. G. (2025) · Pharmaceuticals · doi:10.3390/ph18081139 · Cited by 5
“The field of metascience has emerged in response to the so-called “replication crisis,” reflecting a growing awareness of the need for greater transparency, rigor and reliability in scientific research [24–26].”
Brand, R., Nosrat, S., Späth, C. B., & Timme, S. (2022) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2022.864468 · Cited by 5
Is Social Work Research in Crisis?
Dunleavy, D. J., & Lacasse, J. R. (2023) · Research on Social Work Practice · doi:10.1177/10497315231188684 · Cited by 7
“Meta-research (sometimes referred to as “ meta-science ” ) is a broad and burgeoning area of study, which concerns the methods, reporting, and reproducibility of research (Ioannidis et al., 2015).”
Carvalho, L. P., Filho, S. L., Brandão, M. A., Marinho, É. C., da Silva, M. F., Oliveira, J., & Santoro, F. M. (2024) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3658271.3658300 · Cited by 5
“A meta-ciência se desenvolve a partir de um estudo da ciência, estruturado por meio da prática ou método científicos [21, 22, 32].”
BackgroundInfluentialSobre o Significado da Significância Estatística
Teixeira, P. (2018) · Acta Médica Portuguesa · doi:10.20344/amp.9375 · Cited by 5
Chapter 49. Treatment of Bipolar Disorder
Strakowski, S. M., & Keck, P. E. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as49 · Cited by 5
Lapiedra, E. S., & Carrasco‐Campos, Á. (2018) · Estudos em Comunicação · doi:10.20287/ec.n26.v1.a16 · Cited by 6
Mechanical Thrombectomy and Functional Outcomes After Stroke
Rodrigues, F. B., Langhorne, P., & Costa, J. N. (2016) · JAMA · doi:10.1001/jama.2016.0383 · Cited by 5
The importance of meta-research in dentistry
Sarkis‐Onofre, R., Sofi‐Mahmudi, A., Puljak, L., & Moraes, R. R. (2023) · Evidence-Based Dentistry · doi:10.1038/s41432-023-00880-w · Cited by 5
Ethics: What is the Brazilian Software Engineering Research Scenario?
Carvalho, L. P., Suzano, J. A., Batista, T., Santoro, F. M., & Oliveira, J. (2024) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2024.3395 · Cited by 4
“In order to mature and think about the contributions and scientific communications of the Brazilian Software community, we align this research with the concept of meta-science, or meta-research ( Ioannidis et al., 2015; Enserink, 2018).”
MethodAstaneh, B., Abdullah, R., Astaneh, V., Gupta, S., Brignardello‐Petersen, R., Levine, M., & Guaytt, G. (2024) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe14090165 · Cited by 4
“We use the term systematic survey to differentiate it from systematic reviews of clinical areas [27,28].”
MethodLozada-Martínez, I. D., Visconti-Lopez, F. J., Rojas-Cueva, A. C., Ausejo, F., Castrillón-Lozano, J. A., Pedroza, N. A. C., Ordoñez, R. L. Y., García, A. M., Santana, B. O., Diazgranados-Garcia, M. C., et al. (2024) · International Journal of Surgery Open · doi:10.1097/io9.0000000000000233 · Cited by 4
“However, internal and external validity depends on the quality of the study’s execution and publication (methodological and reporting quality, respectively), forming the basis for the reproducibility and critical approach of evidence-based medicine [ 3 ] .”
Mazzoleni, G. Z., Bergna, A., Buffone, F., Sacchi, A., Misseroni, S., Tramontano, M., & Farra, F. D. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13175181 · Cited by 4
“To better evaluate these aspects, meta-research studies have been increasingly used in recent years, focusing on various aspects such as how research is performed, reported and evaluated [11–13].”
MethodKhanteymoori, A., Peterson, C., Atamny, R., Hohenhaus, M., Beck, J., Howells, D. W., Schwab, J. M., & Watzlawick, R. (2025) · Neurorehabilitation and neural repair · doi:10.1177/15459683241311337 · Cited by 4
“48 Nevertheless, they are subject to intense scrutiny and debate within the statistical community.”
Bibliometrics for Social Validation
Hicks, D. (2016) · PLoS ONE · doi:10.1371/journal.pone.0168597 · Cited by 4
Sandars, J. E., Brown, J., Nwolise, C. H., Patel, M., Dogra, N., Kaehne, A., Garner, J., Watmough, S., Maden, M., & Duckworth, V. (2020) · MedEdPublish · doi:10.15694/mep.2020.000084.1 · Cited by 6
“There has been increasing interest in ‘ meta-research ’ , which critically evaluates how the approach to conducting research determines the findings and subsequently informs future major decisions about policy, practice and future research (Ioannidis et al. , 2015; Ioannidis, 2018).”
Scientometric Analysis of The Research on The Abortion: 2015-2019
Sivankalai, S. (2021) · Türk bilgisayar ve matematik eğitimi dergisi · doi:10.17762/turcomat.v12i4.482 · Cited by 4
Propuesta metodológica integradora para investigar la tematización en la comunicación política
Rojas-Calderón, A. (2022) · Communication & Methods · doi:10.35951/v4i1.155 · Cited by 4
Teaching scientific research integrity: A case study
Saltelli, A. (2023) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2237949 · Cited by 5
“Nosek’s Centre for Open Science (Open Science Collaboration, 2015) and Ioannidis’ Meta-Research Innovation Center at Stanford (METRICS, Ioannidis et al., 2015) are noteworthy institutional attempts to address the crisis.”
Thirteen Years of WebMedia (2010 to 2022) - A Comprehensive Descriptive and Meta-Scientific Analysis
Carvalho, L. P., Filho, S. L., Brandão, M. A., da Silva, M. F., Oliveira, J., & Pimentel, M. D. G. C. (2023) · Brazilian Symposium on Multimedia and the Web · doi:10.1145/3617023.3617048 · Cited by 4
“This work follows the “science of science” trend in metascience, analyzing the scientific context from objective data and information, scientifically supported [20, 21, 33].”
BackgroundMethodInfluentialSpreckelsen, T. F. (2018) · Child and Adolescent Mental Health · doi:10.1111/camh.12277 · Cited by 4
Tillu, G. S. (2019) · Journal of Ayurveda and Integrative Medicine · doi:10.1016/j.jaim.2019.09.001 · Cited by 4
“It covers diverse disciplines ranging from history, epistemology, ethics and philosophy of science to newer advances like data science, scientometrics, policy and economics [1].”
BackgroundKhakshooy, A. M. (2020) · Bioinformation · doi:10.6026/97320630016004 · Cited by 4
Maggio, L. A., Haustein, S., & Artino, A. R. (2020) · Perspectives on Medical Education · doi:10.1007/s40037-020-00627-8 · Cited by 4
Étude exploratoire sur la « recherche sur la recherche » : acteurs et approches
Gruson-Daniel, C., & Anderson-González, M. (2021) · doi:10.52949/24 · Cited by 4
Report of the G7 Open Science - Research on Research Sub-Working Group
Canada, E. E. A. C. C., Cloutier, M., Dacos, M., & de la recherche, M. D. L. S. E. (2023) · doi:10.52949/32 · Cited by 8
Chapter 57. Treatment of Eating Disorders
Agras, W. S. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as57 · Cited by 4
Scholarly Communication and Open Access in Psychology: Current Considerations for ResearchersPreprint
Mullen, L. B. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/2d7um · Cited by 4
Chapter 55. Treatment of Alcohol Use Disorder
Mason, B. J., Ritvo, E. C., & Bonci, A. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as55 · Cited by 3
The New Modern Medicine: Disease, Evidence, and Epidemiological Medicine
Fuller, J. (2025) · doi:10.1093/9780190066178.001.0001 · Cited by 7
Self-citation practices in applied linguistics
Plonsky, L., & Sudina, E. (2025) · Journal of Second Language Studies · doi:10.1075/jsls.00052.plo · Cited by 3
La reproducibilidad de las investigaciones biomédicas: Quo vadis?
Dal‐Ré, R., Bernad, A., & Garesse, R. (2016) · Medicina Clínica · doi:10.1016/j.medcli.2015.11.029 · Cited by 4
Dopamine and Its Actions in the Basal Ganglia System
Bullock, D. H. (2016) · Innovations in cognitive neuroscience · doi:10.1007/978-3-319-42743-0_5 · Cited by 5
Nature of Science als Grundlage einer Wissenschaftsdidaktik der Naturwissenschaftsdidaktiken
Büssing, A. G., Nehring, A., & Bruckermann, T. (2023) · transcript Verlag eBooks · doi:10.14361/9783839462959-013 · Cited by 5
Quality of Flowcharts in Meta-Analyses in Education: A Systematic Review
Martín, M. S., Sánchez, M., Moreno, E. M. O., & Navarro‐Mateu, F. (2025) · ESPIRAL CUADERNOS DEL PROFESORADO · doi:10.25115/ecp.v18i37.10159 · Cited by 3
Tafeni, Z., & Mngomezulu, B. R. (2025) · Journal of Local Government Research and Innovation · doi:10.4102/jolgri.v6i0.236 · Cited by 3
Busatto, G. F., Rosa, P. G. P., Serpa, M. H., Squarzoni, P., & Duran, F. (2020) · Brazilian Journal of Psychiatry · doi:10.1590/1516-4446-2019-0757 · Cited by 3
Os Princípios de Hong Kong e sua importância para o ecossistema científico atual
da Costa, G. G., Alves, C. P. D. L., & Luizeti, B. O. (2020) · Journal of Evidence-Based Healthcare · doi:10.17267/2675-021xevidence.v2i2.3247 · Cited by 5
La investigación post-disciplinaria como alternativa en comunicación
Navarro, R. F. (2021) · Revista de la Asociación Española de Investigación de la Comunicación · doi:10.24137/8.15.1 · Cited by 3
Hayran, O. (2021) · Eskişehir Türk Dünyası Uygulama ve Araştırma Merkezi Halk Sağlığı Dergisi · doi:10.35232/estudamhsd.947591 · Cited by 3
Journalology – academic publishing process
Asgarov, F. (2022) · Problems of Information Technology · doi:10.25045/jpit.v13.i.06 · Cited by 3
Ethics: What is the Research Scenario in the LARS/SBR/WRE?
Carvalho, L. P., Oliveira, J., & Santoro, F. M. (2022) · Latin American Robotics Symposium, Brazilian Symposium on Robotics and Workshop on Robotics in Education (LARS/SBR/WRE) · doi:10.1109/lars/sbr/wre56824.2022.9995832 · Cited by 3
“Additionally, as a meta-scientific analysis [15], we cover elements such as Ethics Committee (EC) and Informed Consent (IC).”
MethodÉtica: Qual o Panorama de Pesquisa no CSBC?
Carvalho, L. P., Louzada, A. N., Batista, T., Oliveira, J., Brandão, M., & Santoro, F. M. (2023) · Anais do Seminário Integrado de Software e Hardware (SEMISH) · doi:10.5753/semish.2023.229675 · Cited by 3
“Métodos, relatórios, reprodutibilidade, avaliação e incentivos são as principais áreas da metaciência; estudando como realizar, comunicar, verificar, avaliar e recompensar pesquisas [Ioannidis et al. 2015].”
BackgroundMeta-research as discipline, field, or spectrum
Derrick, G., Nuseibeh, N., Oancea, A., & Xu, X. (2024) · Edward Elgar Publishing eBooks · doi:10.4337/9781839105722.00006 · Cited by 3
Krauß, A. (2024) · Oxford University Press eBooks · doi:10.1093/9780198937401.003.0013 · Cited by 3
Editors’ Notes: Moving Research on Evaluation Forward
Aston, R., Wanzer, D. L., & Westine, C. D. (2025) · New Directions for Evaluation · doi:10.1002/ev.70003 · Cited by 3
“This is akin to metascience (studying science itself) and meta-research (research on research; Ioannidis et al. 2015).”
Dissemination and Exploitation of European Projects
Kaur, B., & Nikander, J. (2017) · EAEEIE Annual Conference (EAEEIE) · doi:10.1109/eaeeie.2017.8768687 · Cited by 3
“various research methods that have risen, and their application in the various fields [3].”
BackgroundMeio ambiente, recursos naturais e... abelhas? Investigando uma década de WCAMA
Carvalho, L. P., Filho, S. L., Brandão, M. A., Oliveira, J., Santoro, F. M., & da Silva, M. F. (2023) · doi:10.5753/wcama.2023.230600 · Cited by 3
Can patient decision aids be used to improve not only decisional comfort but also adherence?
Tugwell, P., & Knottnerus, J. A. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.10.005 · Cited by 3
“provide the latest evidence on this in a nice example of ‘Meta-research’ (the scientific study of research practices [2])*, they reviewed 59 meta-epidemiological studies with data from 3,199 meta-analyses, 32 networks, and 21,468 trials and found that there was consistent evidence that allocation concealment, sequence…”
BackgroundTowards a Culture of Open Science and Data Sharing in Health and Medical Research
Rowhani-Farid, A. (2018) · Queensland University of Technology · doi:10.5204/thesis.eprints.119697 · Cited by 6
Unifying Classification Schemes for Software Engineering Meta-ResearchPreprint
Kaplan, A., Kuhn, T. S., & Reussner, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.10491 · Cited by 3
“4.1), we obtain classifications to categorize software engineering research that are assigned to the five main meta-research areas according Ioannidis et al. [7] (i.e., Meth-ods, Reporting, Reproducibility, Evaluation, and Incentives).”
BackgroundMethodInfluentialNg, J. Y., & Cramer, H. (2024) · Journal of Integrative and Complementary Medicine · doi:10.1089/jicm.2024.0444 · Cited by 3
How Ten Publishers Retract ResearchPreprint
Oppenlaender, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.19197 · Cited by 3
Poline, B., Vul, E., & Tal (2016) · Cited by 2
Biomedical Research in the 21st Century: Multiple Challenges in Resolving Reproducibility Issues
Mullane, K., Curtis, M. J., & Williams, M. J. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-804725-5.00006-9 · Cited by 4
Muralidhara, M., & Rajini, P. S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-443-19084-1.00014-4 · Cited by 5
The state-of-the-art of research on science research careers
Cañibano, C., Woolley, R., Iversen, E. J., & Corona-Sobrino, C. (2024) · Edward Elgar Publishing eBooks · doi:10.4337/9781839105722.00032 · Cited by 3
Gao, K., & Calabrese, J. R. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as41 · Cited by 2
Cano, B. H. S., da Silva, G. F. M. V., Bottari, G. D., Balbinot, E. L., Uggioni, M. L. R., Roever, L., da Rosa, M. I., & Grande, A. J. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02501-9 · Cited by 2
Maier, J. (2025) · International Journal of Advertising · doi:10.1080/02650487.2025.2536907 · Cited by 2
Siegel, M., Fanninger, S., Riedel, J., & Zemp, M. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1535606 · Cited by 2
Gregersen, L., Moos, C. M., Hikmat, Z., Rasmussen, N. F., Petersen, S. R., Heitmann, B. L., Halldorsson, T. I., Andersen, V. C., & Christensen, R. (2026) · Advances in Nutrition · doi:10.1016/j.advnut.2026.100614 · Cited by 2
“Evidence-based research (metaresearch or research on research) seeks to identify best practices and evaluates reasons for systematic biases by taking a bird ’ s-eye perspective on the steps in the research process rather than criticizing individual studies [13,14].”
Four decades of security and privacy research: evolution of topics, impact, and the community
Demir, N., Böttger, C., Hosseini, H., Demir, M., Wressnegger, C., Pohlmann, N., & Urban, T. (2026) · Journal of Cybersecurity · doi:10.1093/cybsec/tyag007 · Cited by 2
Discovery tools: How powerful new scientific methods and instruments emerge and catalyze innovation
Krauß, A. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag107 · Cited by 2
“Second, meta-researchers have examined methodological questions about science itself, including reproducibility, peer review, and assessing research impact ( 12 – 14).”
Acosta-Monterrosa, A. A., Montoya-Quintero, K. F., Visconti-Lopez, F. J., & Lozada-Martínez, I. D. (2026) · Frontiers in Environmental Science · doi:10.3389/fenvs.2026.1766738 · Cited by 2
Gower, T., Styler, Z. R., & Litz, B. T. (2026) · Clinical Psychology Review · doi:10.1016/j.cpr.2026.102786 · Cited by 2
The general index of software engineering papers
Khalil, Z. A., & Zacchiroli, S. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528494 · Cited by 3
“Meta-research [9, 10] is the use of the scientific method to study science itself.”
MethodKulo, V. A. (2022) · American Educational Research Association Annual Meeting (AERA) · doi:10.3102/1886774 · Cited by 2
Ethics: What is the Research Scenario in the Brazilian Conference BRACIS?
Carvalho, L. P., Murakami, L., Suzano, J. A., Oliveira, J., Revoredo, K. C., & Santoro, F. M. (2022) · doi:10.5753/eniac.2022.227590 · Cited by 4
Carvalho, L. P., Suzano, J. A., Santoro, F. M., Oliveira, J., & Pimentel, M. D. G. C. (2023) · Journal on Interactive Systems · doi:10.5753/jis.2023.3138 · Cited by 2
“…whether applied, practical, conceptual, theoretical, literature reviews, pragmatics, essays, whatever, there is room for ethical deliberation, whether as an application or about the respective research (meta-research) (Recker, 2021; Ioannidis et al., 2015; Ioannidis, 2018; Carvalho et al., 2022b).”
BackgroundMethodInfluentialBreSci na última década: Uma análise baseada em redes sociais
Filho, S. L., Carvalho, L. P., Brandão, M. A., Oliveira, J., Santoro, F. M., & da Silva, M. F. (2023) · doi:10.5753/bresci.2023.233890 · Cited by 2
Chapter 20. Duloxetine, Milnacipran, and Levomilnacipran
Jha, M. K., Mayes, T. L., & Trivedi, M. H. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as20 · Cited by 2
Chapter 11. Tricyclic and Tetracyclic Drugs
Nelson, J. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as11 · Cited by 2
Onze anos de WEI (2013 – 2023), uma análise meta-científica
Carvalho, L. P., Filho, S. L., Brandão, M. A., Oliveira, J., Santoro, F. M., & da Silva, M. F. (2024) · Anais do Workshop sobre Educação em Computação (WEI) · doi:10.5753/wei.2024.2479 · Cited by 2
“…formais [Agassi 2008], neste presente trabalho apresentamos uma visão mate-rialista através de outras variáveis [Latour 1987, Lefèvre 2005], com uma ênfase meta-científica [Ioannidis et al. 2015, Ioannidis 2018] por uma abordagem analítico-descritiva [Marconi e Lakatos 2017, Wazlawick 2014].”
BackgroundEsteban, M. O., Delgado, M., Gil, F., & Thabane, L. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310092 · Cited by 2
Open science in the Republic of Moldova
Cuciureanu, G., Cojocaru, I., Cojocaru, I., Țurcan, N., Grecu, M., Alfreda, R., Rusu, A., Cujba, R., Coşuleanu, I., Ungureanu, E., et al. (2018) · doi:10.57066/sdrm18 · Cited by 3
Meta-Research on Oncology Trials: A Toolkit for Researchers with Limited Resources
Riechelmann, R. P., Péron, J., Šeruga, B., & Saad, E. D. (2018) · The Oncologist · doi:10.1634/theoncologist.2018-0043 · Cited by 2
Errors and Error Management in Biomedical Research
Dirnagl, U., & Bernard, R. (2018) · Springer eBooks · doi:10.1007/978-3-319-76403-0_8 · Cited by 3
He, S., Klatt, K. C., Rahnavard, A., Barberio, M. D., Gernand, A. D., & Smith, E. R. (2020) · Gates Open Research · doi:10.12688/gatesopenres.13199.1 · Cited by 2
Feminism, Leadership, and Social Media: The Feminists From the Nigerian #EndSARS Protests
Martins, A. A. (2022) · International Journal of Social Media and Online Communities · doi:10.4018/ijsmoc.308288 · Cited by 2
Using meta‐research to foster diverse, equitable, and inclusive collaborative research networks
Stevens, E. R., Brody, A. A., Epps, F. R., Sloan, D. H., & Sherman, S. E. (2022) · Journal of the American Geriatrics Society · doi:10.1111/jgs.18217 · Cited by 2
Twenty one years of SBQS (2002 to 2022): A Comprehensive Descriptive and Meta-Scientific Analysis
Carvalho, L. P., Filho, S. L., Brandão, M. A., da Silva, M. F., Oliveira, J., Santos, G., & Rocha, A. R. (2023) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3629479.3629493 · Cited by 2
“Meta-science emerges from the study of Science in a structured and formalized way through scientific practice [23, 24, 38], to scientifically deal with its complexity, including its practice.”
BackgroundMethodInfluentialChapter 6. Pharmacokinetics, Pharmacodynamics, and Pharmacogenomics
Goldberg, J. F. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as06 · Cited by 2
Chapter 8. Personalized Medicine in Psychiatry
Perna, G., Alciati, A., & Nemeroff, C. B. (2024) · American Psychiatric Association Publishing eBooks · doi:10.1176/appi.books.9781615375523.as08 · Cited by 2
Maggio, L. A., Haustein, S., & Artino, A. R. (2020) · Perspectives on Medical Education · doi:10.1007/s40037-020-00640-x · Cited by 2
Gruson-Daniel, C. (2023) · doi:10.31222/osf.io/p4mca · Cited by 2
Metadata
- Authors
- Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N.
- Affiliations
- Stanford University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-10-02
- Volume
- 13
- Issue
- 10
- Pages
- e1002264
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4592065
- OpenAlex
- W1767553509
- Cited by
- 495
- References
- 10
- Citations by year
- 2026: 36 · 2025: 48 · 2024: 134 · 2023: 50 · 2022: 37 · 2021: 40 · 2020: 39 · 2019: 28 · 2018: 29 · 2017: 30 · 2016: 22 · 2015: 1
- FWCI
- 24.65
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Laura and John Arnold Foundation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Business, Management and Accounting › Management of Technology and Innovation › Research, Science, and Academia
- Keywords
- meta-research, research practices, research methodology, reproducibility, research reporting, research evaluation, research incentives, research reliability, research efficiency
- MeSH terms
- Animals, Humans, Meta-Analysis as Topic, Guidelines as Topic, Evidence-Based Medicine, Biomedical Research, Empirical Research, Internationality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.