The DOI matches its Crossref record; 195 of the 197 references it cites were found in a registry or an index.
Evidence appraisal: a scoping review, conceptual framework, and research agenda
Open accessAbstract
Objective: Critical appraisal of clinical evidence promises to help prevent, detect, and address flaws related to study importance, ethics, validity, applicability, and reporting. These research issues are of growing concern. The purpose of this scoping review is to survey the current literature on evidence appraisal to develop a conceptual framework and an informatics research agenda. Methods: We conducted an iterative literature search of Medline for discussion or research on the critical appraisal of clinical evidence. After title and abstract review, 121 articles were included in the analysis. We performed qualitative thematic analysis to describe the evidence appraisal architecture and its issues and opportunities. From this analysis, we derived a conceptual framework and an informatics research agenda. Results: We identified 68 themes in 10 categories. This analysis revealed that the practice of evidence appraisal is quite common but is rarely subjected to documentation, organization, validation, integration, or uptake. This is related to underdeveloped tools, scant incentives, and insufficient acquisition of appraisal data and transformation of the data into usable knowledge. Discussion: The gaps in acquiring appraisal data, transforming the data into actionable information and knowledge, and ensuring its dissemination and adoption can be addressed with proven informatics approaches. Conclusions: Evidence appraisal faces several challenges, but implementing an informatics research agenda would likely help realize the potential of evidence appraisal for improving the rigor and value of clinical evidence.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As vast evidence bases such as PubMed grows exponentially and rapidly, evidence retrieval and appraisal become extremely difficult due to information overload.3”
From A neuro-symbolic method for understanding free-text medical evidence. (2021) · cites this paper as [3]
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Research works printed but not linked: 1
- Cortese, 2007, The Learning Healthcare System: Workshop Summary
Cited by
Tod, D., Booth, A., & Smith, B. M. (2021) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.1952471 · Cited by 169
AI-generated text may have a role in evidence-based medicine
Peng, Y., Rousseau, J. F., Shortliffe, E. H., & Weng, C. (2023) · Nature Medicine · doi:10.1038/s41591-023-02366-9 · Cited by 64
Self-citationZhao, D., Zhu, G., Ding, Y., & Zheng, J. (2018) · Polymers · doi:10.3390/polym10070716 · Cited by 50
The voices of children and young people during COVID ‐19: A critical review of methods
Jörgensen, E., Koller, D., Raman, S., Olatunya, O. S., Asemota, O. A., Ekpenyong, B. N., Gunnlaugsson, G., & Okolo, A. A. (2022) · Acta Paediatrica · doi:10.1111/apa.16422 · Cited by 26
Understanding the nature and scope of clinical research commentaries in PubMed
Rogers, J. R., Mills, H., Liu, L. G., Goldstein, A., & Weng, C. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz209 · Cited by 18
Self-citation
Metadata
- Authors
- Goldstein, A., Venker, E., & Weng, C.
- Affiliations
- Columbia University (US)
- Corresponding author
- Goldstein, Andrew
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-05-24
- Volume
- 24
- Issue
- 6
- Pages
- 1192-1203
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6259661
- OpenAlex
- W2620100062
- Cited by
- 30
- References
- 197
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 5 · 2023: 8 · 2022: 3 · 2021: 2 · 2020: 2 · 2019: 4 · 2018: 3
- FWCI
- 0.61
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, other-oa)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. National Library of Medicine (R01LM009886)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- evidence appraisal, critical appraisal, clinical evidence, conceptual framework, research validity, research utilization, knowledge translation, assessment instruments
- MeSH terms
- Medical Informatics, Periodicals as Topic, Publishing, Peer Review, Research, Evidence-Based Medicine, Translational Research, Biomedical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evidence appraisal: a scoping review, conceptual framework, and research agenda. https://app.cochat.ai/papers/10.1093/jamia/ocx050
- The paper
- Goldstein, A., Venker, E., & Weng, C. (2017). Evidence appraisal: a scoping review, conceptual framework, and research agenda. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocx050
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.