EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Objective: To develop a computable representation for medical evidence and to contribute a gold standard dataset of annotated randomized controlled trial (RCT) abstracts, along with a natural language processing (NLP) pipeline for transforming free-text RCT evidence in PubMed into the structured representation. Materials and methods: Our representation, EvidenceMap, consists of 3 levels of abstraction: Medical Evidence Entity, Proposition and Map, to represent the hierarchical structure of medical evidence composition. Randomly selected RCT abstracts were annotated following EvidenceMap based on the consensus of 2 independent annotators to train an NLP pipeline. Via a user study, we measured how the EvidenceMap improved evidence comprehension and analyzed its representative capacity by comparing the evidence annotation with EvidenceMap representation and without following any specific guidelines. Results: Two corpora including 229 disease-agnostic and 80 COVID-19 RCT abstracts were annotated, yielding 12 725 entities and 1602 propositions. EvidenceMap saves users 51.9% of the time compared to reading raw-text abstracts. Most evidence elements identified during the freeform annotation were successfully represented by EvidenceMap, and users gave the enrollment, study design, and study Results sections mean 5-scale Likert ratings of 4.85, 4.70, and 4.20, respectively. The end-to-end evaluations of the pipeline show that the evidence proposition formulation achieves F1 scores of 0.84 and 0.86 in the adjusted random index score. Conclusions: EvidenceMap extends the participant, intervention, comparator, and outcome framework into 3 levels of abstraction for transforming free-text evidence from the clinical literature into a computable structure. It can be used as an interoperable format for better evidence retrieval and synthesis and an interpretable representation to efficiently comprehend RCT findings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“PubMed Online PICO interface RCT abstracts”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [150]
In the citing paper
“80 COVID-19 RCT abstracts + 229 general RCT abstracts”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [150]
In the citing paper
“Real-life evaluations, such as the percentage of outputs needing human correction, or time saved per article, were reported by four publications, 32 , 46 , 150 , 158”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [150]
Citations
Cites
Objective Criteria for the Evaluation of Clustering Methods
Rand, W. M. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482356 · Cited by 6,566
Utilization of the PICO framework to improve searching PubMed for clinical questions
Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P. (2007) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-7-16 · Cited by 3,053
A Dictionary of Linguistics and Phonetics
Crystal, D. (2008) · Wiley eBooks · doi:10.1002/9781444302776 · Cited by 2,190
The Levels of Evidence and Their Role in Evidence-Based Medicine
Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011) · Plastic and Reconstructive Surgery · doi:10.1097/prs.0b013e318219c171 · Cited by 2,155
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,118
Hanley, D. F., Thompson, R. E., Rosenblum, M., Yenokyan, G., Lane, K., McBee, N., Mayo, S. W., Bistran-Hall, A. J., Gandhi, D., Mould, W. A., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30195-3 · Cited by 927
Peng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899
Self-citationEvaluation of PICO as a knowledge representation for clinical questions.
Huang, X., Lin, J., & Demner‐Fushman, D. (2006) · Europe PMC (PubMed Central) · Cited by 898
Salloway, S., Chalkias, S., Barkhof, F., Burkett, P., Barakos, J., Purcell, D., Suhy, J., Forrestal, F., Tian, Y., Umans, K., et al. (2022) · JAMA Neurology · doi:10.1001/jamaneurol.2021.4161 · Cited by 619
Hopewell, S., Dutton, S., Yu, L.-M., Chan, A.-W., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c723 · Cited by 504
Cited by
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Leveraging generative AI for clinical evidence synthesis needs to ensure trustworthiness
Zhang, G., Jin, Q., Jered McInerney, D., Chen, Y., Wang, F., Cole, C. L., Yang, Q., Wang, Y., Malin, B. A., Peleg, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104640 · Cited by 40
Self-citationData extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
Wang, X., Huey, S. L., Sheng, R., Mehta, S., & Wang, F. (2025) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.70073 · Cited by 13
Li, Y., Luan, Z., Liu, Y., Liu, H., Qi, J., & Han, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1454945 · Cited by 9
Metadata
- Authors
- Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., & Weng, C.
- Affiliations
- Columbia University (US)
- Weill Cornell Medicine (US)
- Corresponding author
- Weng, Chunhua
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-03-15
- Volume
- 30
- Issue
- 6
- Pages
- 1022-1031
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10198523
- OpenAlex
- W4324321244
- Cited by
- 14
- References
- 33
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 6 · 2023: 1
- FWCI
- 1.24
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Bridging the semantic gap between research eligibility criteria and clinical data (5R01LM009886-11)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- randomized controlled trials, natural language processing pipeline, structured knowledge representation, evidence synthesis, evidence retrieval, COVID-19 clinical trials, PICO framework, gold standard dataset
- MeSH terms
- COVID-19, Humans, Natural Language Processing, Comprehension, PubMed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension. https://app.cochat.ai/papers/10.1093/jamia/ocad036
- The paper
- Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., & Weng, C. (2023). EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocad036
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.