CoChat

EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension

Open access

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Abstract

Objective: To develop a computable representation for medical evidence and to contribute a gold standard dataset of annotated randomized controlled trial (RCT) abstracts, along with a natural language processing (NLP) pipeline for transforming free-text RCT evidence in PubMed into the structured representation. Materials and methods: Our representation, EvidenceMap, consists of 3 levels of abstraction: Medical Evidence Entity, Proposition and Map, to represent the hierarchical structure of medical evidence composition. Randomly selected RCT abstracts were annotated following EvidenceMap based on the consensus of 2 independent annotators to train an NLP pipeline. Via a user study, we measured how the EvidenceMap improved evidence comprehension and analyzed its representative capacity by comparing the evidence annotation with EvidenceMap representation and without following any specific guidelines. Results: Two corpora including 229 disease-agnostic and 80 COVID-19 RCT abstracts were annotated, yielding 12 725 entities and 1602 propositions. EvidenceMap saves users 51.9% of the time compared to reading raw-text abstracts. Most evidence elements identified during the freeform annotation were successfully represented by EvidenceMap, and users gave the enrollment, study design, and study Results sections mean 5-scale Likert ratings of 4.85, 4.70, and 4.20, respectively. The end-to-end evaluations of the pipeline show that the evidence proposition formulation achieves F1 scores of 0.84 and 0.86 in the adjusted random index score. Conclusions: EvidenceMap extends the participant, intervention, comparator, and outcome framework into 3 levels of abstraction for transforming free-text evidence from the clinical literature into a computable structure. It can be used as an interoperable format for better evidence retrieval and synthesis and an interpretable representation to efficiently comprehend RCT findings.

Overview

Integrity signals

Registry

Crossref · 10.1093/jamia/ocad036

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 33 references

Cited by

See all 14

Metadata

Authors
Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., & Weng, C.
Affiliations
  • Columbia University (US)
  • Weill Cornell Medicine (US)
Corresponding author
Weng, Chunhua
Venue
Journal of the American Medical Informatics Association
Published
2023-03-15
Volume
30
Issue
6
Pages
1022-1031
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W4324321244
Cited by
14
References
33
Citations by year
2026: 1 · 2025: 6 · 2024: 6 · 2023: 1
FWCI
1.24
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
Bridging the semantic gap between research eligibility criteria and clinical data (5R01LM009886-11)
Topics
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
randomized controlled trials, natural language processing pipeline, structured knowledge representation, evidence synthesis, evidence retrieval, COVID-19 clinical trials, PICO framework, gold standard dataset
MeSH terms
COVID-19, Humans, Natural Language Processing, Comprehension, PubMed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension. https://app.cochat.ai/papers/10.1093/jamia/ocad036
The paper
Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., & Weng, C. (2023). EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocad036

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.