EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Objective: To develop a computable representation for medical evidence and to contribute a gold standard dataset of annotated randomized controlled trial (RCT) abstracts, along with a natural language processing (NLP) pipeline for transforming free-text RCT evidence in PubMed into the structured representation. Materials and methods: Our representation, EvidenceMap, consists of 3 levels of abstraction: Medical Evidence Entity, Proposition and Map, to represent the hierarchical structure of medical evidence composition. Randomly selected RCT abstracts were annotated following EvidenceMap based on the consensus of 2 independent annotators to train an NLP pipeline. Via a user study, we measured how the EvidenceMap improved evidence comprehension and analyzed its representative capacity by comparing the evidence annotation with EvidenceMap representation and without following any specific guidelines. Results: Two corpora including 229 disease-agnostic and 80 COVID-19 RCT abstracts were annotated, yielding 12 725 entities and 1602 propositions. EvidenceMap saves users 51.9% of the time compared to reading raw-text abstracts. Most evidence elements identified during the freeform annotation were successfully represented by EvidenceMap, and users gave the enrollment, study design, and study Results sections mean 5-scale Likert ratings of 4.85, 4.70, and 4.20, respectively. The end-to-end evaluations of the pipeline show that the evidence proposition formulation achieves F1 scores of 0.84 and 0.86 in the adjusted random index score. Conclusions: EvidenceMap extends the participant, intervention, comparator, and outcome framework into 3 levels of abstraction for transforming free-text evidence from the clinical literature into a computable structure. It can be used as an interoperable format for better evidence retrieval and synthesis and an interpretable representation to efficiently comprehend RCT findings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“PubMed Online PICO interface RCT abstracts”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [150]
In the citing paper
“80 COVID-19 RCT abstracts + 229 general RCT abstracts”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [150]
In the citing paper
“Real-life evaluations, such as the percentage of outputs needing human correction, or time saved per article, were reported by four publications, 32 , 46 , 150 , 158”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [150]
Citations
Cites
Objective Criteria for the Evaluation of Clustering Methods
Rand, W. M. (1971) · Journal of the American Statistical Association · doi:10.1080/01621459.1971.10482356 · Cited by 6,566
Utilization of the PICO framework to improve searching PubMed for clinical questions
Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P. (2007) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-7-16 · Cited by 3,053
A Dictionary of Linguistics and Phonetics
Crystal, D. (2008) · Wiley eBooks · doi:10.1002/9781444302776 · Cited by 2,190
The Levels of Evidence and Their Role in Evidence-Based Medicine
Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011) · Plastic and Reconstructive Surgery · doi:10.1097/prs.0b013e318219c171 · Cited by 2,155
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,118
Hanley, D. F., Thompson, R. E., Rosenblum, M., Yenokyan, G., Lane, K., McBee, N., Mayo, S. W., Bistran-Hall, A. J., Gandhi, D., Mould, W. A., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30195-3 · Cited by 927
Peng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899
Self-citationEvaluation of PICO as a knowledge representation for clinical questions.
Huang, X., Lin, J., & Demner‐Fushman, D. (2006) · Europe PMC (PubMed Central) · Cited by 898
Salloway, S., Chalkias, S., Barkhof, F., Burkett, P., Barakos, J., Purcell, D., Suhy, J., Forrestal, F., Tian, Y., Umans, K., et al. (2022) · JAMA Neurology · doi:10.1001/jamaneurol.2021.4161 · Cited by 619
Hopewell, S., Dutton, S., Yu, L.-M., Chan, A.-W., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c723 · Cited by 504
Altorki, N. K., McGraw, T. E., Borczuk, A. C., Saxena, A., Port, J. L., Stiles, B. M., Lee, B. E., Sanfilippo, N. J., Scheff, R. J., Pua, B. B., et al. (2021) · The Lancet Oncology · doi:10.1016/s1470-2045(21)00149-2 · Cited by 342
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
Nye, B., Li, J. J., Patel, R., Yang, Y., Marshall, I., Nenkova, A., & Wallace, B. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1019 · Cited by 224
New Thinking about Propositions
King, J. C., Soames, S., & Speaks, J. (2014) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199693764.001.0001 · Cited by 219
(2016) · PubMed · Cited by 134
Automating Biomedical Evidence Synthesis: RobotReviewer
Marshall, I., Kuiper, J., Banner, E., & Wallace, B. C. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4002 · Cited by 96
Blake, C. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.11.001 · Cited by 91
Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature.
Tian, K., Shirui, Z., & Chunhua, W. (2019) · Studies in Health Technology and Informatics · doi:10.3233/shti190209 · Cited by 45
On the composition of scientific abstracts
Atanassova, I., Bertin, M., & Larivière, V. (2016) · Journal of Documentation · doi:10.1108/jdoc-09-2015-0111 · Cited by 41
Ghanei, M., Solaymani–Dodaran, M., Qazvini, A., Ghazale, A. H., Setarehdan, S. A., Saadat, S. H., Ghobadi, H., Hoseininia, S., Elahikhah, M., Takaldani, A. H. S., et al. (2021) · Respiratory Research · doi:10.1186/s12931-021-01833-6 · Cited by 38
The effect of platelet-rich plasma injection on post-internal urethrotomy stricture recurrence
Rezaei, M., Badiei, R., & Badiei, R. (2018) · World Journal of Urology · doi:10.1007/s00345-018-2597-8 · Cited by 30
Classification of PICO elements by text features systematically extracted from PubMed abstracts.
Huang, K.-C., Liu, C. C.-H., Yang, S.-S., Xiao, F., Wong, J.-M., Liao, C.-C., & Chiang, I.-J. (2011) · 2011 IEEE International Conference on Granular Computing · doi:10.1109/grc.2011.6122608 · Cited by 26
Islam, N. S., Riley, L., Wyatt, L. C., Tandon, S. D., Tanner, M., Mukherji-Ratnam, R., Rey, M. J., & Trinh‐Shevrin, C. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-177 · Cited by 26
Hernández‐Cardenas, C. M., Thirión-Romero, I. I., Rodríguez‐Llamazares, S., Rivera-Martínez, N. E., Meza-Meneses, P. A., Remigio-Luna, A., & Pérez‐Padilla, R. (2021) · PLoS ONE · doi:10.1371/journal.pone.0257238 · Cited by 24
Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time
Nye, B., Nenkova, A., Marshall, I., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.9 · Cited by 23
Rivaroxaban Plus Aspirin in Obese and Overweight Patients With Vascular Disease in the COMPASS Trial
Guzik, T. J., Ramasundarahettige, C. F., Вачиковна, П. Н., López-Jaramillo, P. A., Dyal, L., Berkowitz, S. D., Muehlhofer, E., Bhatt, D. L., Fox, K. A. A., Yusuf, S., et al. (2021) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.11.061 · Cited by 20
Trial banks: an informatics foundation for evidence-based medicine
Rennels, G. D., Musen, M. A., & Sim, I. (1998) · Cited by 19
Combination of conditional random field with a rule based method in the extraction of PICO elements.
Chabou, S., & Iglewski, M. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0699-2 · Cited by 19
Finn, J., Brink, D., McKenzie, N., García, A., Tohira, H., Perkins, G. D., Arendts, G., Fatovich, D. M., Hendrie, D. V., McQuillan, B., et al. (2021) · Emergency Medicine Journal · doi:10.1136/emermed-2020-210256 · Cited by 17
Kaiser, J., Niesen, W., Probst, P., et al. (2019) · Trials · doi:10.1186/s13063-019-3442-0 · Cited by 15
Annotating German Clinical Documents for De-Identification
Kolditz, T., Lohr, C., Hellrich, J., et al. (2019) · Studies in health technology and informatics · doi:10.3233/shti190212 · Cited by 14
Levitsky, A., Kisten, Y., Lind, S., et al. (2019) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2018.04.007 · Cited by 12
A neuro-symbolic method for understanding free-text medical evidence.
Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab077 · Cited by 11
Self-citation
Cited by
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Leveraging generative AI for clinical evidence synthesis needs to ensure trustworthiness
Zhang, G., Jin, Q., Jered McInerney, D., Chen, Y., Wang, F., Cole, C. L., Yang, Q., Wang, Y., Malin, B. A., Peleg, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104640 · Cited by 40
Self-citationData extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
Wang, X., Huey, S. L., Sheng, R., Mehta, S., & Wang, F. (2025) · Campbell Systematic Reviews: Better Evidence for a Better World · doi:10.1002/cl2.70073 · Cited by 13
Li, Y., Luan, Z., Liu, Y., Liu, H., Qi, J., & Han, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1454945 · Cited by 9
Metadata
- Authors
- Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., & Weng, C.
- Affiliations
- Columbia University (US)
- Weill Cornell Medicine (US)
- Corresponding author
- Weng, Chunhua
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-03-15
- Volume
- 30
- Issue
- 6
- Pages
- 1022-1031
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10198523
- OpenAlex
- W4324321244
- Cited by
- 14
- References
- 33
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 6 · 2023: 1
- FWCI
- 1.24
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Bridging the semantic gap between research eligibility criteria and clinical data (5R01LM009886-11)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- randomized controlled trials, natural language processing pipeline, structured knowledge representation, evidence synthesis, evidence retrieval, COVID-19 clinical trials, PICO framework, gold standard dataset
- MeSH terms
- COVID-19, Humans, Natural Language Processing, Comprehension, PubMed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.