Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.

A neuro-symbolic method for understanding free-text medical evidence.

Open access
Kang, Tian, Turfah, Ali, Kim, Jaehyun, Perotte, Adler, Weng, Chunhua · 2021 · Journal of the American Medical Informatics Association

Abstract

Objective: We introduce Medical evidence Dependency (MD)–informed attention, a novel neuro-symbolic model for understanding free-text clinical trial publications with generalizability and interpretability. Materials and Methods: We trained one head in the multi-head self-attention model to attend to the Medical evidence Ddependency (MD) and to pass linguistic and domain knowledge on to later layers (MD informed). This MD-informed attention model was integrated into BioBERT and tested on 2 public machine reading comprehension benchmarks for clinical trial publications: Evidence Inference 2.0 and PubMedQA. We also curated a small set of recently published articles reporting randomized controlled trials on COVID-19 (coronavirus disease 2019) following the Evidence Inference 2.0 guidelines to evaluate the model’s robustness to unseen data. Results: The integration of MD-informed attention head improves BioBERT substantially in both benchmark tasks—as large as an increase of +30% in the F1 score—and achieves the new state-of-the-art performance on the Evidence Inference 2.0. It achieves 84% and 82% in overall accuracy and F1 score, respectively, on the unseen COVID-19 data. Conclusions: MD-informed attention empowers neural reading comprehension models with interpretability and generalizability via reusable domain knowledge. Its compositionality can benefit any transformer-based architecture for machine reading comprehension of free-text medical evidence.

Overview

CoChat Summary

TL;DR

The paper introduces Medical evidence Dependency (MD)–informed attention, a neuro-symbolic attention mechanism that replaces one self-attention head in BioBERT with a symbolic MD matrix. It improves BioBERT on clinical-trial machine reading comprehension benchmarks and achieves new state-of-the-art performance on Evidence Inference 2.0.

Key findings

  • On Evidence Inference 2.0, BioBERT-MDAtt achieved macro-F1 0.843, micro-F1 0.844, and accuracy 0.84, over 4% absolute macro-F1 above previously reported best models; this was almost a +0.30 increase in macro-F1 and accuracy over the BioBERT baseline (0.55 macro-F1, 0.56 accuracy).
  • On PubMedQA, BioBERT-MDAtt reached macro-F1 0.482 and accuracy 0.61, compared with BioBERT at 0.311 macro-F1 and 0.53 accuracy, and Final Phase Only at 0.287 macro-F1 and 0.57 accuracy; the reported PubMedQA state of the art was 0.527 macro-F1 using about 200,000 abstracts, while BioBERT-MDAtt used 1,000 abstracts.
  • On an unseen COVID-19 dataset of 50 prompt-abstract pairs from 10 RCT abstracts, BioBERT-MDAtt achieved 84% accuracy and 82% F1, with 42 of 50 pairs answered correctly; F1 dropped only slightly from 0.84 on Evidence Inference 2.0 to 0.82.
  • With attention masking at p = .4, performance dropped slightly but still outperformed the previous state of the art on Evidence Inference 2.0 (F1 0.819) and improved over BioBERT on PubMedQA (F1 0.463).

Methods

  • The model defines Medical evidence Dependency (MD) using PICO elements plus observation and count attributes, with directional relations fixed as Intervention(Comparator) → Observation → Outcome; an MD matrix replaces the scaled Query-Key dot product in one self-attention head of BioBERT, while other heads remain conventional.
  • The MD parser treats medical evidence element extraction as named entity recognition and MD parsing as relation extraction, fine-tuning a biomedical BERT; it achieved micro-F1 0.72 for 5-class named entity recognition and 0.92 for extracting MDs among PICO elements.
  • Evaluation used Evidence Inference 2.0 and PubMedQA (PQA-L only; 500 QA pairs by 10-fold cross-validation), plus a curated set of 50 prompt-abstract pairs from 10 COVID-19 RCT abstracts; models used BERT-Base with 12 attention heads, maximum sequence length 384, Adam optimizer with learning rate 2e-5, TensorFlow 1.14.0, and 4× NVIDIA GeForce RTX 2080 Ti GPUs.

Limitations

  • The paper states that the MD parser can be replaced when a more advanced method or tool is available, that PubMedQA is an extremely low-resource setting with inherent ambiguity in the “maybe” class, and that future work should incorporate MD-informed attention into more advanced models.
  • The paper reports that the most common error is misclassifying “no significant difference” as one of the other labels, including failure to reason over an adversative transition in the COVID-19 example.
  • Evident from the text: evaluation is limited to two benchmarks plus a small 50-pair COVID-19 set, and the approach replaces only one attention head; parser errors could propagate into the MD matrix, though this is not explicitly quantified.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/jamia/ocab077

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 15 Not checked · 40 Own results and interpretations

18 more claims

Create a free account to see all 55 claims and to verify the 16 references still open.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 18

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 18 references

Research works printed but not linked: 1

  1. 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9

Cited by

See all 18

Metadata

Authors
Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C.
Affiliations
  • Columbia University (US)
  • Department of Biomedical Informatics, Columbia University, New York, USA
Corresponding author
Weng, Chunhua
Venue
Journal of the American Medical Informatics Association
Published
2021-05-06
Volume
28
Issue
8
Pages
1703-1711
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W3158862932
Cited by
11
References
18
Citations by year
2026: 2 · 2025: 3 · 2024: 4 · 2023: 2
FWCI
0.47
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
Bridging the semantic gap between research eligibility criteria and clinical data (5R01LM009886-11)
Topics
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
neuro-symbolic models, clinical trial publication, machine reading comprehension, BioBERT, PubMedQA, interpretability, generalizability, domain knowledge integration
MeSH terms
COVID-19, Artificial Intelligence, Clinical Trials as Topic, Computer Simulation, Humans, Models, Neurological, Natural Language Processing, Software, Information Storage and Retrieval, Data Mining

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A neuro-symbolic method for understanding free-text medical evidence. https://app.cochat.ai/papers/10.1093/jamia/ocab077
The paper
Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C. (2021). A neuro-symbolic method for understanding free-text medical evidence. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocab077

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.