The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
A neuro-symbolic method for understanding free-text medical evidence.
Open accessAbstract
Objective: We introduce Medical evidence Dependency (MD)–informed attention, a novel neuro-symbolic model for understanding free-text clinical trial publications with generalizability and interpretability. Materials and Methods: We trained one head in the multi-head self-attention model to attend to the Medical evidence Ddependency (MD) and to pass linguistic and domain knowledge on to later layers (MD informed). This MD-informed attention model was integrated into BioBERT and tested on 2 public machine reading comprehension benchmarks for clinical trial publications: Evidence Inference 2.0 and PubMedQA. We also curated a small set of recently published articles reporting randomized controlled trials on COVID-19 (coronavirus disease 2019) following the Evidence Inference 2.0 guidelines to evaluate the model’s robustness to unseen data. Results: The integration of MD-informed attention head improves BioBERT substantially in both benchmark tasks—as large as an increase of +30% in the F1 score—and achieves the new state-of-the-art performance on the Evidence Inference 2.0. It achieves 84% and 82% in overall accuracy and F1 score, respectively, on the unseen COVID-19 data. Conclusions: MD-informed attention empowers neural reading comprehension models with interpretability and generalizability via reusable domain knowledge. Its compositionality can benefit any transformer-based architecture for machine reading comprehension of free-text medical evidence.
Overview
CoChat Summary
TL;DR
The paper introduces Medical evidence Dependency (MD)–informed attention, a neuro-symbolic attention mechanism that replaces one self-attention head in BioBERT with a symbolic MD matrix. It improves BioBERT on clinical-trial machine reading comprehension benchmarks and achieves new state-of-the-art performance on Evidence Inference 2.0.
Key findings
- On Evidence Inference 2.0, BioBERT-MDAtt achieved macro-F1 0.843, micro-F1 0.844, and accuracy 0.84, over 4% absolute macro-F1 above previously reported best models; this was almost a +0.30 increase in macro-F1 and accuracy over the BioBERT baseline (0.55 macro-F1, 0.56 accuracy).
- On PubMedQA, BioBERT-MDAtt reached macro-F1 0.482 and accuracy 0.61, compared with BioBERT at 0.311 macro-F1 and 0.53 accuracy, and Final Phase Only at 0.287 macro-F1 and 0.57 accuracy; the reported PubMedQA state of the art was 0.527 macro-F1 using about 200,000 abstracts, while BioBERT-MDAtt used 1,000 abstracts.
- On an unseen COVID-19 dataset of 50 prompt-abstract pairs from 10 RCT abstracts, BioBERT-MDAtt achieved 84% accuracy and 82% F1, with 42 of 50 pairs answered correctly; F1 dropped only slightly from 0.84 on Evidence Inference 2.0 to 0.82.
- With attention masking at p = .4, performance dropped slightly but still outperformed the previous state of the art on Evidence Inference 2.0 (F1 0.819) and improved over BioBERT on PubMedQA (F1 0.463).
Methods
- The model defines Medical evidence Dependency (MD) using PICO elements plus observation and count attributes, with directional relations fixed as Intervention(Comparator) → Observation → Outcome; an MD matrix replaces the scaled Query-Key dot product in one self-attention head of BioBERT, while other heads remain conventional.
- The MD parser treats medical evidence element extraction as named entity recognition and MD parsing as relation extraction, fine-tuning a biomedical BERT; it achieved micro-F1 0.72 for 5-class named entity recognition and 0.92 for extracting MDs among PICO elements.
- Evaluation used Evidence Inference 2.0 and PubMedQA (PQA-L only; 500 QA pairs by 10-fold cross-validation), plus a curated set of 50 prompt-abstract pairs from 10 COVID-19 RCT abstracts; models used BERT-Base with 12 attention heads, maximum sequence length 384, Adam optimizer with learning rate 2e-5, TensorFlow 1.14.0, and 4× NVIDIA GeForce RTX 2080 Ti GPUs.
Limitations
- The paper states that the MD parser can be replaced when a more advanced method or tool is available, that PubMedQA is an extremely low-resource setting with inherent ambiguity in the “maybe” class, and that future work should incorporate MD-informed attention into more advanced models.
- The paper reports that the most common error is misclassifying “no significant difference” as one of the other labels, including failure to reason over an adversative transition in the COVID-19 example.
- Evident from the text: evaluation is limited to two benchmarks plus a small 50-pair COVID-19 set, and the approach replaces only one attention head; parser errors could propagate into the MD matrix, though this is not explicitly quantified.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 15 Not checked · 40 Own results and interpretations
In the paper
“Our baseline model, fine-tuning BioBERT on PQA-L, achieves comparable results to Final Phase Only system from the PubMedQA article.”
§ A neuro-symbolic method for understanding free-text medical evidence > RESULTS AND DISCUSSION > PubMedQA
What the sources say
- Not checked yetcites [40] Jin, 2019
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The integration of MD-informed attention head improves BioBERT substantially in both benchmark tasks—as large as an increase of +30% in the F1 score—and achieves the new state-of-the-art performance on the Evidence Inference 2.0.”
§ A neuro-symbolic method for understanding free-text medical evidence > Abstract > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 1 shows the performance of the model with P = .4. The performance drops slightly compared with the model with the attention mask, but MD-informed attention still outperforms the previous state of the art.”
§ A neuro-symbolic method for understanding free-text medical evidence > RESULTS AND DISCUSSION > Evidence inference 2.0
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 1 shows the performance of the model with P = .4. The performance drops slightly compared with the model with the attention mask, but MD-informed attention still outperforms the previous state of the art.”
§ A neuro-symbolic method for understanding free-text medical evidence > RESULTS AND DISCUSSION > Evidence inference 2.0
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The integration of MD-informed attention head improves BioBERT substantially in both benchmark tasks—as large as an increase of +30% in the F1 score—and achieves the new state-of-the-art performance on the Evidence Inference 2.0.”
§ A neuro-symbolic method for understanding free-text medical evidence > Abstract > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 55 claims and to verify the 16 references still open.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 18
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“PubMed Online PICO interface RCT abstracts”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [99]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A separate practical caveat with relation-extraction models are longer dependencies, i.e. bigger gaps between salient pieces of information in text that lead to a conclusion.”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [99]
Experimental — accuracy not yet measuredIn the citing paper
“This leads to increased complexity of the task and thus to reduced performance. 99”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [99]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A neuro-symbolic method for understanding free-text medical evidence > RESULTS AND DISCUSSION > Evidence inference 2.0
“| PubMedQA | Micro-average | 0.53 | 0.61 | 0.57 | 0.61 | 0.53 | 0.61 | 500 |”
§ A neuro-symbolic method for understanding free-text medical evidence > RESULTS AND DISCUSSION > PubMedQA
“| BioBERT | 0.53 | 0.311 | 0.315 | 0.34 |”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Sackett, D. L. (1997) · Seminars in Perinatology · doi:10.1016/s0146-0005(97)80013-4 · Cited by 1,061
Respiratory rehabilitation in elderly patients with COVID-19: A randomized controlled study
Liu, K., Zhang, W., Yang, Y., Zhang, J., Li, Y., & Chen, Y. (2020) · Complementary Therapies in Clinical Practice · doi:10.1016/j.ctcp.2020.101166 · Cited by 744
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
PubMedQA: A Dataset for Biomedical Research Question AnsweringPreprint
Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.06146 · Cited by 553
Research works printed but not linked: 1
- 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9
Cited by
Neuro-symbolic artificial intelligence: a survey
Bhuyan, B. P., Ramdane-Cherif, A., Tomar, R., & Singh, T. P. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09960-z · Cited by 190
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 74
Neuro-symbolic AI for auditable cognitive information extraction from medical reports
Prenosil, G., Weitzel, T., Bello, S. C., Mingels, C., Manzini, G., Meier, L. P., Shi, K., Rominger, A., & Afshar‐Oromieh, A. (2025) · Communications Medicine · doi:10.1038/s43856-025-01194-x · Cited by 48
A Study on Neuro-Symbolic Artificial Intelligence: Healthcare PerspectivesPreprint
Hossain, D., & Chen, J. Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.18213 · Cited by 45
Metadata
- Authors
- Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C.
- Affiliations
- Columbia University (US)
- Department of Biomedical Informatics, Columbia University, New York, USA
- Corresponding author
- Weng, Chunhua
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-05-06
- Volume
- 28
- Issue
- 8
- Pages
- 1703-1711
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC8135980
- OpenAlex
- W3158862932
- Semantic Scholar
- b5315496f4d6ff1789d0576181bef74027bdb92a
- Cited by
- 11
- References
- 18
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 4 · 2023: 2
- FWCI
- 0.47
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Bridging the semantic gap between research eligibility criteria and clinical data (5R01LM009886-11)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- neuro-symbolic models, clinical trial publication, machine reading comprehension, BioBERT, PubMedQA, interpretability, generalizability, domain knowledge integration
- MeSH terms
- COVID-19, Artificial Intelligence, Clinical Trials as Topic, Computer Simulation, Humans, Models, Neurological, Natural Language Processing, Software, Information Storage and Retrieval, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A neuro-symbolic method for understanding free-text medical evidence. https://app.cochat.ai/papers/10.1093/jamia/ocab077
- The paper
- Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C. (2021). A neuro-symbolic method for understanding free-text medical evidence. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocab077
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.