CoChat
Sign Up
Cited by 142Biomedical Text Mining and OntologiesArtificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

Desiderata for delivering NLP to accelerate healthcare AI advancement and a Mayo Clinic NLP-as-a-service implementation

Open access
Wen, Andrew, Fu, Sunyang, Moon, Sungrim, El Wazir, Mohamed, Rosenbaum, Andrew, Kaggal, Vinod C., Liu, Sijia, Sohn, Sunghwan +2 more·2019·npj Digital Medicine
DOI 10.1038/s41746-019-0208-8PMC PMC6917754
Open this paper in CoChat — free accountOpen PDF

Abstract

Data is foundational to high-quality artificial intelligence (AI). Given that a substantial amount of clinically relevant information is embedded in unstructured data, natural language processing (NLP) plays an essential role in extracting valuable information that can benefit decision making, administration reporting, and research. Here, we share several desiderata pertaining to development and usage of NLP systems, derived from two decades of experience implementing clinical NLP at the Mayo Clinic, to inform the healthcare AI community. Using a framework, we developed as an example implementation, the desiderata emphasize the importance of a user-friendly platform, efficient collection of domain expert inputs, seamless integration with clinical data, and a highly scalable computing infrastructure.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-019-0208-8

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Hypertension, diabetes, myocardial infarction, peripheral artery disease, and atrial fibrillation were determined using a validated natural language processing program to abstract clinical notes.

    “Hypertension, diabetes, myocardial infarction, peripheral artery disease, and atrial fibrillation were determined using a validated natural language processing program to abstract clinical notes.18”

    From Clinician Adoption of an Artificial Intelligence Algorithm to Detect Left Ventricular Systolic Dysfunction in Primary Care. (2022) · cites this paper as [18]

    The Backend and cardiac-sarcoidosis sections (pages unavailable) describe an NLP program that extracts information from clinical documents and identifies clinical concepts in unstructured text. They do not establish that the program was validated to determine hypertension, diabetes, myocardial infarction, peripheral artery disease, or atrial fibrillation from clinical notes. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 35

  1. [1]Nemati S An interpretable machine learning model for accurate prediction of sepsis in the ICU Crit. Care Med. 2018 46 547 553 10.1097/CCM.0000000000002936 29286945 PMC5851825 · DOI · Paper page
  2. [2]Wu CC An artificial intelligence approach to early predict non-ST-elevation myocardial infarction patients with chest pain Comput. Methods Prog. Biomed. 2019 173 109 117 10.1016/j.cmpb.2019.01.013 31046985 · DOI
  3. [3]Lee EJ Kim YH Kim N Kang DW Deep into the brain: artificial intelligence in stroke imaging J. Stroke 2017 19 277 285 10.5853/jos.2017.02054 29037014 PMC5647643 · DOI
  4. [4]Enshaei A Robson CN Edmondson RJ Artificial intelligence systems as prognostic and predictive tools in ovarian cancer Ann. Surg. Oncol. 2015 22 3970 3975 10.1245/s10434-015-4475-6 25752894 · DOI
  5. [5]Wong TY Bressler NM Artificial intelligence with deep learning technology looks into diabetic retinopathy screening JAMA 2016 316 2366 2367 10.1001/jama.2016.17563 27898977 · DOI
  6. [6]Martin-Sanchez F Verspoor K Big data in medicine is driving big changes Yearb. Med Inf. 2014 9 14 20 PMC4287083 25123716
  7. [7]Wang Y Clinical information extraction applications: a literature review J. Biomed. Inform. 2018 77 34 49 10.1016/j.jbi.2017.11.011 29162496 PMC5771858 · DOI · Paper page
  8. [8]Afzal N Mining peripheral arterial disease cases from narrative clinical notes using natural language processing J. Vasc. Surg. 2017 65 1753 1761 10.1016/j.jvs.2016.11.031 28189359 PMC5438905 · DOI · Paper page
  9. [9]Lacson R Evaluation of an automated information extraction tool for imaging data elements to populate a breast cancer screening registry J. Digit. Imaging 2015 28 567 575 10.1007/s10278-014-9762-4 25561069 PMC4570892 · DOI
  10. [10]Jiang F Artificial intelligence in healthcare: past, present and future Stroke Vasc. Neurol. 2017 2 230 243 10.1136/svn-2017-000101 29507784 PMC5829945 · DOI · Paper page
  11. [11]Scheitel M Effect of a novel clinical decision support tool on the efficiency and accuracy of treatment recommendations for cholesterol management Appl. Clin. Inform. 2017 26 124 136 10.4338/ACI-2016-07-RA-0114 PMC5373758 28174820 · DOI
  12. [12]Sohn Sunghwan Wang Yanshan Wi Chung-Il Krusemark Elizabeth A Ryu Euijung Ali Mir H Juhn Young J Liu Hongfang Clinical documentation variations and NLP system portability: a case study in asthma birth cohorts across institutions Journal of the American Medical Informatics Association 2017 25 3 353 359 10.1093/jamia/ocx138 PMC7378885 29202185 · DOI
  13. [13]Zheng K Ease of adoption of clinical natural language processing software: an evaluation of five systems J. Biomed. Inf. 2015 58 Suppl S189 S196 10.1016/j.jbi.2015.07.008 PMC4974203 26210361 · DOI
  14. [14]Afzal N Natural language processing of clinical notes for identification of critical limb ischemia Int. J. Med. Inf. 2018 111 83 89 10.1016/j.ijmedinf.2017.12.024 PMC5808583 29425639 · DOI
  15. [15]Chen D Postoperative bleeding risk prediction for patients undergoing colorectal surgery Surgery 2018 164 1209 1216 10.1016/j.surg.2018.05.043 30033185 PMC6263850 · DOI
  16. [16]Fu S Natural language processing for the identification of silent brain infarcts from neuroimaging reports JMIR Med. Inf. 2019 7 e12109 10.2196/12109 PMC6524454 31066686 · DOI
  17. [17]Kaggal VC Toward a learning health-care system - knowledge delivery at the point of care empowered by big data and NLP Biomed. Inf. Insights 2016 8 13 22 PMC4920204 27385912
  18. [18]Shen F Populating physician biographical pages based on EMR data AMIA Jt. Summits Transl. Sci. Proc. 2017 2017 522 530 28815152 PMC5543344
  19. [19]McPadden J Health care and precision medicine research: analysis of a scalable data science platform J. Med. Internet Res. 2019 21 e13043 10.2196/13043 30964441 PMC6477571 · DOI
  20. [20]Chrimes D Zamani H Using distributed data over HBase in big data analytics pfor clinical services Comput Math. Methods Med. 2017 2017 6120820 10.1155/2017/6120820 29375652 PMC5742497 · DOI
  21. [21]Sun Y Xiong Y Xu Q Wei D A hadoop-based method to predict potential effective drug combination Biomed. Res. Int. 2014 2014 196858 25147789 PMC4134802
  22. [22]Adibuzzaman M DeLaurentis P Hill J Benneyworth BD Big data in healthcare – the promises, challenges and opportunities from a research perspective: A case study with a model database AMIA Annu. Symp. Proc. 2017 2017 384 392 29854102 PMC5977694
  23. [24]Zobel J Moffat A Inverted files for text search engines ACM Comput. Surv. (CSUR) 2006 38 6 10.1145/1132956.1132959 · DOI
  24. [25]Zaharia M Apache spark Commun. ACM 2016 59 56 65 10.1145/2934664 · DOI
  25. [26]Torii M Hu Z Wu CH Liu H BioTagger-GM: a gene/protein name recognition system J. Am. Med. Inf. Assoc. 2009 16 247 255 10.1197/jamia.M2844 PMC2649315 19074302 · DOI
  26. [27]Ferrucci D Lally A UIMA: an architectural approach to unstructured information processing in the corporate research environment Nat. Lang. Eng. 2004 10 327 348 10.1017/S1351324904003523 · DOI
  27. [28]Thusoo A Hive Proc. VLDB Endow. 2009 2 1626 1629 10.14778/1687553.1687609 · DOI
  28. [29]Vavilapalli, V. K. et al. Apache Hadoop YARN: Yet Another Resource Negotiator. In Proceedings of the 4th Annual Symposium on Cloud Computing, 10.1145/2523616.2523633 (2013). · DOI · Paper page
  29. [31]Harkema H Dowling JN Thornblade T Chapman WW ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports J. Biomed. Inf. 2009 42 839 851 10.1016/j.jbi.2009.05.002 PMC2757457 19435614 · DOI
  30. [32]Fanning JP Wong AA Fraser JF The epidemiology of silent brain infarction: a systematic review of population-based cohorts BMC Med. 2014 12 119 10.1186/s12916-014-0119-0 25012298 PMC4226994 · DOI
  31. [33]Fanning JP Wesley AJ Wong AA Fraser JF Emerging spectra of silent brain infarction Stroke 2014 45 3461 3471 10.1161/STROKEAHA.114.005919 25293663 · DOI
  32. [34]Vermeer SE Longstreth WT Jr Koudstaal PJ Silent brain infarcts: a systematic review Lancet Neurol. 2007 6 611 619 10.1016/S1474-4422(07)70170-9 17582361 · DOI · Paper page
  33. [35]Malmasi S Extracting healthcare quality information from unstructured data. American Medical Informatics Association Annual Symposium proceedings AMIA Symp. 2018 2017 1243 1252 PMC5977624 29854193
  34. [36]Afshar M Development and application of a high throughput natural language processing architecture to convert all clinical documents in a clinical data warehouse into standardized medical vocabularies J. Am. Med. Inform. Assoc. 2019 26 1364 1369 10.1093/jamia/ocz068 31145455 PMC7647210 · DOI
  35. [37]Peterson KJ Jiang G Brue SM Shen F Liu H Mining hierarchies and similarity clusters from value set repositories. American Medical Informatics Association Annual Symposium proceedings AMIA Symp. 2018 2017 1372 1381 PMC5977603 29854206

Cited by

Sorted by citations

Authors
Wen, A., Fu, S., Moon, S., El Wazir, M., Rosenbaum, A., Kaggal, V. C., Liu, S., Sohn, S., Liu, H., & Fan, J.
Affiliations
Mayo Clinic (US)
Corresponding author
  • Liu, Hongfang
  • Fan, Jungwei
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2019-12-17
Volume
2
Issue
1
Pages
130-130
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-019-0208-8
PMC
PMC6917754
OpenAlex
W2995349381
Cited by
142
References
0
Citations by year
2026: 8 · 2025: 22 · 2024: 33 · 2023: 29 · 2022: 22 · 2021: 15 · 2020: 13
FWCI
1.10
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
U.S. Department of Health & Human Services | National Institutes of Health (U01TR002062)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
clinical natural language processing, artificial intelligence in healthcare, unstructured clinical data, clinical information extraction, clinical data integration, Mayo Clinic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review