CoChat
Sign Up
Cited by 16Biomedical Text Mining and OntologiesMachine Learning in HealthcareElectronic Health Records Systems

Representing and utilizing clinical textual data for real world studies: An OHDSI approach

Open access
Keloth, Vipina K., Banda, Juan M., Gurley, Michael, Heider, Paul M., Kennedy, Georgina, Liu, Hongfang, Liu, Feifan, Miller, Timothy +19 more·2023·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2023.104343PMC PMC10428170
Open this paper in CoChat — free account

Abstract

Clinical documentation in electronic health records contains crucial narratives and details about patients and their care. Natural language processing (NLP) can unlock the information conveyed in clinical notes and reports, and thus plays a critical role in real-world studies. The NLP Working Group at the Observational Health Data Sciences and Informatics (OHDSI) consortium was established to develop methods and tools to promote the use of textual data and NLP in real-world observational studies. In this paper, we describe a framework for representing and utilizing textual data in real-world evidence generation, including representations of information from clinical text in the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM), the workflow and tools that were developed to extract, transform and load (ETL) data from clinical notes into tables in OMOP CDM, as well as current applications and specific use cases of the proposed OHDSI NLP solution at large consortia and individual institutions with English textual data. Challenges faced and lessons learned during the process are also discussed to provide valuable insights for researchers who are planning to implement NLP solutions in real-world studies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2023.104343

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most current studies do not use the full spectrum of EHR data, and more efforts are being devoted to including unstructured textual data in EHRs for real world evidence generation.

    “However, most current studies do not use the full spectrum of EHR data, and more efforts are being devoted to including unstructured textual data in EHRs for real world evidence generation14.”

    From Social determinants of health extraction from clinical notes across institutions using large language models (2025) · cites this paper as [14]

    The Introduction (page not provided) describes efforts to incorporate clinical text into observational studies, and the Conclusion (page not provided) describes a framework for using textual data to generate real-world evidence. The paper does not establish the claim that most current studies fail to use the full spectrum of EHR data. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. [1]Corrigan-Curay J, Sacks L, Woodcock J. Real-world evidence and real-world data for evaluating drug safety and effectiveness. Jama. 2018; 320(9): 867– 8. 30105359
  2. [2]Baumfeld Andre E, Reynolds R, Caubel P, Azoulay L, Dreyer NA. Trial designs using real-world data: the changing landscape of the regulatory approval process. Pharmacoepidemiology and Drug Safety. 2020; 29(10): 1201– 12. 31823482 PMC7687110
  3. [3]Skovlund E, Leufkens H, Smyth J. The use of real-world data in cancer drug development. European Journal of Cancer. 2018; 101: 69– 76. 30031168
  4. [4]Trojano M, Tintore M, Montalban X, Hillert J, Kalincik T, Iaffaldano P, Treatment decisions in multiple sclerosis—insights from real-world observational studies. Nature Reviews Neurology. 2017; 13(2): 105– 18. 28084327
  5. [6]Sherman RE, Anderson SA, Dal Pan GJ, Gray GW, Gross T, Hunter NL, Real-world evidence—what is it and what can it tell us? New England Journal of Medicine. 2016; 375(23): 2293– 7. 27959688
  6. [7]Patorno E, Gopalakrishnan C, Franklin JM, Brodovicz KG, Masso-Gonzalez E, Bartels DB, Claims-based studies of oral glucose-lowering medications can achieve balance in critical clinical variables only observed in electronic health records. Diabetes, Obesity and Metabolism. 2018; 20(4): 974– 84. PMC6207375 29206336
  7. [8]Richesson RL, Hammond WE, Nahm M, Wixted D, Simon GE, Robinson JG, Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory. Journal of the American Medical Informatics Association. 2013; 20(e2): e226– e31. 23956018 PMC3861929
  8. [9]Khozin S, Blumenthal GM, Pazdur R. Real-world data for clinical evidence generation in oncology. JNCI: Journal of the National Cancer Institute. 2017; 109(11): djx187. 29059439
  9. [10]Cinelli M, Quattrociocchi W, Galeazzi A, Valensise CM, Brugnoli E, Schmidt AL, The COVID-19 social media infodemic. Scientific reports. 2020; 10(1): 1– 10. 33024152 PMC7538912
  10. [11]Ates HC, Yetisen AK, Güder F, Dincer C. Wearable devices for the detection of COVID-19. Nature Electronics. 2021; 4(1): 13– 4.
  11. [12]Jeon J, Baruah G, Sarabadani S, Palanica A. Identification of risk factors and symptoms of COVID-19: Analysis of biomedical literature and social media data. Journal of medical Internet research. 2020; 22(10): e20509. 32936770 PMC7537723
  12. [14]Klann JG, Abend A, Raghavan VA, Mandl KD, Murphy SN. Data interchange using i2b2. Journal of the American Medical Informatics Association. 2016; 23(5): 909– 15. 26911824 PMC4997035
  13. [16]Toh S, Rasmussen-Torvik LJ, Harmata EE, Pardee R, Saizan R, Malanga E, The National Patient-Centered Clinical Research Network (PCORnet) bariatric study cohort: rationale, methods, and baseline characteristics. JMIR research protocols. 2017; 6(12): e8323. PMC5736875 29208590
  14. [17]Platt R, Carnahan RM, Brown JS, Chrischilles E, Curtis LH, Hennessy S, The US Food and Drug Administration’s Mini-Sentinel program: status and direction. Pharmacoepidemiology and drug safety. 2012; 21: 1– 8. 22262586
  15. [21]Velupillai S, Suominen H, Liakata M, Roberts A, Shah AD, Morley K, Using clinical natural language processing for health outcomes research: overview and actionable suggestions for future advances. Journal of biomedical informatics. 2018; 88: 11– 9. 30368002 PMC6986921
  16. [22]Si Y, Wang J, Xu H, Roberts K. Enhancing clinical concept extraction with contextual embeddings. Journal of the American Medical Informatics Association. 2019; 26(11): 1297– 304. 31265066 PMC6798561
  17. [23]Savova GK, Danciu I, Alamudun F, Miller T, Lin C, Bitterman DS, Use of natural language processing to extract clinical cancer phenotypes from electronic medical records. Cancer research. 2019; 79(21): 5463– 70. 31395609 PMC7227798
  18. [24]Friedman C, Hripcsak G, DuMouchel W, Johnson SB, Clayton PD. Natural language processing in an operational clinical information system. Natural Language Engineering. 1995; 1(1): 83– 108.
  19. [25]Aronson AR, Lang F-M. An overview of MetaMap: historical perspective and recent advances. Journal of the American Medical Informatics Association. 2010; 17(3): 229– 36. 20442139 PMC2995713
  20. [26]Demner-Fushman D, Rogers WJ, Aronson AR. MetaMap Lite: an evaluation of a new Java implementation of MetaMap. Journal of the American Medical Informatics Association. 2017; 24(4): 841– 4. 28130331 PMC6080672
  21. [27]Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informatics Association. 2010; 17(5): 507– 13. 20819853 PMC2995668
  22. [28]Soysal E, Wang J, Jiang M, Wu Y, Pakhomov S, Liu H, CLAMP–a toolkit for efficiently building customized clinical natural language processing pipelines. Journal of the American Medical Informatics Association. 2018; 25(3): 331– 6. 29186491 PMC7378877
  23. [31]McCarty CA, Chisholm RL, Chute CG, Kullo IJ, Jarvik GP, Larson EB, The eMERGE Network: a consortium of biorepositories linked to electronic medical records data for conducting genomic studies. BMC medical genomics. 2011; 4(1): 1– 11. 21269473 PMC3038887
  24. [35]Cronin RM, Jerome RN, Mapes B, Andrade R, Johnston R, Ayala J, Development of the initial surveys for the All of Us Research Program. Epidemiology (Cambridge, Mass). 2019; 30(4): 597. 31045611 PMC6548672
  25. [36]Haendel MA, Chute CG, Bennett TD, Eichmann DA, Guinney J, Kibbe WA, The National COVID Cohort Collaborative (N3C): rationale, design, infrastructure, and deployment. Journal of the American Medical Informatics Association. 2021; 28(3): 427– 43. 32805036 PMC7454687
  26. [37]Liu H, Bielinski SJ, Sohn S, Murphy S, Wagholikar KB, Jonnalagadda SR, An information extraction framework for cohort identification using electronic health records. AMIA Summits on Translational Science Proceedings. 2013; 2013: 149. PMC3845757 24303255
  27. [38]Liu S, Wen A, Wang L, He H, Fu S, Miller R, An Open Natural Language Processing Development Framework for EHR-based Clinical Research: A case demonstration using the National COVID Cohort Collaborative (N3C). arXiv preprint arXiv:211010780. 2021. PMC10654844 37555837
  28. [39]Lynch KE, Deppen SA, DuVall SL, Viernes B, Cao A, Park D, Incrementally transforming electronic medical records into the observational medical outcomes partnership common data model: a multidimensional quality assurance approach. Applied clinical informatics. 2019; 10(05): 794– 803. 31645076 PMC6811349
  29. [40]FitzHenry F, Patterson OV, Denton J, Brannen J, Reeves RM, DuVall SL, OMOP CDM for Natural Language Processing: Piloting a VA NLP Data Set. OHDSI Conference; 2017.
  30. [41]Shi J, Mowery D, Zhang M, Sanders J, Chapman W, Gawron L. Extracting intrauterine device usage from clinical texts using natural language processing. 2017 IEEE International Conference on Healthcare Informatics (ICHI); 2017: IEEE.
  31. [42]Johnson SA, Signor EA, Lappe KL, Shi J, Jenkins SL, Wikstrom SW, A comparison of natural language processing to ICD-10 codes for identification and characterization of pulmonary embolism. Thrombosis Research. 2021; 203: 190– 5. 34044246
  32. [43]Hirsch JS, Tanenbaum JS, Lipsky Gorman S, Liu C, Schmitz E, Hashorva D, HARVEST, a longitudinal patient record summarizer. Journal of the American Medical Informatics Association. 2015; 22(2): 263– 74. 25352564 PMC4394965
  33. [45]Shang N, Liu C, Rasmussen LV, Ta CN, Caroll RJ, Benoit B, Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network. Journal of biomedical informatics. 2019; 99: 103293. 31542521 PMC6894517
  34. [46]Zachariah P, Hill-Ricciuti A, Saiman L, Natarajan K. Using the “Who, What, and When” of free text documentation to improve hospital infectious disease surveillance. American Journal of Infection Control. 2020; 48(10): 1261– 3. 32070629
  35. [48]Fan Y, Zhang R. Using natural language processing methods to classify use status of dietary supplements in clinical notes. BMC medical informatics and decision making. 2018; 18(2): 15– 22. 30066648 PMC6069512
  36. [49]Fan Y, Zhou S, Li Y, Zhang R. Deep learning approaches for extracting adverse events and indications of dietary supplements from clinical text. Journal of the American Medical Informatics Association. 2021; 28(3): 569– 77. 33150942 PMC7936508
  37. [50]Mohammad HA, Sivarajkumar S, Viggiano S, Oniani D, Visweswaran S, Wang Y. Extraction of Sleep Information from Clinical Notes of Alzheimer’s Disease Patients Using Natural Language Processing. medRxiv. 2022. PMC11413436 39001795
  38. [52]Honnibal M, Johnson M. An improved non-monotonic transition system for dependency parsing. Proceedings of the 2015 conference on empirical methods in natural language processing; 2015.
  39. [53]Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Transformers: State-of-the-art natural language processing. Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations; 2020.
  40. [54]Heider PM, Pipaliya RM, Meystre SM. A Natural Language Processing Tool Offering Data Extraction for COVID-19 Related Information (DECOVRI). MEDINFO 2021: The 18th World Congress on Medical and Health Informatics; 2021. 35673206
  41. [55]Ferrucci D, Lally A. UIMA: an architectural approach to unstructured information processing in the corporate research environment. Natural Language Engineering. 2004; 10(3–4): 327– 48.
  42. [56]Eyre H, Chapman AB, Peterson KS, Shi J, Alba PR, Jones MM, Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python. arXiv preprint arXiv:210607799. 2021. PMC8861690 35308962
  43. [60]Paris N, Parrot A. MIMIC in the OMOP Common Data Model. medRxiv. 2020. PMC8715361 34904958
  44. [61]Ryu B, Yoon E, Kim S, Lee S, Baek H, Yi S, Transformation of pathology reports into the common data model with oncology module: use case for colon cancer. Journal of medical Internet research. 2020; 22(12): e18526. 33295294 PMC7758167
  45. [62]Sharma H, Mao C, Zhang Y, Vatani H, Yao L, Zhong Y, Developing a portable natural language processing based phenotyping system. BMC Medical Informatics and Decision Making. 2019; 19(3): 79– 87. 30943974 PMC6448187
  46. [63]Datta S, Posada J, Olson G, Li W, O’Reilly C, Balraj D, A new paradigm for accelerating clinical data science at Stanford Medicine. arXiv preprint arXiv:200310534. 2020.
  47. [64]Zuo X, Li J, Zhao B, Zhou Y, Dong X, Duke J, Normalizing Clinical Document Titles to LOINC Document Ontology: An Initial Study. AMIA Annual Symposium Proceedings; 2020: American Medical Informatics Association. PMC8075502 33936520
  48. [66]Tao S, Cui L, Wu X, Zhang G-Q. Facilitating cohort discovery by enhancing ontology exploration, query management and query sharing for large clinical data repositories. AMIA Annual Symposium Proceedings; 2017: American Medical Informatics Association. PMC5977665 29854239
  49. [67]Huang Y, Li X, Zhang G-Q. ELII: A novel inverted index for fast temporal query, with application to a large Covid-19 EHR dataset. Journal of Biomedical Informatics. 2021; 117: 103744. 33775815 PMC9759789
  50. [68]Callahan A, Polony V, Posada JD, Banda JM, Gombar S, Shah NH. ACE: the Advanced Cohort Engine for searching longitudinal patient records. Journal of the American Medical Informatics Association. 2021; 28(7): 1468– 79. 33712854 PMC8279796
  51. [70]Banda JM, Halpern Y, Sontag D, Shah NH. Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network. AMIA Summits on Translational Science Proceedings. 2017; 2017: 48. PMC5543379 28815104
  52. [71]Agarwal V, Podchiyska T, Banda JM, Goel V, Leung TI, Minty EP, Learning statistical models of phenotypes using noisy labeled training data. Journal of the American Medical Informatics Association. 2016; 23(6): 1166– 73. 27174893 PMC5070523
  53. [72]Halpern Y, Horng S, Choi Y, Sontag D. Electronic medical record phenotyping using the anchor and learn framework. Journal of the American Medical Informatics Association. 2016; 23(4): 731– 40. 27107443 PMC4926745
  54. [73]Zeng Z, Deng Y, Li X, Naumann T, Luo Y. Natural language processing for EHR-based computational phenotyping. IEEE/ACM transactions on computational biology and bioinformatics. 2018; 16(1): 139– 53. 29994486 PMC6388621
  55. [74]Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products [ cited 2022 May 12]. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/real-world-data-assessing-electronic-health-records-and-medical-claims-data-support-regulatory. PMC9320939 35471704
  56. [75]Digan W, Névéol A, Neuraz A, Wack M, Baudoin D, Burgun A, Can reproducibility be improved in clinical natural language processing? A study of 7 clinical NLP suites. Journal of the American Medical Informatics Association. 2021; 28(3): 504– 15. 33319904 PMC7936396
  57. [76]Stubbs A, Kotfila C, Uzuner Ö. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1. Journal of biomedical informatics. 2015; 58: S11– S9. 26225918 PMC4989908
  58. [77]Luo C, Islam M, Sheils NE, Buresh J, Reps J, Schuemie MJ, DLMM as a lossless one-shot algorithm for collaborative multi-site distributed linear mixed models. Nature communications. 2022; 13(1): 1– 10. PMC8967932 35354802

Cited by

Sorted by citations

Authors
Keloth, V. K., Banda, J. M., Gurley, M., Heider, P. M., Kennedy, G., Liu, H., Liu, F., Miller, T., Natarajan, K., V Patterson, O., Peng, Y., Raja, K., Reeves, R. M., Rouhizadeh, M., Shi, J., Wang, X., Wang, Y., Wei, W.-Q., Williams, A. E., Zhang, R., Belenkaya, R., Reich, C., Blacketer, C., Ryan, P., Hripcsak, G., Elhadad, N., & Xu, H.
Affiliations
  • Yale University (US)
  • Georgia State University (US)
  • Northwestern University (US)
  • Medical University of South Carolina (US)
  • Ingham Institute (AU)
  • Mayo Clinic (US)
  • University of Massachusetts Chan Medical School (US)
  • Boston Children's Hospital (US)
  • Columbia University Irving Medical Center (US)
  • Weill Cornell Medicine (US)
  • United States Department of Veterans Affairs (US)
  • Johns Hopkins University (US)
  • University of Utah (US)
  • Sema4 (United States) (US)
  • University of Pittsburgh (US)
  • Vanderbilt University Medical Center (US)
  • Tufts University (US)
  • University of Minnesota (US)
  • Memorial Sloan Kettering Cancer Center (US)
  • IQVIA (United States) (US)
  • Erasmus MC (NL)
Corresponding author
Xu, Hua
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2023-06
Volume
142
Pages
104343
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2023.104343
PMC
PMC10428170
OpenAlex
W4327703607
Cited by
16
References
2
Citations by year
2026: 4 · 2025: 3 · 2024: 9
FWCI
1.38
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
  • PubMed Central (acceptedVersion)
  • Digital Commons-TMC (Texas Medical Center) (submittedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
Funders
  • Patient-Centered Outcomes Research Institute (ME-2018C3-14549, UL1TR002494)
  • National Institutes of Health (UL1TR002494, U01TR002062)
  • National Institute on Aging (P30AG059307, P30AG066515)
  • National Cancer Institute (P30CA008748)
  • U.S. National Library of Medicine (R00LM013001, LM006910, R01LM006910)
  • National Center for Advancing Translational Sciences
  • National Center for Complementary and Integrative Health (R01AT009457)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
clinical text, natural language processing, real-world evidence, OMOP Common Data Model, clinical notes, clinical documentation
MeSH terms
Data Science, Humans, Medical Informatics, Natural Language Processing, Narration, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review