CoChat

Randomized controlled trial of an automated problem list with improved sensitivity

Meystre, Stéphane M., Haug, Peter J. · 2008 · International Journal of Medical Informatics
Cited by 33

The DOI matches its Crossref record.

Abstract

Purpose To improve the completeness and timeliness of an electronic problem list, we have developed a system using Natural Language Processing (NLP) to automatically extract potential medical problems from clinical, free-text documents; these problems are then proposed for inclusion in an electronic problem list management application. Methods A prospective randomized controlled evaluation of the Automatic Problem List (APL) system in an intensive care unit and in a cardiovascular surgery unit is reported here. A total of 247 patients were enrolled: 76 in an initial control phase and 171 in the randomized controlled trial that followed. During this latter phase, patients were randomly assigned to a control or an intervention group. All patients had their documents analyzed by the system, but the medical problems discovered were only proposed in the problem list for intervention patients. We measured the sensitivity, specificity, positive and negative predictive values, likelihood ratios and the timeliness of the problem lists. Results Our system significantly increased the sensitivity of the problem lists in the intensive care unit, from about 9% to 41%, and even 77% if problems automatically proposed but not acknowledged by users were also considered. Timeliness of addition of problems to the list was greatly improved, with a time between a problem's first mention in a clinical document and its addition to the problem list reduced from about 6 days to less than 2 days. No si…

Overview

Integrity signals

Registry

Crossref · 10.1016/j.ijmedinf.2007.12.001

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 31

  1. Bayegan, 2002, The helpful patient record system: problem oriented and knowledge based, Proc. AMIA Symp., 36
  2. Clayton, 2003, Building a comprehensive clinical information system from components. The approach at Intermountain Health Care, Methods Inf. Med., 42, 1, 10.1055/s-0038-1634203 · DOI · Paper page
  3. Meystre, 2006, Natural language processing to extract medical problems from electronic clinical documents: performance evaluation, J Biomed. Inform., 39, 589, 10.1016/j.jbi.2005.11.004 · DOI · Paper page
  4. Meystre, 2005, Automation of a problem list using natural language processing, BMC Med. Inform. Decis. Mak., 5, 30, 10.1186/1472-6947-5-30 · DOI · Paper page
  5. Weed, 1968, Medical records that guide and teach, N. Engl. J. Med., 278, 593, 10.1056/NEJM196803142781105 · DOI · Paper page
  6. Weed, 1968, Medical records that guide and teach, N. Engl. J. Med., 278, 652, 10.1056/NEJM196803212781204 · DOI · Paper page
  7. Campbell, 1998, Strategies for problem list implementation in a complex clinical enterprise, Proc. AMIA Symp., 285
  8. Campbell, 1994, A comparison of four schemes for codification of problem lists, Proc. Annu. Symp. Comput. Appl. Med. Care, 201
  9. Donaldson, 1985, Improving the master problem list: a case study in changing clinician behavior, QRB Qual. Rev. Bull., 11, 327
  10. Elkin, 1997, Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System, Proc. AMIA Annu. Fall Symp., 500
  11. Goldberg, 1998, An evaluation of UMLS as a controlled terminology for the Problem List Toolkit, Medinfo, 9, 609
  12. Hales, 1998, Extracting medical knowledge for a coded problem list vocabulary from the UMLS Knowledge Sources, Proc. AMIA Symp., 275
  13. Starmer, 1998, Development of a structured problem list management system at vanderbilt, Proc. AMIA Annu. Fall Symp., 1083
  14. Simborg, 1976, Information factors affecting problem follow-up in ambulatory care, Med. Care, 14, 848, 10.1097/00005650-197610000-00005 · doi:10.1097/00005650-197610000-00005
  15. Starfield, 1979, Concordance between medical records and observations regarding information on coordination of care, Med. Care, 17, 758, 10.1097/00005650-197907000-00006 · doi:10.1097/00005650-197907000-00006
  16. Institute of Medicine (U.S.), Committee on Improving the Patient Record, R.S. Dick, E.B. Steen, D.E. Detmer, The Computer-based Patient Record: An Essential Technology for Health Care, Rev. ed., National Academy Press, Washington, DC, 1997.
  17. van Ginneken, 2002, The computerized patient record: balancing effort and benefit, Int. J. Med. Inform., 65, 97, 10.1016/S1386-5056(02)00007-2 · DOI
  18. A.A. Bui, R.K. Taira, S. El-Saden, A. Dordoni, D.R. Aberle, Automated medical problem list generation: towards a patient timeline, Medinfo, San Francisco, CA, 2004, pp. 587–591.
  19. Scherpbier, 1994, A simple approach to physician entry of patient problem list, Proc. Annu. Symp. Comput. Appl. Med. Care, 206
  20. Wasserman, 2003, An applied evaluation of SNOMED CT as a clinical vocabulary for the computerized diagnosis and problem list, Proc. AMIA Symp., 699
  21. Payne, 1993, How useful is the UMLS metathesaurus in developing a controlled vocabulary for an automated problem list?, Proc. Annu. Symp. Comput. Appl. Med. Care, 705
  22. Zelingher, 1995, Categorization of free-text problem lists: an effective method of capturing clinical data, Proc. Annu. Symp. Comput. Appl. Med. Care, 416
  23. Warren, 1998, Just-in-time coding of the problem list in a clinical environment, Proc. AMIA Symp., 280
  24. Wang, 2003, Automated coded ambulatory problem lists: evaluation of a vocabulary and a data entry tool, Int. J. Med. Inform., 72, 17, 10.1016/j.ijmedinf.2003.08.002 · DOI
  25. Pratt, 1973, Medicine, computers, and linguistics, Adv. Biomed. Eng., 3, 97, 10.1016/B978-0-12-004903-5.50007-8 · DOI
  26. Spyns, 1996, Natural language processing in medicine: an overview, Methods Inf. Med., 35, 285
  27. Reichert, 2002, Using LOINC to link an EMR to the pertinent paragraph in a structured reference knowledge base, Proc. AMIA Symp., 652
  28. Aronson, 2001, Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program, Proc. AMIA Symp., 17
  29. Aronson, 2000, The NLM indexing initiative, Proc. AMIA Symp., 17
  30. Ashton, 1999, An empirical assessment of the validity of explicit and implicit process-of-care criteria for quality assessment, Med. Care, 37, 798, 10.1097/00005650-199908000-00009 · DOI · Paper page
  31. Szeto, 2002, Accuracy of computerized outpatient diagnoses in a Veterans Affairs general medicine clinic, Am. J. Manage. Care, 8, 37

Cited by

See all 33

Metadata

Authors
Meystre, S. M., & Haug, P. J.
Venue
International Journal of Medical Informatics
Publisher
Elsevier BV
Published
2008-09
Volume
77
Issue
9
Pages
602-612
ISSN
1386-5056
Cited by
33
References
0
Language
en
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Randomized controlled trial of an automated problem list with improved sensitivity. https://app.cochat.ai/papers/10.1016/j.ijmedinf.2007.12.001
The paper
Meystre, S. M., & Haug, P. J. (2008). Randomized controlled trial of an automated problem list with improved sensitivity. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2007.12.001

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.