CoChat
Sign Up
Cited by 42Machine Learning in HealthcareArtificial Intelligence in HealthcareElectronic Health Records Systems

Healthcare Process Modeling to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals): Development and evaluation of a conceptual framework

Open access
Rossetti, Sarah Collins, Knaplund, Chris, Albers, Dave, Dykes, Patricia C, Kang, Min Jeoung, Korach, Tom Z, Zhou, Li, Schnock, Kumiko +6 more·2021·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocab006PMC PMC8200261
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: There are signals of clinicians’ expert and knowledge-driven behaviors within clinical information systems (CIS) that can be exploited to support clinical prediction. Describe development of the Healthcare Process Modeling Framework to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals). Materials and Methods: We employed an iterative framework development approach that combined data-driven modeling and simulation testing to define and refine a process for phenotyping clinician behaviors. Our framework was developed and evaluated based on the Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) predictive model to detect and leverage signals of clinician expertise for prediction of patient trajectories. Results: Seven themes—identified during development and simulation testing of the CONCERN model—informed framework development. The HPM-ExpertSignals conceptual framework includes a 3-step modeling technique: (1) identify patterns of clinical behaviors from user interaction with CIS; (2) interpret patterns as proxies of an individual’s decisions, knowledge, and expertise; and (3) use patterns in predictive models for associations with outcomes. The CONCERN model differentiated at risk patients earlier than other early warning scores, lending confidence to the HPM-ExpertSignals framework. Discussion: The HPM-ExpertSignals framework moves beyond transactional data analytics to model clinical knowledge, decision making, and CIS interactions, which can support predictive modeling with a focus on the rapid and frequent patient surveillance cycle. Conclusions: We propose this framework as an approach to embed clinicians’ knowledge-driven behaviors in predictions and inferences to facilitate capture of healthcare processes that are activated independently, and sometimes well before, physiological changes are apparent.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocab006

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI is able to interpret nursing documentation to recognize patient types and predict clinical deterioration.

    “AI is able to interpret nursing documentation to recognize patient types and predict clinical deterioration.43–47”

    From Potential uses of AI for perioperative nursing handoffs: a qualitative study (2023) · cites this paper as [46]

    The Methods section (page unavailable) says the CONCERN model uses natural language processing of nursing notes and comments and assigns patients low, moderate, or high risk scores. The Results section (page unavailable) reports that those scores predict adverse clinical events. The paper supports interpreting nursing documentation to predict deterioration, but it describes risk categories and phenotypes of clinician behavior, not recognition of patient types. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Nurse concern has been shown to be a valid and frequent reason for calling a rapid response.

    “nurse concern has been shown to be a valid and frequent reason for calling a rapid response10.”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [10]

    In Materials and Methods (CONCERN model; page unavailable), the paper says a nurse’s concern is a valid reason to intervene without waiting for physiological changes. It does not say that nurse concern is a frequent reason for calling a rapid response team. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Lack of shared team situational awareness delays medical interventions.

    “Unfortunately, when there is a lack of shared team situational awareness, medical interventions are delayed10,12–14.”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [10]

    The Discussion says responding to patient deterioration depends on effective team communication and activating interventions. The Methods section describes nurses’ concerns that are rarely viewed and a tool intended to increase team situational awareness. The paper does not establish that a lack of shared situational awareness delays medical interventions; it proposes a way to address that risk. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • CONCERN EWS was developed by an interdisciplinary team using rigorous methods for translating a predictive model to a clinical setting, including input on health care process effects, integration with workflows, transparency and explainability, and robust evaluation for external validity and model fairness.

    “CONCERN EWS was developed by an interdisciplinary team that employed rigorous methods for translating a predictive model to a clinical setting, including nurse and prescribing-provider input on health care process effects and real-time data availability, integration with existing clinical workflows, transparency and explainability to gain trust10,45, as well as robust evaluation for external”

    From Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial (2025) · cites this paper as [10]

    The Materials and Methods describe iterative model refinement with clinical experts, simulation feedback from nurses and physicians, and a user-centered tool designed not to require new nursing workflows. They also report performance evaluation and corrections for demographic biases. The paper does not establish the claimed robust evaluation for external validity or model fairness, or specifically document transparency and explainability measures to gain trust; it describes broader applicability as future work (Discussion; page numbers unavailable). (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • CONCERN leverages evidence that nurses alter their documentation behavior or use selected language in their narrative notes when they are concerned about a patient's changing clinical state.

    “As such, it leverages evidence that nurses alter their documentation behavior or use selected language in their narrative notes when they are concerned about a patient’s changing clinical state [21-24].”

    From Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study (2022) · cites this paper as [22]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 58 references couldn’t be matched to a known paper.

Research works printed but not linked: 6

  1. [10]Collins SA, Cato K, Albers D, et al Relationship between nursing documentation and mortality. Am J Crit Care 2013; 22 (4): 306– 13. 23817819 PMC3771321
  2. [28]Klann J, Kang MJ, Schnock K, et al A data warehousing design for multi-domain inpatient nursing data in i2b2. In: Abstract at AMIA 2019 Clinical Informatics Conference; April 30–May 2, 2019; Atlanta, GA.
  3. [29]Kang M-J, Dykes PC, Korach TZ, et al Identifying and grading nurses’ concern concepts about patient deterioration using a standard nursing terminology. In: AMIA 2019 Informatics Summit; March 27, 2019; San Francisco, CA. PMC6957124 31707264
  4. [39]Levine ME, Albers DJ, Burgermaster M, Davidson PG, Smaldone AM, Mamykina L. Behavioral-clinical phenotyping with type 2 diabetes self-monitoring data. arXiv, doi: https://arxiv.org/abs/1802.08761, 23 Feb 2018, preprint: not peer reviewed. · arXiv
  5. [51]Cato K. Nursing medication administration data quality: assessment for reuse in predictive analytics. In: AMIA 2019 Clinical Informatics Conference; May 2, 2019; Atlanta, GA.
  6. [52]Rajkomar A, Oren E, Chen K, et al Scalable and accurate deep learning for electronic health records. arXiv, doi: https://arxiv.org/abs/1801.07860, 26 Jan 2001, preprint: not peer reviewed. · arXiv

Cited by

Sorted by citations

Related

Authors
Rossetti, S. C., Knaplund, C., Albers, D., Dykes, P. C., Kang, M. J., Korach, T. Z., Zhou, L., Schnock, K., Garcia, J., Schwartz, J., Fu, L.-H., Klann, J. G., Lowenthal, G., & Cato, K.
Affiliations
  • Columbia University (US)
  • Department of Biomedical Informatics, Columbia University, New York, New York, USA
  • Brigham and Women's Hospital (US)
  • Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
  • Harvard University (US)
Corresponding author
Rossetti, Sarah Collins
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2021-02-24
Volume
28
Issue
6
Pages
1242-1251
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocab006
PMC
PMC8200261
OpenAlex
W3132307970
Cited by
42
References
58
Citations by year
2026: 6 · 2025: 9 · 2024: 11 · 2023: 5 · 2022: 5 · 2021: 6
FWCI
2.39
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • National Institute of Nursing Research (1R01NR016941-01)
  • CONCERN): Clinical Decision Support Communication for Risky Patient States and the National Institute of Nursing Research Reducing Health Disparities Through Informatics (T32NR007969)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
clinical expertise, clinical information systems, early warning scores, registered nurses, conceptual framework, healthcare process modeling, clinical decision-making, patient surveillance
MeSH terms
Data Science, Computer Simulation, Delivery of Health Care, Humans, Models, Theoretical, Phenotype

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review