Developing Software to “Track and Catch” Missed Follow-up of Abnormal Test Results in a Complex Sociotechnical Environment
Open accessAbstract
Summary: Background: Abnormal test results do not always receive timely follow-up, even when providers are notified through electronic health record (EHR)-based alerts. High workload, alert fatigue, and other demands on attention disrupt a provider’s prospective memory for tasks required to initiate follow-up. Thus, EHR-based tracking and reminding functionalities are needed to improve follow-up. Objectives: The purpose of this study was to develop a decision-support software prototype enabling individual and system-wide tracking of abnormal test result alerts lacking follow-up, and to conduct formative evaluations, including usability testing. Methods: We developed a working prototype software system, the Alert Watch And Response Engine (AWARE), to detect abnormal test result alerts lacking documented follow-up, and to present context-specific reminders to providers. Development and testing took place within the VA’s EHR and focused on four cancer-related abnormal test results. Design concepts emphasized mitigating the effects of high workload and alert fatigue while being minimally intrusive. We conducted a multifaceted formative evaluation of the software, addressing fit within the larger socio-technical system. Evaluations included usability testing with the prototype and interview questions about organizational and workflow factors. Participants included 23 physicians, 9 clinical information technology specialists, and 8 quality/safety managers. Results: Evaluation results indicated that our software prototype fit within the technical environment and clinical workflow, and physicians were able to use it successfully. Quality/safety managers reported that the tool would be useful in future quality assurance activities to detect patients who lack documented follow-up. Additionally, we successfully installed the software on the local facility’s “test” EHR system, thus demonstrating technical compatibility. Conclusion: To address the factors involved in missed test results, we developed a software prototype to account for technical, usability, organizational, and workflow needs. Our evaluation has shown the feasibility of the prototype as a means of facilitating better follow-up for cancer-related abnormal test results.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A sample size of 30 participants, each reviewing 35 test results, was required to achieve 80% power to detect a statistically significant difference in cognitive workload and performance.
“On the basis of the anticipated rate of appropriately identified abnormal test results in the literature12,13,14,15,16,17,18,19,20,21 and the anticipated magnitude of the association of the enhanced EHR, we required a sample size of 30 participants, each reviewing 35 test results, to achieve 80% power to detect a statistically significant difference in cognitive workload and performance.”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [17]
Even in the EHR environment, with alerts and reminders in place, physicians continue to often inappropriately manage abnormal test results.
“Even in the EHR environment, with alerts and reminders in place, physicians continue to often inappropriately manage abnormal test results.12,13,14,15,16,17,18,19,20,21”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [17]
Cites
Ericsson (1993) · doi:10.1037/0033-295x.87.3.215 · Cited by 4,539 · see citing works
How a Cockpit Remembers Its Speeds
Hutchins, E. (1995) · Cognitive Science · doi:10.1207/s15516709cog1903_1 · Cited by 1,880 · cited 1× · see citing works
“Because alerts no longer appear after being viewed, they cannot serve as external memory aids [24–26].”
BackgroundWork system design for patient safety: the SEIPS model
Carayon, P., Schoofs Hundt, A., Karsh, B.-T., Gurses, A. P., Alvarado, C. J., Smith, M., & Flatley Brennan, P. (2006) · Quality in Health Care · doi:10.1136/qshc.2005.015842 · Cited by 1,795 · see citing works
Self-citationDiagnostic error in internal medicine
Graber, M. L., Franklin, N., & Gordon, R. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.13.1493 · Cited by 1,549 · see citing works
Groupware and social dynamics: eight challenges for developers
Grudin, J. (1994) · Communications of the ACM · doi:10.1145/175222.175230 · Cited by 1,228 · see citing works
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117 · see citing works
Finding usability problems through heuristic evaluation
Nielsen, J. (1992) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/142750.142834 · Cited by 1,066 · see citing works
Sittig, D. F., & Singh, H. (2010) · Quality and Safety in Health Care · doi:10.1136/qshc.2010.042085 · Cited by 897 · see citing works
Self-citationMissed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims
Gandhi, T. K., Kachalia, A., Thomas, E. J., Puopolo, A. L., Yoon, C., Brennan, T. A., & Studdert, D. M. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-145-7-200610030-00006 · Cited by 662 · see citing works
Jaspers, M. W. M. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.10.002 · Cited by 649 · see citing works
If not now, when?: the effects of interruption at different moments within task execution
Adamczyk, P. D., & Bailey, B. P. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985727 · Cited by 531 · see citing works
The Extent and Importance of Unintended Consequences Related to Computerized Provider Order Entry
Ash, J. S., Sittig, D. F., Poon, E. G., Guappone, K. P., Campbell, E., & Dykstra, R. H. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2373 · Cited by 500 · see citing works
Self-citationLaxmisan, A., Hakimzada, F., Sayan, O. R., Green, R. A., Zhang, J., & Patel, V. L. (2007) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2006.09.019 · Cited by 449 · see citing works
Self-citationComputer Technology and Clinical Work: Still Waiting for Godot
Wears, R. L., & Berg, M. (2005) · JAMA · doi:10.1001/jama.293.10.1261 · Cited by 448 · see citing works
Leveson, N. G., Dulac, N., Marais, K. B., & Carroll, J. S. (2009) · Organization Studies · doi:10.1177/0170840608101478 · Cited by 411 · see citing works
Health information technology: fallacies and sober realities
Karsh, B.-T., Weinger, M. B., Abbott, P. A., & Wears, R. L. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.005637 · Cited by 365 · cited 1× · see citing works
“In this way AWARE avoids the problem of burdening providers with data entry tasks for administrative purposes [58].”
BackgroundThe impact of interruptions on clinical task completion
Westbrook, J. I., Coiera, E. W., Dunsmuir, W. T. M., Brown, B. M., Kelk, N. J., Paoloni, R., & Tran, C. (2010) · BMJ Quality & Safety · doi:10.1136/qshc.2009.039255 · Cited by 344 · see citing works
Poon, E. G., Gandhi, T. K., Sequist, T. D., Murff, H. J., Karson, A. S., & Bates, D. W. (2004) · Archives of Internal Medicine · doi:10.1001/archinte.164.20.2223 · Cited by 301 · see citing works
Exploring Barriers and Facilitators to the Use of Computerized Clinical Reminders
Saleem, J. J., Patterson, E. S., Militello, L. G., Render, M. L., Orshansky, G., & Asch, S. M. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1777 · Cited by 250 · see citing works
Systems with Human Monitors: A Signal Detection Analysis
Sorkin, R. D., & Woods, D. D. (1985) · Human-Computer Interaction · doi:10.1207/s15327051hci0101_2 · Cited by 213 · see citing works
Singh, H., Thomas, E. J., Mani, S., Sittig, D., Arora, H., Espadas, D., Khan, M. M., & Petersen, L. A. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.263 · Cited by 212 · cited 2× · see citing works
“In previous work involving 2,500 asynchronous alerts related to abnormal laboratory and imaging results, we found that 7% of laboratory and 8% of imaging result alerts lacked timely follow-up at 4 weeks [3, 4], even in cases with explicit acknowledgement of the notification by the recipient.”
BackgroundSelf-citationInformation Overload and Missed Test Results in Electronic Health Record–Based Settings
Singh, H., Spitzmueller, C., Petersen, N. J., Sawhney, M. K., & Sittig, D. F. (2013) · JAMA Internal Medicine · doi:10.1001/2013.jamainternmed.61 · Cited by 209 · see citing works
Self-citationCombating omission errors through task analysis and good reminders
REASON, J. T. (2002) · BMJ Quality & Safety · doi:10.1136/qhc.11.1.40 · Cited by 201 · see citing works
Brittleness in the design of cooperative problem-solving systems: the effects on user performance
Smith, P. J., McCoy, C. E., & Layton, C. F. (1997) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/3468.568744 · Cited by 165 · see citing works
Singh, H., Naik, A. D., Rao, R., & Petersen, L. A. (2008) · Journal of General Internal Medicine · doi:10.1007/s11606-007-0393-z · Cited by 162 · cited 2× · see citing works
“In order to address this in a multifaceted sociotechnical context, we intended the functional prototype design to meet the following goals, many of which were conceived based on previous research findings: [1, 3, 4, 27–29, 33–37] 1.”
BackgroundMethodSelf-citationSingh, H., Thomas, E. J., Sittig, D. F., Wilson, L., Espadas, D., Khan, M. M., & Petersen, L. A. (2010) · The American Journal of Medicine · doi:10.1016/j.amjmed.2009.07.027 · Cited by 160 · cited 3× · see citing works
“In order to address this in a multifaceted sociotechnical context, we intended the functional prototype design to meet the following goals, many of which were conceived based on previous research findings: [1, 3, 4, 27–29, 33–37] 1.”
BackgroundMethodSelf-citationMurphy, D. R., Reis, B., Sittig, D. F., & Singh, H. (2012) · The American Journal of Medicine · doi:10.1016/j.amjmed.2011.07.029 · Cited by 142 · see citing works
Self-citationCommunication outcomes of critical imaging results in a computerized notification system
Singh, H., Arora, H. S., Vij, M. S., Rao, R., Khan, M. M., & Petersen, L. A. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2280 · Cited by 140 · see citing works
Self-citationSingh, H., Hirani, K., Kadiyala, H., Rudomiotov, O., Giardina, T. D., Khan, M. M., & Wahls, T. L. (2010) · Journal of Clinical Oncology · doi:10.1200/jco.2009.25.6636 · Cited by 135 · see citing works
Self-citationSociotechnical systems analysis in health care: a research agenda
Carayon, P., Bass, E. J., Bellandi, T., Gürses, A. P., Hallbeck, M. S., & Mollo, V. (2011) · IIE Transactions on Healthcare Systems Engineering · doi:10.1080/19488300.2011.619158 · Cited by 132 · see citing works
Errors in cancer diagnosis: current understanding and future directions
Singh, H., Sethi, S., Raber, M., & Petersen, L. A. (2007) · Journal of Clinical Oncology · doi:10.1200/jco.2007.13.2142 · Cited by 121 · see citing works
Self-citationGROUPWARE AND SOCIAL DYNAMICS: EIGHT CHALLENGES FOR DEVELOPERS
Grudin, J. T. (1995) · Elsevier eBooks · doi:10.1016/b978-0-08-051574-8.50079-0 · Cited by 119 · see citing works
The Messy Details: Insights From the Study of Technical Work in Healthcare
Nemeth, C., Cook, R. I., & Woods, D. D. (2004) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2004.836802 · Cited by 111 · see citing works
Design and implementation of a comprehensive outpatient Results Manager
Poon, E. G., Wang, S. J., Gandhi, T. K., Bates, D. W., & Kuperman, G. J. (2003) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00061-3 · Cited by 111 · see citing works
Exploring situational awareness in diagnostic errors in primary care
Singh, H., Giardina, T. D., Petersen, L. A., Smith, M. W., Paul, L. W., Dismukes, K., Bhagwath, G., & Thomas, E. J. (2011) · BMJ Quality & Safety · doi:10.1136/bmjqs-2011-000310 · Cited by 103 · see citing works
Self-citationInteractive Critiquing as a Form of Decision Support: An Empirical Evaluation
Guerlain, S. A. E., Smith, P. J., Obradovich, J. H., Rudmann, S. V., Strohm, P. L., Smith, J. W., Svirbely, J. R., & Sachs, L. A. (1999) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872099779577363 · Cited by 94 · see citing works
Singh, H., Spitzmueller, C., Petersen, N. J., Sawhney, M. K., Smith, M. W., Murphy, D. R., Espadas, D., Laxmisan, A., & Sittig, D. F. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001267 · Cited by 92 · see citing works
Self-citationEight Recommendations for Policies for Communicating Abnormal Test Results
Singh, H., & Vij, M. S. (2010) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(10)36037-5 · Cited by 83 · see citing works
Self-citationHow Not to Have to Navigate Through Too Many Displays
Woods, D. D., & Watts, J. C. (1997) · Elsevier eBooks · doi:10.1016/b978-044481862-1.50092-3 · Cited by 82 · see citing works
Raab, S. S., & Grzybicki, D. M. (2010) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.20068 · Cited by 77 · see citing works
Cognitive ergonomics, socio-technical systems, and the impact of healthcare information technologies
Lawler, E., Hedge, A., & Pavlovic-Veselinovic, S. (2011) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2011.02.006 · Cited by 74 · see citing works
Saleem, J. J., Russ, A. L., Sanderson, P., Johnson, T. R., Zhang, J., & Sittig, D. F. (2009) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638638 · Cited by 72 · see citing works
Self-citationDumas, J. S., & Salzman, M. (2006) · Reviews of Human Factors and Ergonomics · doi:10.1177/1557234x0600200105 · Cited by 71 · see citing works
Understanding the management of electronic test result notifications in the outpatient setting
Hysong, S. J., Sawhney, M. K., Wilson, L., Sittig, D. F., Esquivel, A., Singh, S., & Singh, H. (2011) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-11-22 · Cited by 68 · see citing works
Self-citationProvider management strategies of abnormal test result alerts: a cognitive task analysis
Hysong, S. J., Sawhney, M. K., Wilson, L., Sittig, D. F., Espadas, D., Davis, T., & Singh, H. (2010) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3200 · Cited by 65 · see citing works
Self-citationSingh, H., Wilson, L., Petersen, L. A., Sawhney, M. K., Reis, B., Espadas, D., & Sittig, D. F. (2009) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-9-49 · Cited by 64 · see citing works
Self-citationUsing a Multifaceted Approach to Improve the Follow-Up of Positive Fecal Occult Blood Test Results
Singh, H., Kadiyala, H., Bhagwath, G., Shethia, A., El‐Serag, H. B., Walder, A. M., Velez, M. E., & Petersen, L. A. (2009) · The American Journal of Gastroenterology · doi:10.1038/ajg.2009.55 · Cited by 63 · see citing works
Self-citationReducing Diagnostic Error Through Medical Home-Based Primary Care Reform
Singh, H., & Graber, M. L. (2010) · JAMA · doi:10.1001/jama.2010.1035 · Cited by 52 · see citing works
Self-citationMurphy, D. R. (2012) · Archives of Internal Medicine · doi:10.1001/archinternmed.2011.740 · Cited by 52 · see citing works
Self-citationRemembrance of Things Future: Prospective Memory in Laboratory, Workplace, and Everyday Settings
Dismukes, R. K. (2010) · Reviews of Human Factors and Ergonomics · doi:10.1518/155723410x12849346788705 · Cited by 47 · see citing works
Ten Strategies to Improve Management of Abnormal Test Result Alerts in the Electronic Health Record
Singh, H., Wilson, L., Reis, B., Sawhney, M. K., Espadas, D., & Sittig, D. F. (2010) · Journal of Patient Safety · doi:10.1097/pts.0b013e3181ddf652 · Cited by 43 · see citing works
Self-citationPrevalence of delayed clinician response to elevated prostate-specific antigen values
Nepple, K. G., Joudi, F. N., Hillis, S. L., & Wahls, T. L. (2008) · Mayo Clinic Proceedings · doi:10.4065/83.4.439 · Cited by 22 · see citing works
What Happens After an Elevated PSA Test: The Experience of 13,591 Veterans
Zeliadt, S. B., Hoffman, R. M., Etzioni, R., Ginger, V. A. T., & Lin, D. W. (2010) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1468-9 · Cited by 21 · see citing works
Wahls, T. (2007) · Journal of Ambulatory Care Management · doi:10.1097/01.jac.0000290402.89284.a9 · Cited by 21 · see citing works
INVITED COMMENTARY—Electronic Medical Records and Preserving Primary Care Physicians' Time
McDonald, C. J. (2012) · Archives of Internal Medicine · doi:10.1001/archinternmed.2011.1678 · Cited by 18 · see citing works
Handoffs and Transitions of Care
Powell, S. K. (2006) · Lippincott s Case Management · doi:10.1097/00129234-200609000-00001 · Cited by 17 · see citing works
Schiff, G. D. (2011) · JAMA · doi:10.1001/jama.2011.496 · Cited by 12 · see citing works
Distributed Prospective Memory: An Approach to Understanding how Nurses Remember Tasks
Grundgeiger, T., Sanderson, P. M., MacDougall, H. G., & Venkatesh, B. (2009) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120905301127 · Cited by 10 · see citing works
Raab, S. S. (2011) · Yearbook of Pathology and Laboratory Medicine · doi:10.1016/s1077-9108(10)79520-7 · Cited by 6 · see citing works
Handoff Communication: Implications For Design
Pennathur, P. R., Bass, E. J., Rayo, M. F., Perry, S. J., Rosen, M. A., & Gürses, A. P. (2012) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181312561181 · Cited by 5 · see citing works
Applying Cognitive Psychology to System Development
Marras, W. S. (2006) · CRC Press eBooks · doi:10.1201/9781420003635-28 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Trans-ethnic association study of blood pressure determinants in over 750,000 individuals
Understanding Society Scientific Group, Giri, A., Program, M. V., Hellwege, J. N., Keaton, J. M., Park, J., Qiu, C., Warren, H. R., Torstenson, E. S., Kovesdy, C. P., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0303-9 · Cited by 444 · see citing works
Mazur, L. M., Mosaly, P. R., Moore, C., & Marks, L. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1709 · Cited by 127 · see citing works
Communication breakdowns and diagnostic errors: a radiology perspective
Murphy, D. R., Singh, H., & Berlin, L. (2014) · Diagnosis · doi:10.1515/dx-2014-0035 · Cited by 52 · see citing works
Self-citationMurphy, D. R., Thomas, E. J., Meyer, A. N. D., & Singh, H. (2015) · Radiology · doi:10.1148/radiol.2015142530 · Cited by 43 · see citing works
Self-citationDiagnostic errors: moving beyond ‘no respect’ and getting ready for prime time
Singh, H. (2013) · BMJ Quality & Safety · doi:10.1136/bmjqs-2013-002387 · Cited by 32 · see citing works
Self-citationMurphy, D. R., Meyer, A. N. D., Vaghani, V., Russo, E. M., Sittig, D. F., Wei, L., Wu, L., & Singh, H. (2017) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2017.08.007 · Cited by 31 · see citing works
Self-citationLack of Timely Follow-Up of Abnormal Imaging Results and Radiologists’ Recommendations
Al-Mutairi, A., Meyer, A. N. D., Chang, P. J., & Singh, H. (2015) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2014.09.031 · Cited by 30 · see citing works
Self-citationMaking electronic health records support quality management: A narrative review
Triantafillou, P. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.03.003 · Cited by 27 · see citing works
Bowen, J. L., OʼBrien, B. C., Ilgen, J. S., Irby, D. M., & ten Cate, O. (2018) · Medical Education · doi:10.1111/medu.13509 · Cited by 25 · see citing works
Cutrona, S. L., Fouayzi, H., Burns, L., Sadasivam, R. S., Mazor, K. M., Gurwitz, J. H., Garber, L., Sundaresan, D., Houston, T. K., & Field, T. S. (2017) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4146-3 · Cited by 24 · see citing works
Application of Electronic Algorithms to Improve Diagnostic Evaluation for Bladder Cancer
Murphy, D. R., Meyer, A. N. D., Vaghani, V., Russo, E. M., Sittig, D. F., Richards, K. A., Wei, L., Wu, L., & Singh, H. (2017) · Applied Clinical Informatics · doi:10.4338/aci-2016-10-ra-0176 · Cited by 19 · see citing works
Self-citationJoseph, A. L., Borycki, E. M., Kushniruk, A. W., & |, |. (2021) · Knowledge Management & E-Learning An International Journal · doi:10.34105/j.kmel.2021.13.027 · Cited by 18 · see citing works
Banjar, H. R., Alkhatabi, H. A., Alganmi, N. A., & Almouhana, G. I. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17218066 · Cited by 15 · see citing works
Test results management and distributed cognition in electronic health record–enabled primary care
Smith, M. W., Hughes, A. M., Brown, C. R., Russo, E. M., Giardina, T. D., Mehta, P., & Singh, H. (2018) · Health Informatics Journal · doi:10.1177/1460458218779114 · Cited by 14 · see citing works
Self-citationMeaningful time-related aspects of alerts in Clinical Decision Support Systems. A unified framework
Cánovas-Segura, B., Morales, A., Juárez, J. M., & Campos, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104397 · Cited by 11 · see citing works
A Sociotechnical Approach to Electronic Health Record Related Safety
Sittig, D. F., & Singh, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-809523-2.00014-5 · Cited by 11 · see citing works
Self-citationBarnett, M. L., Mehrotra, A., Frolkis, J. P., Spinks, M., Steiger, C. E., Hehir, B., Greenberg, J. O., & Singh, H. (2015) · Journal of General Internal Medicine · doi:10.1007/s11606-015-3516-y · Cited by 11 · see citing works
Self-citationSoleimani, J., Pinevich, Y., Barwise, A. K., Huang, C., Dong, Y., Herasevich, V., Gajic, O., & Pickering, B. W. (2020) · Applied Clinical Informatics · doi:10.1055/s-0040-1713750 · Cited by 9 · see citing works
Next steps for electronic health records to improve the diagnostic process
Liebovitz, D. (2015) · Diagnosis · doi:10.1515/dx-2014-0070 · Cited by 9 · see citing works
Dibble, E. H., Swenson, D. W., Cobb, C., Paul, T. J., Karn, A. E., Portelli, D. C., & Movson, J. S. (2016) · Emergency Radiology · doi:10.1007/s10140-016-1452-8 · Cited by 9 · see citing works
Prioritization of Free-Text Clinical Documents: A Novel Use of a Bayesian Classifier
Singh, M., Murthy, A., & Singh, S. R. (2015) · JMIR Medical Informatics · doi:10.2196/medinform.3793 · Cited by 8 · see citing works
Singh, H., & Sittig, D. F. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004140 · Cited by 8 · see citing works
Self-citationFrom Bench to Bed: Bridging from Informatics Theory to Practice
Haux, R., & Lehmann, C. U. (2014) · Methods of Information in Medicine · doi:10.3414/me14-01-0098 · Cited by 6 · see citing works
Factors Affecting Adherence to Outpatient Radiology Report Recommendations
Kavandi, H., Wei, J. L., Bulushi, Y. A., Brook, A., Brook, O. R., & Siewert, B. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.03.009 · Cited by 6 · see citing works
Clinical Decision Support: History and Basic Concepts
McCallie, D. P. (2015) · Health informatics · doi:10.1007/978-3-319-20765-0_1 · Cited by 6 · see citing works
Lo, C., Yang, Y., & Tseng, H. (2021) · Journal of Healthcare Engineering · doi:10.1155/2021/3831453 · Cited by 5 · see citing works
Lamy, J.-B., Jacques Bouaud, A., & Support, S. E. F. T. I. Y. S. O. D. (2014) · Yearbook of Medical Informatics · doi:10.15265/iy-2014-0036 · Cited by 3 · see citing works
Thinking Together: Modeling Clinical Decision-Support as a Sociotechnical System.
Hussain, M., Reynolds, T. L., Mousavi, F. E., Chen, Y., & Zheng, K. (2017) · PubMed Central · Cited by 1 · see citing works
Dadlez, N. M., Clair, A. M. L., Wasima, S., Mayer, N., Harvey, W. F., Roberts, K. E., Mazzullo, J. M., Lominac, E., Koethe, B. C., & Weingart, S. N. (2023) · BMJ Open Quality · doi:10.1136/bmjoq-2023-002334 · Cited by 1 · see citing works
Assessing Clinical Software User Needs for Improved Clinical Decision Support Tools
Denney, K. (2015) · ScholarWorks (Walden University)
Olson, A. P. J., Sloane, J., Zimolzak, A. J., Kaul, B., Vaghani, V., Matin, R., Cholankeril, R. T., & Singh, H. (2025) · Diagnosis · doi:10.1515/dx-2025-0034
Self-citation
- Authors
- Smith, M., Murphy, D., Laxmisan, A., Sittig, D., Reis, B., Esquivel, A., & Singh, H.
- Affiliations
- Michael E. DeBakey VA Medical Center (US)
- Memorial Hermann (US)
- Baylor St. Luke's Medical Center (US)
- Venue
- Applied Clinical Informatics
- Publisher
- Georg Thieme Verlag KG
- Published
- 2013
- Volume
- 04
- Issue
- 03
- Pages
- 359-375
- ISSN
- 1869-0327 (ISSN-L)
- PMC
- PMC3799004
- OpenAlex
- W2068338307
- Cited by
- 31
- References
- 61
- Citations by year
- 2025: 1 · 2023: 3 · 2021: 2 · 2020: 2 · 2019: 1 · 2018: 3 · 2017: 6 · 2016: 1 · 2015: 8 · 2014: 3 · 2013: 1
- FWCI
- 7.48
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Applied Clinical Informatics (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. Department of Veterans Affairs
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- missed follow-up, laboratory abnormalities, electronic health record alerts, alarm fatigue, clinical decision support, prospective memory, provider reminders, cancer care, Veterans Affairs health system, clinical workflow, usability, quality assurance, socio-technical systems
- MeSH terms
- Follow-Up Studies, Humans, Neoplasms, Software, Decision Support Systems, Clinical, Workflow, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
