Evaluation of a comprehensive EHR based on the DeLone and McLean model for IS success: Approach, results, and success factors
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: The article describes the methodological approach to, and results of an evaluation of a comprehensive electronic health record (EHR) in the shake down phase, shortly after its implementation at a regional hospital in Denmark. DESIGN: A formative evaluation based on a mixed-methods case study, designed to be interactive and concurrent was conducted at two hospital departments based on the updated DeLone and McLean framework for evaluating information systems success. METHODS: To ascertain user assessments of the EHR, we distributed a questionnaire two months after implementation to four groups of staff (physicians, nurses, medical secretaries, and physiotherapists; n=244), and at the same time we conducted thirteen individual, semi-structured interviews with representatives from these four groups. Subsequently, seven follow-up focus group interviews were conducted with the four above-mentioned groups, in order to go deeper into specific user assessments. Simultaneously, focus group interviews with two IT departments and the implementation team were conducted, to gain insight into system provider assessments of the implementation process and the EHR. Before, during, and after implementation, 88 h of ethnographic observation were carried out, to give the researchers an understanding of the daily routine of staff, and their use of health records. Finally, daily system performance data were obtained, to gather factual information on system response and downtime. RESULTS: Overall, staff had positive experiences with the EHR and its operational reliability, response time, login and support. Performance was acceptable. Medical secretaries found the use of the patient administration module cumbersome, and physicians found the establishment of the overview of professionally relevant data challenging. There were demands for improvements to these and other functionalities, and for the EHR to be integrated with other systems and databases. LIMITATIONS: Evaluations immediately following implementation are inherently difficult, but was required because a key role was to inform decision-making upon enrollment at other hospitals and systematically identify barriers in this respect. The strength of the evaluation is the mixed-methods approach. Further, the evaluation was based on assessments from staff in two departments that comprise around 50% of hospital staff. A weakness may be that staff assessment plays a major role in interviews and survey. These though are supplemented by performance data and observation. Also, the evaluation primarily reports upon the dimension 'user satisfaction', since use of the EHR is mandatory. Finally, generalizability may be low, since the evaluation was not based on a validated survey. All in all, however, the evaluation proposes an evaluation design in constrained circumstances. CONCLUSIONS: Despite inherent limitations, evaluation of a comprehensive EHR shortly after implementation may be necessary, can be conducted, and may inform political decision making. The updated DeLone and McLean framework was constructive in the overall design of the evaluation of the EHR implementation, and allowed the model to be adapted to the health care domain by being methodological flexible. The mixed-methods case study produced valid and reliable results, and was accepted by staff, system providers, and political decision makers. The successful implementation may be attributed to the configurability of the EHR and to factors such as an experienced, competent implementation organization at the hospital, upgraded soft- and hardware, and a high degree of user involvement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1984) · Cited by 81,983
Case Study Research: Design and Methods
Chaiklin, H. (1991) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-199102000-00025 · Cited by 39,535
Hartley, J. (2004) · Essential Guide to Qualitative Methods in Organizational Research · doi:10.4135/9781446280119.n26 · Cited by 15,461
The DeLone and McLean Model of Information Systems Success: A Ten-Year Update
(2003) · Journal of Management Information Systems · doi:10.1080/07421222.2003.11045748 · Cited by 11,791 · cited 7×
“However, earlier studies have shown that successful implementation of EHRs also depends on a wide range of contextual and organizational factors [13,16,51], and hence the positive evaluation results merit further discussion.”
BackgroundMethodInfluentialTask-Technology Fit and Individual Performance
Goodhue, D. L., & Thompson, R. L. (1995) · MIS Quarterly · doi:10.2307/249689 · Cited by 6,006 · cited 1×
“[17–20].”
The Case Research Strategy in Studies of Information Systems
Benbasat, I., Goldstein, D. K., & Mead, M. (1987) · MIS Quarterly · doi:10.2307/248684 · Cited by 4,716 · cited 2×
“Case studes are particularly suited to the study of phenomena over hich the researcher has little or no control, the context f the phenomena under investigation is important, and he interest is in understanding what happens, why, and here [21–23].”
BackgroundA Respecification and Extension of the DeLone and McLean Model of IS Success
Seddon, P. B. (1997) · Information Systems Research · doi:10.1287/isre.8.3.240 · Cited by 2,437 · cited 1×
“[17–20].”
Reliability and Validity in Qualitative Research
(2004) · The SAGE Encyclopedia of Social Science Research Methods · doi:10.4135/9781412950589.n846 · Cited by 2,184
Measuring information systems success: models, dimensions, measures, and interrelationships
Petter, S., DeLone, W., & McLean, E. (2008) · European Journal of Information Systems · doi:10.1057/ejis.2008.15 · Cited by 1,884
Assessing the Validity of IS Success Models: An Empirical Test and Theoretical Analysis
Rai, A., Lang, S. S., & Welker, R. B. (2002) · Information Systems Research · doi:10.1287/isre.13.1.50.96 · Cited by 1,611
Use of Electronic Health Records in U.S. Hospitals
Jha, A. K., DesRoches, C. M., Campbell, E. G., Donelan, K., Rao, S. R., Ferris, T. G., Shields, A., Rosenbaum, S., & Blumenthal, D. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0900592 · Cited by 1,473
Critical factors for successful implementation of enterprise systems
Fui‐Hoon Nah, F., Lee‐Shang Lau, J., & Kuang, J. (2001) · Business Process Management Journal · doi:10.1108/14637150110392782 · Cited by 1,330
HAYRINEN, K., SARANTO, K., & NYKANEN, P. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.09.001 · Cited by 1,297
(2000) · Cited by 1,267
Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study1
Kaplan, B., & Duchon, D. (1988) · MIS Quarterly · doi:10.2307/249133 · Cited by 1,074
Statistical process control as a tool for research and healthcare improvement
Benneyan, J. C. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.6.458 · Cited by 1,045
Yusof, M. M., Kuljis, J., Papazafeiropoulou, A., & Stergioulas, L. K. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.08.011 · Cited by 740
A Partial Test and Development of Delone and Mclean's Model of IS Success
Seddon, P., & Kiew, M.-Y. (1996) · Australasian Journal of Information Systems · doi:10.3127/ajis.v4i1.379 · Cited by 738
LUDWICK, D., & DOUCETTE, J. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.06.005 · Cited by 668 · cited 2×
“However, earlier studies have shown that successful implementation of EHRs also depends on a wide range of contextual and organizational factors [13,16,51], and hence the positive evaluation results merit further discussion.”
BackgroundGREENHALGH, T., POTTS, H. W. W., WONG, G., BARK, P., & SWINGLEHURST, D. (2009) · The Milbank Quarterly · doi:10.1111/j.1468-0009.2009.00578.x · Cited by 541
Marcelli, M., Karsenty, G., Estrade, J. P., Cravello, L., Serment, G., GAMERRE, M., & Agostini, A. (2011) · Cited by 539
Evaluation of health information systems—problems and challenges
Ammenwerth, E., Gräber, S., Herrmann, G., Bürkle, T., & König, J. (2003) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(03)00131-x · Cited by 412 · cited 5×
“A mixed-methods approach is recommended y several authors, owing to the diverse and diffuse effects f health informatics, and in order to combine results, to chieve a varied and profound understanding of the causal inks involved in the implementation and use of IT in health are [14,24–29].”
BackgroundMethodResultInfluentialKaplan, B. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00184-8 · Cited by 380
Determinants of Success of Inpatient Clinical Information Systems: A Literature Review
van der Meijden, M. J. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1094 · Cited by 363 · cited 2×
“[50] M.”
Berner, E. S. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1664 · Cited by 340 · cited 2×
“ubstantial capital and high expectations are presently nvested in the implementation of health information sysems and electronic health records (EHRs) in the USA [1], nd in European countries such as the UK [2], Austria [3], nd Norway [4] (for a comprehensive overview of studies of HRs, see [5]).”
BackgroundThe Updated DeLone and McLean Model of Information Systems Success
Urbach, N., & Müller, B. (2012) · Integrated Series in Information Systems · doi:10.1007/978-1-4419-6108-2_1 · Cited by 307 · cited 2×
“[54] N.”
BackgroundThe Kaiser Permanente Electronic Health Record: Transforming And Streamlining Modalities Of Care
Chen, C., Garrido, T., Chock, D., Okawa, G., & Liang, L. (2009) · Health Affairs · doi:10.1377/hlthaff.28.2.323 · Cited by 299
Information Systems Theory: Explaining and Predicting Our Digital Society, Vol. 1
(2012) · Integrated Series in Information Systems · doi:10.1007/978-1-4419-6108-2 · Cited by 283
A Progress Report On Electronic Health Records In U.S. Hospitals
Jha, A. K., DesRoches, C. M., Kralovec, P. D., & Joshi, M. S. (2010) · Health Affairs · doi:10.1377/hlthaff.2010.0502 · Cited by 225
Somers, T. M., Nelson, K., & Karimi, J. (2003) · Decision Sciences · doi:10.1111/j.1540-5414.2003.02428.x · Cited by 221 · cited 2×
“The questionnaire obtained a response rate of 63%, which is slightly higher than comparable studies reporting user evaluations of EHRs, or their private sector counterparts (Enterprise Resource Planning systems (ERP)) [15,36,37].”
ResultAsh, J. S., Sittig, D. F., Dykstra, R., Campbell, E., & Guappone, K. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.07.015 · Cited by 207 · cited 2×
“[31] J.”
MethodAddressing organizational issues into the evaluation of medical systems
Kaplan, B. (1997) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1997.0040094 · Cited by 175 · cited 4×
“However, follow-up or longituinal studies are recommended when evaluating EHRs [29].”
BackgroundMethodInfluentialPhysician and nurse satisfaction with an Electronic Medical Record system
Likourezos, A., Chalfin, D. B., Murphy, D. G., Sommer, B., Darcy, K., & Davidson, S. J. (2004) · The Journal of Emergency Medicine · doi:10.1016/j.jemermed.2004.03.019 · Cited by 174 · cited 2×
“[36] A.”
ResultWestbrook, J. I., Braithwaite, J., Georgiou, A., Ampt, A., Creswick, N., Coiera, E., & Iedema, R. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2462 · Cited by 174
Triangulation in Evaluation: Design and Analysis Issues
Greene, J., & McClintock, C. (1985) · Evaluation Review · doi:10.1177/0193841x8500900501 · Cited by 171 · cited 2×
“This evaluation applies a case-based and mixed-methods pproach, designed to be interactive and concurrent [34] (see able 1).”
MethodUrbach, N., Smolnik, S., & Riempp, G. (2009) · Business & Information Systems Engineering · doi:10.1007/s12599-009-0059-y · Cited by 170 · cited 1×
“[45] N.”
An Electronic Health Record Based on Structured Narrative
Johnson, S. B., Bakken, S., Dine, D., Hyun, S., Mendonca, E., Morrison, F., Bright, T., Van Vleck, T., Wrenn, J., & Stetson, P. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2131 · Cited by 150
Evaluating the success of an emergency response medical information system
Petter, S., & Fruhling, A. (2011) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.03.010 · Cited by 130 · cited 1×
“[47,48]), as has a sysematic review of health care studies [49].”
BackgroundERP evaluation during the shakedown phase: lessons from an after‐sales division
Häkkinen, L., & Hilmola, O. (2007) · Information Systems Journal · doi:10.1111/j.1365-2575.2007.00261.x · Cited by 116 · cited 4×
“The questionnaire obtained a response rate of 63%, which is slightly higher than comparable studies reporting user evaluations of EHRs, or their private sector counterparts (Enterprise Resource Planning systems (ERP)) [15,36,37].”
BackgroundResultInfluentialBossen, C., & Markussen, R. (2010) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-010-9131-x · Cited by 110
Self-citationGuideline for good evaluation practice in health informatics (GEP-HI)
Nykänen, P., Brender, J., Talmon, J., de Keizer, N., Rigby, M., Beuscart-Zephir, M.-C., & Ammenwerth, E. (2011) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.08.004 · Cited by 105
Vikkelsø, S. (2005) · Scandinavian Journal of Information Systems · Cited by 104
Take Note(s): Differential EHR Satisfaction with Two Implementations under One Roof
O'Connell, R. T., Cho, C., Shah, N., Brown, K., & Shiffman, R. N. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1409 · Cited by 94
Evaluating eHealth: How to Make Evaluation More Methodologically Robust
Lilford, R. J., Foster, J., & Pringle, M. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000186 · Cited by 79 · cited 2×
“[30] R.”
BackgroundA Pragmatist Argument for Mixed Methodology in Medical Informatics
Scott, P. J., & Briggs, J. S. (2009) · Journal of Mixed Methods Research · doi:10.1177/1558689809334209 · Cited by 77 · cited 2×
“A mixed-methods approach is recommended y several authors, owing to the diverse and diffuse effects f health informatics, and in order to combine results, to chieve a varied and profound understanding of the causal inks involved in the implementation and use of IT in health are [14,24–29].”
MethodOtieno, G., Toyama, H., Asonuma, M., Koide, D., & Keiko, N. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.01.002 · Cited by 77 · cited 1×
“[47,48]), as has a sysematic review of health care studies [49].”
BackgroundReliability and Validity in Qualitative Research
Bellenger, D. N., Kirk, J., Miller, M. L., & Agar, M. (1986) · Journal of Marketing Research · doi:10.2307/3151819 · Cited by 74
The Role of Integration in Health-Based Information Infrastructures
Ellingsen, G., & Røed, K. (2010) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-010-9122-y · Cited by 49 · cited 2×
“ubstantial capital and high expectations are presently nvested in the implementation of health information sysems and electronic health records (EHRs) in the USA [1], nd in European countries such as the UK [2], Austria [3], nd Norway [4] (for a comprehensive overview of studies of HRs, see [5]).”
BackgroundAccounting and Co-Constructing: The Development of a Standard for Electronic Health Records
Bossen, C. (2011) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-011-9141-3 · Cited by 38 · cited 2×
“[53] C.”
MethodSelf-citationCampion, T. R., Waitman, L. R., May, A. K., Özdaş, A., Lorenzi, N. M., & Gadd, C. S. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2009.09.004 · Cited by 32
Bonner, L. M., Simons, C. E., Parker, L. E., Yano, E. M., & Kirchner, J. E. (2010) · Implementation Science · doi:10.1186/1748-5908-5-63 · Cited by 26
Cited by
Understanding the impact of m-banking on individual performance: DeLone & McLean and TTF perspective
Tam, C., & Oliveira, T. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.03.016 · Cited by 383
Literature review of mobile banking and individual performance
Tam, C., & Oliveira, T. (2017) · International Journal of Bank Marketing · doi:10.1108/ijbm-09-2015-0143 · Cited by 202
“(Bossen et al., 2013) Evaluation of a comprehensive electronic health record D&M 244 professionals, ANOVA The results produced using the D&M framework are valid and reliable, and were accepted by staff, system providers, and political decision makers.”
BackgroundMethodInfluentialSharma, S. K., Gaur, A., Saddikuti, V., & Rastogi, A. (2017) · Behaviour & Information Technology · doi:10.1080/0144929x.2017.1340973 · Cited by 172
“Bossen, Jensen, and Udsen (2013) examined the user acceptance of the electronic health records in the healthcare domain.”
BackgroundUnlocking AI Chatbot Potential in Healthcare: Trust-Enhanced DeLone & McLean IS Success Model
Sarhan, M. Y., Alarify, M., & Khojah, M. (2026) · Healthcare · doi:10.3390/healthcare14101324 · Cited by 2
“The D&M ISS Model provides a practical framework for evaluating the effectiveness of health IS [70].”
Metadata
- Authors
- Bossen, C., Jensen, L. G., & Udsen, F. W.
- Venue
- International Journal of Medical Informatics
- Publisher
- Elsevier BV
- Published
- 2013-10
- Volume
- 82
- Issue
- 10
- Pages
- 940-953
- ISSN
- 1386-5056 (ISSN-L)
- OpenAlex
- W1977488000
- Cited by
- 183
- References
- 51
- FWCI
- 13.59
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.