The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Bad Health Informatics Can Kill – Is Evaluation the Answer?
Abstract
Summary: Objective: Health care is entering the age of information society. It is evident that the use of modern information and communication technology (ICT) offers tremendous opportunities to improve health care. However, there are also hazards associated with ICT in health care. We want to present an overview of typical hazards associated with ICT in health care, and to discuss how ICT evaluation can be a solution. Methods: We analyze examples of failures and problems associated with ICT in health care. This collection is also made available on a website. Results and Conclusion: Systematic, continuous evaluation of quality and effects of ICT during the whole life cycle of ICT components seems to be one important approach to detect and prevent possible ICT hazards and failures, supporting a higher quality of patient care. However, empirical studies proving this assumption are needed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Relatively little time, thought, or resources have been devoted to assessing the potential risks associated with eHealth interventions.
“it is of concern that, in this rush, relatively little time, thought, or resources have been devoted to assessing the potential risks associated with eHealth interventions [11]–[13],[33]–[36].”
From Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation (2009) · cites this paper as [12]
eHealth applications that are inappropriately specified, have functional errors, are unreliable, user-unfriendly, or in a poorly prepared or supported environment can put patients' and the health service at risk.
“eHealth applications that are inappropriately specified, have functional errors, are unreliable, user-unfriendly, or in a poorly prepared or supported environment can put patients' and the health service at risk.”
From Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation (2009) · cites this paper as [12]
eHealth may compromise patient safety in a number of ways.
“eHealth may compromise patient safety in a number of ways.”
From Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation (2009) · cites this paper as [12]
Increased workloads on clinical users can in turn decrease the quality of patient care.
“increased workloads on clinical users can in turn decrease in the quality of patient care”
From Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation (2009) · cites this paper as [12]
Delays in answering emergency calls due to problems with emergency dispatching systems may lead to delays in emergency treatment.
“delays in answering emergency calls due to problems with emergency dispatching systems may lead to delays in emergency treatment [12].”
From Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation (2009) · cites this paper as [12]
Citations
Cites
Institute of Medicine (2000) · National Academies Press eBooks · doi:10.17226/9728 · Cited by 14,367
Ash, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789
Safeware: System Safety and Computers
Leveson, N. G. (1995) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,688
Evaluating computerised health information systems: hard lessons still to be learnt
Littlejohns, P., Wyatt, J. C., & Garvican, L. (2003) · BMJ · doi:10.1136/bmj.326.7394.860 · Cited by 407
Visions and strategies to improve evaluation of health information systems
Ammenwerth, E., Brender, J., Nykänen, P., Prokosch, H., Rigby, M., & Talmon, J. L. (2004) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2004.04.004 · Cited by 233
Self-citationHealth care in the information society. A prognosis for the year 2013
Haux, R., Ammenwerth, E., Herzog, W., & Knaup, P. (2002) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(02)00030-8 · Cited by 201
Self-citationLessons from a failed information systems initiative: issues for complex organisations☆
Southon, G., SAUER, C., & Dampney, K. (1999) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(99)00018-0 · Cited by 157
Future Directions in Evaluation Research: People, Organizational, and Social Issues
Shaw, N. T., & Kaplan, B. (2004) · Methods of Information in Medicine · doi:10.1055/s-0038-1633862 · Cited by 144
Self-citationVerifying quality and safety in health informatics services
Rigby, M., Forsström, J., Roberts, R., & Wyatt, J. C. (2001) · BMJ · doi:10.1136/bmj.323.7312.552 · Cited by 100
When health information systems fail.
Beynon‐Davies, P., & Lloyd-Williams, M. (1999) · PubMed · Cited by 43
Side effects and responsibility of medical informatics
Gell, G. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00220-9 · Cited by 16
Taking Health Telematics into the 21st Century
Roberts, R. (2000) · Cited by 10
Flagged first, then by citations
Cited by
Sorted by citations
Magrabi, F., Ammenwerth, E., McNair, J. B., De Keizer, N. F., Hyppönen, H., Nykänen, P., Rigby, M., Scott, P. J., Vehko, T., Wong, Z. S.-Y., et al. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677903 · Cited by 347
Self-citationBaniasadi, T., Ayyoubzadeh, S. M., & Mohammadzadeh, N. (2020) · Oman Medical Journal · doi:10.5001/omj.2020.43 · Cited by 338
Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Catwell, L., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000126 · Cited by 307
Design and Evaluation in eHealth: Challenges and Implications for an Interdisciplinary Field
Pagliari, C. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e15 · Cited by 275
Towards Semantic Interoperability for Electronic Health Records
Knaup, P., Hovenga, E. J. S., Heard, S., & Garde, S. (2007) · Methods of Information in Medicine · doi:10.1160/me5001 · Cited by 244
Zahabi, M., Kaber, D. B., & Swangnetr, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720815576827 · Cited by 240
STARE-HI—Statement on reporting of evaluation studies in Health Informatics
Talmon, J. L., Ammenwerth, E., Brender, J., DEKEIZER, N., Nykänen, P., & Rigby, M. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.09.002 · Cited by 237
Self-citationIssues in Biomedical Research Data Management and Analysis: Needs and Barriers
Anderson, N., Lee, E. S., Brockenbrough, J. S., Minie, M., Fuller, S. S., Brinkley, J. F., & Tarczy-Hornoch, P. Z. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2114 · Cited by 152
Guideline for good evaluation practice in health informatics (GEP-HI)
Nykänen, P., Brender, J., Talmon, J., de Keizer, N., Rigby, M., Beuscart-Zephir, M.-C., & Ammenwerth, E. (2011) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.08.004 · Cited by 105
Self-citationImpact of CPOE on Mortality Rates – Contradictory Findings, Important Messages
Talmon, J. L., Ash, J. S., Bates, D. W., Beuscart-Zéphir, M.-C., Duhamel, A., Elkin, P. L., Gardner, R. M., Geissbühler, A., & Ammenwerth, E. (2006) · Methods of Information in Medicine · doi:10.1055/s-0038-1634123 · Cited by 98
Self-citationHarrington, L., Kennerly, D., & Johnson, C. (2011) · Journal of Healthcare Management · doi:10.1097/00115514-201101000-00006 · Cited by 93
The effect of electronic health records on patient safety: A qualitative exploratory study
Tubaishat, A. (2017) · Informatics for Health and Social Care · doi:10.1080/17538157.2017.1398753 · Cited by 89
Wu, C., Xu, H., Bai, D., Chen, X., Gao, J., & Jiang, X. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-066322 · Cited by 88
Teixeira, L., Ferreira, C., & Santos, B. S. (2012) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2010.10.007 · Cited by 76
Using Information Technology to Improve the Quality and Safety of Emergency Care
Handel, D. A., Wears, R. L., Nathanson, L. A., & Pines, J. M. (2011) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2011.01070.x · Cited by 74
A methodology for shifting the focus of e-health support design onto user needs
De Rouck, S., Jacobs, A., & Leys, M. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2007.11.004 · Cited by 70
March, C. A., Steiger, D., Scholl, G., Mohan, V., Hersh, W., & Gold, J. A. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-002549 · Cited by 64
Mobile applications in oncology: A systematic review of health science databases
Ayora, A. F., Loreto, M.-S., Montesinos, M. J. L., Salas‐Medina, P., Julio, M. P. M., & Llor, A. M. S. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.104001 · Cited by 63
Cahill, M. R., Cleary, B., & Cullinan, S. (2025) · BMC Health Services Research · doi:10.1186/s12913-024-12060-2 · Cited by 81
Borycki, E. M., & Kushniruk, A. W. (2005) · Healthcare Quarterly · doi:10.12927/hcq..17673 · Cited by 54
STARE-HI -Statement on Reporting of Evaluation Studies in Health Informatics
Ammenwerth, E., Brender, J., de Keizer, N. F., Nykänen, P., Rigby, M., & Talmon, J. L. (2009) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638633 · Cited by 53
Self-citationFactors influencing the use of IT in the emergency department: A qualitative study
Ayatollahi, H., Bath, P. A., & Goodacre, S. W. (2010) · Health Informatics Journal · doi:10.1177/1460458210377480 · Cited by 48
Social care informatics as an essential part of holistic health care: A call for action
Rigby, M., Hill, P., Koch, S., & Keeling, D. I. (2011) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.06.001 · Cited by 47
Requirements engineering in health care: the example of chemotherapy planning in paediatric oncology
Sebastian Garde, A., & Knaup, P. (2006) · Requirements Engineering · doi:10.1007/s00766-006-0029-6 · Cited by 42
Ndira, S. P., Rosenberger, K. D., & Wetter, T. (2008) · Methods of Information in Medicine · doi:10.3414/me0511 · Cited by 40
Qin, Y., Zhou, R., Wu, Q., Huang, X., Chen, X., Wang, W., Wang, X., Xu, H. X., Zheng, J., Qian, S., et al. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0569-3 · Cited by 36
The computerized patient record: where do we stand ?
Jaspers, M., Knaup, P., & Schmidt, D. (2006) · PubMed · Cited by 35
Gaps in the gaze: Informatic practice and the work of public health surveillance
French, M. A. (2014) · Surveillance & Society · doi:10.24908/ss.v12i2.4750 · Cited by 31
Ammenwerth, E., & de Keizer, N. F. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2276 · Cited by 31
Self-citationVision and challenges of Evidence-Based Health Informatics: A case study of a CPOE meta-analysis
Ammenwerth, E., Schnell‐Inderst, P., & Siebert, U. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.11.003 · Cited by 30
Self-citationEvidence Based Health Informatics: 10 Years of Efforts to Promote the Principle
Ammenwerth, E., Beuscart-Zéphir, M.-C., Brender, J., Hyppönen, H., Melia, S., Nykänen, P., Talmon, J. L., de Keizer, N. F., & Rigby, M. (2013) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638830 · Cited by 29
Self-citationClinical decision support systems: Need for evidence, need for evaluation
Ammenwerth, E., Nykänen, P., Rigby, M., & de Keizer, N. F. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2013.05.001 · Cited by 29
Self-citationBaniasadi, T., Kalhori, S. R. N., Ayyoubzadeh, S. M., Zakerabasali, S., & Pourmohamadkhan, M. (2018) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x18804747 · Cited by 28
The impact of electronic records on patient safety: a qualitative study
Scantlebury, A., Adamson, J., Watt, I. S., Sheard, L., Cairns, P. A., & Wright, J. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0299-y · Cited by 28
Improving Evaluation to Address the Unintended Consequences of Health Information Technology
Ammenwerth, E., Hyppönen, H., de Keizer, N. F., Nykänen, P., Rigby, M., Scott, P., Talmon, J. L., Georgiou, A., & Magrabi, F. (2016) · Yearbook of Medical Informatics · doi:10.15265/iy-2016-013 · Cited by 22
Self-citationHealth Information Technology and Victory
Wears, R. L. (2014) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2014.08.024 · Cited by 23
Biomedical Informatics – A Confluence of Disciplines?
Ammenwerth, E., Dickhaus, H., Knaup, P., Lovis, C., Mantas, J., Maojo, V., Martin-Sanchez, F. J., Musen, M. A., Patel, V. L., Surján, G., et al. (2011) · Methods of Information in Medicine · doi:10.3414/me11-06-0003 · Cited by 19
Self-citationStatement on Reporting of Evaluation Studies in Health Informatics
Talmon, J. L., Ammenwerth, E., Brender, J., de Keizer, N. F., Nykänen, P., Rigby, M., Aarts, J., Knaup, P., Niazkhani, Z., Nøhr, C. G., et al. (2007) · Cited by 19
Self-citationQuality of Health IT Evaluations
Pirkko, N., & Viitanen, J. (2016) · Studies in health technology and informatics · doi:10.3233/978-1-61499-635-4-291 · Cited by 19
Evaluation – The Cinderella Science of ICT in Health
Rigby, M. (2006) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638485 · Cited by 17
Scott, P. (2015) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v22i1.139 · Cited by 14
On the Safety of Electronic Medical Records
Weber-Jahnke, J. H., & Mason-Blakley, F. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-32355-3_11 · Cited by 12
Cancer and the computerized family: towards a clinical ethics of “indirect” Internet use
Simon, C. M., & Schramm, S. (2008) · Medicine Health Care and Philosophy · doi:10.1007/s11019-008-9127-1 · Cited by 12
The safety of Electronic Medical Record (EMR) systems
Weber-Jahnke, J. H., & Mason-Blakley, F. (2011) · ACM SIGHIT Record · doi:10.1145/2047478.2047480 · Cited by 10
Clinical information systems in the intensive care unit: primum non nocere
Lapinsky, S. E. (2009) · Critical Care · doi:10.1186/cc7143 · Cited by 8
Section 2: Patient Records: The Computerized Patient Record: Where Do We Stand?
Knaup, P., Schmidt, D., & Jaspers, M. (2006) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638464 · Cited by 8
Health Informatics: The Relevance of Open Source and Multilevel Modeling
Cavalini, L. T., & Cook, T. W. (2011) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-3-642-24418-6_29 · Cited by 7
Assessment of mHealth Interventions: Need for New Studies, Methods, and Guidelines for Study Designs
Ologeanu‐Taddei, R. (2020) · JMIR Medical Informatics · doi:10.2196/21874 · Cited by 7
Using technology to engage boys and men in the prevention of sexual assault
Oesterle, D. W., Schipani‐McLaughlin, A. M., Salazar, L. F., & Gilmore, A. K. (2022) · Elsevier eBooks · doi:10.1016/b978-0-12-819202-3.00006-7 · Cited by 6
Implementing Software Development Guidelines in a Medical Informatics Research Project
Spreckelsen, C., Weßel, C., & Ißler, L. (2007) · Methods of Information in Medicine · doi:10.1055/s-0038-1625422 · Cited by 5
Video observation in HIT development: lessons learned on benefits and challenges
Høstgaard, A. M. B., & Bertelsen, P. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-91 · Cited by 5
Charpiat, B., Derfoufi, S., Larger, M., Janoly-Duménil, A., Mouchoux, C., Allenet, B., Tod, M. M., Grassin, J., Boulieu, R., Catala, O., et al. (2016) · Annales Pharmaceutiques Françaises · doi:10.1016/j.pharma.2016.01.001 · Cited by 5
Appleton, A., Sadek, K., Dawson, I. G. J., & de Lusignan, S. (2013) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v20i4.13 · Cited by 4
Davoody, N., Eghdam, A., Koch, S., & Hägglund, M. (2023) · JMIR Human Factors · doi:10.2196/43861 · Cited by 4
Shaw, N. T. (2006) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm.2006.124 · Cited by 4
Self-citationEmerging Technologies for Health Communication
Pappas, Y., & Car, J. (2011) · doi:10.1002/9781444346855.ch18 · Cited by 4
Wyatt, J. C. (2019) · Studies in health technology and informatics · doi:10.3233/shti190105 · Cited by 3
Hackbarth, G., & McQuade, J. (2009) · International Journal of Healthcare Delivery Reform Initiatives · doi:10.4018/jhdri.2009040101 · Cited by 2
Health Information Systems Evaluation Criteria: Overview of Systematic Reviews
Kia, A. S., Beheshti, M., & Shahmoradi, L. (2022) · Frontiers in Health Informatics · doi:10.30699/fhi.v11i1.376 · Cited by 3
Crossing Boundaries : Transferring eHealth services across the Northern Periphery
Pesola, U.-M. (2013) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 2
Data Mining Driven Learning Apprentice System For Medical Billing Compliance
Abdullah, U. (2012) · HEC National Digital Library · Cited by 2
A Biomedical Informatics Perspective on Human Factors:
Meyer, R., & Factors, S. E. F. T. I. Y. S. O. H. (2012) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1639432 · Cited by 1
Scott, P., de Keizer, N. F., Georgiou, A., Hyppönen, H., Craven, C. K., & Rigby, M. (2015) · Cited by 1
The remodelling of patient care pathway for e-health
Jones, K. (2009) · Brunel University Research Archive (BURA) (Brunel University London) · Cited by 1
Quantitative Method of Medication System Interface Evaluation
Pingenot, A., Shanteau, J., & PINGENOT, J. D. F. (2007) · CIN Computers Informatics Nursing · doi:10.1097/01.ncn.0000280591.73808.e7 · Cited by 1
Rigby, M., Talmon, J. L., Ammenwerth, E., Brender, J., de Keizer, N. F., & Nykänen, P. (2009) · VBN Forskningsportal (Aalborg Universitet) · Cited by 1
Self-citationClinical decision support systems in the care of hospitalised patients with diabetes
Nirantharakumar, K. (2013) · University of Birmingham Institutional Research Archive (University of Birmingham)
Improving patient safety using computerised decision support systems
Aronsky, D. (2006) · Swiss Medical Informatics · doi:10.4414/smi.22.108
Evaluation of Medication System Interface
Shanteau, J., Pingenot, A., & Pingenot, J. D. F. (2009) · Electronic workshops in computing · doi:10.14236/ewic/ndm2009.15
Why is medical software so hard?
Wetter, T. C. (2008) · Informatik - Forschung und Entwicklung · doi:10.1007/s00450-008-0043-7
The Remodelling of Patient Care Pathway for
Jones, K. L. (2009)
Commentary Clinical information systems in the intensive care unit: primum non nocere
Lapinsky, S. E. (2009)
Safety in health care systems : a synthesis of the literature
Weatherby, J. (2015) · doi:10.6083/m4cf9n4s
Specific Factors for ERP-Success Measurement in Healthcare
Kronbichler, S. A., & Ostermann, H. (2012) · IGI Global eBooks · doi:10.4018/978-1-4666-0170-3.ch011
Special Section: Assessing Information Technologies for Health: Health Technology Assessment
Ammenwerth, E., & Health, M. E. F. T. I. Y. S. O. A. I. T. F. (2006) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638474
Self-citationThilakasiri, P. (2019) · International Journal of Advanced Research · doi:10.21474/ijar01/9156
Herasevich, V., Dziadzko, M. A., & Pickering, B. W. (2019) · Springer eBooks · doi:10.1007/978-3-662-59307-3_8
(2006) · Elsevier eBooks · doi:10.1016/b978-012370464-1.50014-9
Hackbarth, G., & McQuade, J. (2011) · IGI Global eBooks · doi:10.4018/978-1-60960-183-6.ch006
Beyond Trial and Error: Strategic Assessment of Decentralized Identity in US HealthcarePreprint
Magdalena, S. S. M., Meier, A., Elshan, E., Malekan, O., & Leimeister, J. M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5025855
Cohen, C. (2014) · The University of Bath Online Publications Store (The University of Bath)
Metadata
- Authors
- Shaw, N. T., & Ammenwerth, E.
- Affiliations
- UMIT TIROL - Private University for Health Sciences and Health Technology (AT)
- Corresponding author
- Ammenwerth, E.
- Venue
- Methods of Information in Medicine
- Publisher
- Georg Thieme Verlag KG
- Published
- 2005
- Volume
- 44
- Issue
- 01
- Pages
- 1-3
- ISSN
- 0026-1270 (ISSN-L)
- 2511-705X
- OpenAlex
- W157321957
- Cited by
- 82
- References
- 12
- Citations by year
- 2025: 2 · 2023: 2 · 2022: 2 · 2020: 2 · 2019: 5 · 2018: 1 · 2017: 2 · 2016: 4 · 2015: 4 · 2014: 3 · 2013: 6 · 2012: 5
- FWCI
- 16.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Methods of Information in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Keywords
- health informatics, information and communication technology in healthcare, ICT evaluation, life cycle assessment, quality of care, health information systems
- MeSH terms
- Medical Informatics, Technology Assessment, Biomedical, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
