Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; none of the 28 sources its claims cite is checked for support yet.
Abstract
This is the first in a monthly series of three articles on evaluating eHealth.The Essay section contains opinion pieces on topics of broad interest to a general medical audience.
Overview
CoChat Summary
TL;DR
This paper argues that eHealth should be evaluated as a health care intervention throughout its lifecycle, not only after deployment. The authors emphasize that benefits are plausible but not yet empirically demonstrated, and that poorly designed systems can waste resources and harm patients.
Key findings
- eHealth is expected to play a substantial role in 21st-century health care, but the paper notes that adoption of ICT in health care is still about 25 to 30 years behind many other sectors.
- The paper argues that eHealth interventions should be evaluated with the same rigor as drugs or treatment programs, because poorly designed systems can fail to deliver benefits and can compromise patient safety.
- Examples of potential harms include erroneous prescribing support from poor decision support software, major access/cybersecurity risks in health portals, and an unexpected increase in mortality after implementation of a commercially sold computerised physician order entry system.
- The authors propose continuous systematic multifaceted evaluations throughout the lifecycle of eHealth interventions, with iterative formative evaluations at four key stages and a final summative evaluation after deployment.
Methods
- This is a conceptual/perspective article, not an empirical study.
- The paper synthesizes prior literature and examples of eHealth implementations, including telemedicine, electronic medical records, health portals, and prescribing software.
- Its main analysis is an argument for a lifecycle-based evaluation model, organized into phases from problem definition through design, testing, deployment, formative evaluation, and summative evaluation.
Limitations
- The paper does not report original data, a sample, or effect estimates; its claims are based on discussion and cited examples rather than direct empirical testing.
- The text itself makes evident that the proposed evaluation approach is not validated in the paper with a study, so its effectiveness is argued rather than demonstrated.
- The paper notes that eHealth definitions and applications are diverse and there is no universal agreement on the term, which limits one-size-fits-all evaluation approaches.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 18 Not checked · 7 Own results and interpretations
None of the sources cited for this claim have been checked yet.
In the paper
“it is of concern that, in this rush, relatively little time, thought, or resources have been devoted to assessing the potential risks associated with eHealth interventions [11]–[13],[33]–[36].”
§ Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation > Recognising the Risk of Harm
What the sources say
- Not checked yetcites [11] Mudur G 2004 The great technological divide. BMJ 328 788 Available: http://www.bmj.com/cgi/content/full/328/7443/788. Accessed 22 July 2009 15070625 PMC383368 source
- Not checked yetcites [12] Ammenwerth E Shaw NT 2005 Bad Health Informatics Can Kill – Is Evaluation the Answer? Editorial. Methods Inf Med 44 1 3 Available: http://iig.umit.at/dokumente/z29.pdf. Accessed 22 July 2009 15778787 source
- Not checked yetcites [13] Han YY Carcillo JA Ventkataraman ST Clark RSB Watson SR 2005 Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 116 1506 1512 Available: http://pediatrics.aappublications.org/cgi/content/full/116/6/1506?maxtoshow=&HITS=10&hits=10&RESULTFORMAT=&fulltext=cpoe&searchid=1&FIRSTINDEX=0&resourcetype=HWCIT Accessed 22 July 2009 16322178 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Large investments in eHealth may, by diverting resources result in a shortfall in funding for basic infrastructure, equipment, and staffing elsewhere in the system.”
§ Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation > Recognising the Risk of Harm > Box 2. Examples of problems associated with eHealth projects
What the sources say
- Not checked yetcites [11] Mudur G 2004 The great technological divide. BMJ 328 788 Available: http://www.bmj.com/cgi/content/full/328/7443/788. Accessed 22 July 2009 15070625 PMC383368 source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Take the example of poorly developed computer decision support functionality, which can result in the issuing of erroneous prescribing support and advice, as was demonstrated by Fernando et al. in their assessment of primary care prescribing software [33].”
§ Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation > Recognising the Risk of Harm
What the sources say
- Not checked yetcites [33] Fernando B Boki SP Savelyich A Avery J Sheikh A 2004 Prescribing safety features of general practice computer systems: evaluation using simulated test cases. BMJ 328 1171 1172 15142922 PMC411094 source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“eHealth may compromise patient safety in a number of ways.”
§ Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation > Recognising the Risk of Harm
What the sources say
- Not checked yetcites [11] Mudur G 2004 The great technological divide. BMJ 328 788 Available: http://www.bmj.com/cgi/content/full/328/7443/788. Accessed 22 July 2009 15070625 PMC383368 source
- Not checked yetcites [12] Ammenwerth E Shaw NT 2005 Bad Health Informatics Can Kill – Is Evaluation the Answer? Editorial. Methods Inf Med 44 1 3 Available: http://iig.umit.at/dokumente/z29.pdf. Accessed 22 July 2009 15778787 source
- Not checked yetcites [13] Han YY Carcillo JA Ventkataraman ST Clark RSB Watson SR 2005 Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 116 1506 1512 Available: http://pediatrics.aappublications.org/cgi/content/full/116/6/1506?maxtoshow=&HITS=10&hits=10&RESULTFORMAT=&fulltext=cpoe&searchid=1&FIRSTINDEX=0&resourcetype=HWCIT Accessed 22 July 2009 16322178 source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Within biomedicine, randomised controlled trials (RCTs) are often seen as the “gold standard” methodological approach, rightly so because of their unique ability to control for the impact of known and unknown confounding factors [16].”
§ Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation > Implications for Evaluation > Box 3. Suggested seven steps from statement of a problem to definition of a solution [37]
What the sources say
- Not checked yetcites [16] Kaplan B 2001 Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 64 39 56 Available: http://66.102.1.104/scholar?q=cache:w4_uGY7cGi0J:scholar.google.com/&hl=en Accessed 22 July 2009 11673101 source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
9 more claims
Create a free account to see all 25 claims and to verify the 28 references still open.
- Probably not a substantial claim. 8
- It may state two separate claims as one. 4
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Also of note is that we found virtually no evidence in support of the cost-effectiveness claims (Tables 2– 11) that are frequently being made by policy makers when constructing business cases to raise funding for the large-scale eHealth deployments that are now taking place in many parts of the world [1].”
From The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview (2011) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Mock-ups, storyboards, or paper prototypes [101] are created and tested sequentially and iteratively with intended users [23,26,29,48,60] and, as a result, the prototypes are refined.”
From A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies (2011) · cites this paper as [23]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This is noted in several of the reviewed frameworks [23,29,48,51,60].”
From A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies (2011) · cites this paper as [23]
In the citing paper
“Although there is international pressure for hospitals to implement electronic records, based on the belief that these systems will transform the quality and safety of healthcare [1, 2], there is little empirical evidence to support this.”
From Exploring the implementation of an electronic record into a maternity unit: a qualitative study using Normalisation Process Theory (2017) · cites this paper as [1]
In the citing paper
“IT systems can introduce harms as well as benefits.”
From Evaluating eHealth: How to Make Evaluation More Methodologically Robust (2009) · cites this paper as [1]
Citations
Cites
Han, Y. Y., Carcillo, J. A., Venkataraman, S. T., Clark, R. S. B., Watson, R. S., Nguyen, T. C., Bayir, H., & Orr, R. A. (2005) · Pediatrics · doi:10.1542/peds.2005-1287 · Cited by 1,149
Cognitive and usability engineering methods for the evaluation of clinical information systems.
Kushniruk, A. W., & Patel, V. L. (2004) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2004.01.003 · Cited by 760 · cited 6×
“eHealth interventions whilst minimising risks, we must be able simultaneously to evaluate eHealth interventions while they are being designed, developed, and deployed [20].”
BackgroundInfluentialWhat Is eHealth (4): A Scoping Exercise to Map the Field
Pagliari, C., Sloan, D., Gregor, P., Sullivan, F., Detmer, D., Kahan, J. P., Oortwijn, W., & MacGillivray, S. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e9 · Cited by 479 · cited 2×
“One such definition that includes these various facets is that proposed by Pagliari [30], who defined eHealth as:”
BackgroundKaplan, B. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00184-8 · Cited by 380 · cited 3×
“But whilst RCTs and similar experimental design methods may be appropriate for studying interventions under controlled clinical conditions, these design methodologies alone are often less well suited to evaluate the impact of eHealth interventions in a complex environment or to study the effect they have on the delive…”
BackgroundElectronic Health Records: A Systematic Review on Quality Requirements
Ammenwerth, E., & Hoerbst, A. (2010) · Methods of Information in Medicine · doi:10.3414/me10-01-0038 · Cited by 303
Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions
(2009) · National Academies Press eBooks · doi:10.17226/12572 · Cited by 282
The World Bank participation sourcebook.
(1996) · World Bank, Washington, DC eBooks · doi:10.1596/0-8213-3558-8 · Cited by 264
Visions and strategies to improve evaluation of health information systems
Ammenwerth, E., Brender, J., Nykänen, P., Prokosch, H.-U., Rigby, M., & Talmon, J. (2004) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2004.04.004 · Cited by 233 · cited 1×
“The argument about the need rigorously to evaluate medical technologies themselves as well as their social and economic impact is not new [14–16].”
BackgroundElectronic Medical Records: Privacy, Confidentiality, Liability
Steward, M. (2005) · Journal of Legal Medicine · doi:10.1080/01947640500364762 · Cited by 191
Electronic Medical Records vs. Electronic Health Records:Yes,There Is a Difference
Yang, H. (2007) · Chinese Hospitals · Cited by 174
Research works printed but not linked: 1
- National Center for Health Statistics Health 2007 United States, 2007, With Chartbook on Trends in the Health of Americans Hyattsville (Maryland). Available: http://www.cdc.gov/nchs/data/hus/hus07.pdf. Accessed 22 July 2009 20698068
Cited by
Items 101–200 of 313
Marquard, J. L., & Zayas‐Cabán, T. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000338 · Cited by 31
Krasuska, M., Williams, R., Sheikh, A., Franklin, B. D., Hinder, S., TheNguyen, H., Lane, W., Mozaffar, H., Mason, K., Eason, S., et al. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100429 · Cited by 31
“19 20 This type of evaluation collects evidence on the processes involved in stimulating digital transformation through HIT change initiatives and on an array of emerging outcomes including consequences not anticipated/intended by programme architects.”
Self-citationClinical decision support systems: Need for evidence, need for evaluation
Ammenwerth, E., Nykänen, P., Rigby, M., & de Keizer, N. F. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2013.05.001 · Cited by 32
van der Hout, A., van Uden-Kraan, C. F., Holtmaat, K., Jansen, F., Lissenberg-Witte, B. I., Nieuwenhuijzen, G. A. P., Hardillo, J. A., Baatenburg de Jong, R. J., Tiren-Verbeet, N. L., Sommeijer, D. W., et al. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100429 · Cited by 29
Telemanagement in asthma: an innovative and effective approach
Van Gaalen, J. L., Hashimoto, S., & Sont, J. K. (2012) · Current Opinion in Allergy & Clinical Immunology · doi:10.1097/aci.0b013e3283533700 · Cited by 28
Vancea, M., & Solé‐Casals, J. (2016) · Aging and Disease · doi:10.14336/ad.2015.1214 · Cited by 34
A Robust and Reusable ECG-Based Authentication and Data Encryption Scheme for eHealth Systems
Huang, P., Li, B., Guo, L., Jin, Z., & Chen, Y. (2016) · Globecom · doi:10.1109/glocom.2016.7841541 · Cited by 31
“For example, UK has invested at least £ 12.8 billion in National Programme for Information Technology (NPfIT) for the National Health Service, and the Obama administration in the United States has similarly committed to a $38 billion eHealth investment in healthcare [3].”
BackgroundInfluentialA systematic review of eHealth technologies for breast cancer supportive care
Gyawali, B., Bowman, M., Sharpe, I., Jalink, M., Srivastava, S., & Wijeratne, D. T. (2023) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2023.102519 · Cited by 28
Mobile phone text message reminders: Measuring preferences of people with antipsychotic medication
Kauppi, K., Kannisto, K. A., Hätönen, H. M., Anttila, M., Löyttyniemi, E., Adams, C. E., & Valimaki, M. A. (2015) · Schizophrenia Research · doi:10.1016/j.schres.2015.07.044 · Cited by 29
“This analysis serves as a fidelity measure to support a later assessment of how intervention works and to gain knowledge of how intervention could be better in the future (Catwell and Sheikh, 2009).”
Drawing on human factors engineering to evaluate the effectiveness of health information technology
Cresswell, K. M., BLANDFORD, A. E., & Sheikh, A. (2017) · Journal of the Royal Society of Medicine · doi:10.1177/0141076817712252 · Cited by 31
Self-citationA Study of eHealth from the Perspective of Social Sciences
Uribe-Toril, J., Ruiz–Real, J. L., & Soriano, B. J. N. (2021) · Healthcare · doi:10.3390/healthcare9020108 · Cited by 30
Baines, D. L., Gahir, I. K., Hussain, A., Khan, A., Schneider, P. J., Hasan, S. S., & Babar, Z. (2018) · Frontiers in Pharmacology · doi:10.3389/fphar.2018.00678 · Cited by 33
Trends in telemedicine assessment indicate neglect of key criteria for predicting success
Jackson, D. E., & McClean, S. I. (2012) · Journal of Health Organization and Management · doi:10.1108/14777261211251553 · Cited by 31
Harst, L., Wollschlaeger, B., Birnstein, J., Fuchs, T., & Timpel, P. (2021) · International Journal of Integrated Care · doi:10.5334/ijic.5529 · Cited by 28
Colucci, M., Baldo, V., Baldovin, T., & Bertoncello, C. (2017) · Health Informatics Journal · doi:10.1177/1460458217747109 · Cited by 31
Spithoff, S., Mathieson, S., Sullivan, F., Guan, Q., Sud, A., Hum, S., & O'Brien, M. A. (2020) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2020.04.190199 · Cited by 25
Cresswell, K. M., Coleman, J. J., Slee, A., Morrison, Z., & Sheikh, A. (2013) · Journal of the Royal Society of Medicine · doi:10.1177/0141076813502955 · Cited by 25
“19 In doing so, it is crucial that unintended and unanticipated consequences (such as new prescribing and administration errors) are continuously explored, collated and addressed 20 ; and that opportunities from extending ePrescribing use or introducing new associated functionality (e.g. reporting…”
BackgroundSelf-citationMuhammad, I., Teoh, S. Y., & Wickramasinghe, N. S. (2012) · International Journal of Actor-Network Theory and Technological Innovation · doi:10.4018/jantti.2012040105 · Cited by 28
“…healthcare service delivery where different human and non-human actors interact with each other in multiple ways (Ammenwerth, Iller, & Mahler, 2006; Catwell & Sheikh, 2009; Cresswell, Worth, & Sheikh, 2010; Lorenzi, Kouroubali, Detmer, & Bloomrosen, 2009; DesRoches et al., 2008; André, Ringdal,…”
BackgroundDuplaga, M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0153173 · Cited by 24
Tieman, J., & Bradley, S. L. (2013) · Australian Journal of Primary Health · doi:10.1071/py13030 · Cited by 24
Lee, A., Sandvei, M., Asmussen, H. C., Skougaard, M., Macdonald, J., Závada, J., Bliddal, H., Taylor, P. C., & Gudbergsen, H. (2018) · JMIR Research Protocols · doi:10.2196/resprot.9521 · Cited by 26
“Ongoing and systematic evaluation is important in the development of digital health solutions [16].”
BackgroundResultInfluentialTeledermatology in times of COVID‐19
Stadler, P., Senner, S., Frey, S., Clanner-Engelshofen, B. M., Frommherz, L. H., French, L. E., & Reinholz, M. (2021) · The Journal of Dermatology · doi:10.1111/1346-8138.15812 · Cited by 24
Rahimi, K., Velardo, C., Triantafyllidis, A. K., Conrad, N., Shah, S. A., Chantler, T., Mohseni, H., Stoppani, E., Moore, F., Paton, C., et al. (2015) · European Heart Journal - Quality of Care and Clinical Outcomes · doi:10.1093/ehjqcco/qcv013 · Cited by 24
Budrionis, A., Wynn, R., Marco-Ruiz, L., Yigzaw, K. Y., Bergvik, S., Oyeyemi, S. O., & Bellika, J. G. (2020) · Journal of Medical Internet Research · doi:10.2196/13118 · Cited by 25
Approaches to Recording Drug Allergies in Electronic Health Records: Qualitative Study
Fernando, B., Morrison, Z., Kalra, D., Cresswell, K. M., & Sheikh, A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0093047 · Cited by 22
“Due consideration to human and organisational factor considerations is crucial to the successful implementation and adoption of these systems [13,14].”
Self-citationConstructive eHealth evaluation: lessons from evaluation of EHR development in 4 Danish hospitals
Høstgaard, A. M. B., Bertelsen, P., & Nøhr, C. G. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0444-2 · Cited by 27
“Focusing on improving eHealth development or on implementation, they provide feedback between all groups involved, without any specific focus on end-users [5, 32, 77].”
BackgroundMaathuis, I., Niezen, M. G. H., Buitenweg, D. C., Bongers, I. L., & van Nieuwenhuizen, C. (2019) · Science and Engineering Ethics · doi:10.1007/s11948-019-00142-y · Cited by 22
“Because ICT uptake in health care is often several years behind other sectors (Catwell and Sheikh 2009), many popular applications slowly diffuse into health care through a commercial “push” (Scholz 2008).”
BackgroundHarsha, P., Paul, J., Chong, M., Buckley, N., Tidy, A., Clarke, A., Buckley, D. N., Sirko, Z., Vanniyasingam, T., Walsh, J., et al. (2019) · JMIR Medical Informatics · doi:10.2196/14603 · Cited by 20
Cresswell, K. M., Bates, D. W., Phansalkar, S., & Sheikh, A. (2011) · BMJ Quality & Safety · doi:10.1136/bmjqs.2010.048934 · Cited by 20
“44 45 A particularly important issue in relation to this is the relatively common occurrence of clinicians over-riding or ignoring alerts.”
BackgroundSelf-citationTowards systemic evaluation in turbulent times – Second-order practice shift
Schmidt-Abbey, B., Reynolds, M., & Ison, R. L. (2020) · Evaluation · doi:10.1177/1356389020909120 · Cited by 19
Peckham, D. G., Etherington, C., White, H., Mehta, A., Shaw, N., Morton, A. M., Pollard, K., Lee, T., Brownlee, K. G., Taylor, J., et al. (2014) · Journal of Cystic Fibrosis · doi:10.1016/j.jcf.2014.03.002 · Cited by 20
“In the US $38 billion has been committed to e-health [2].”
BackgroundPlanned implementations of ePrescribing systems in NHS hospitals in England: a questionnaire study
Crowe, S., Cresswell, K. M., Avery, A. J., Slee, A., Coleman, J. J., & Sheikh, A. (2010) · JRSM Short Reports · doi:10.1258/shorts.2010.010040 · Cited by 25
Self-citationVera, M. A. D., Sadatsafavi, M., Tsao, N., Lynd, L. D., Lester, R. T., Gastonguay, L., Galo, J. S., FitzGerald, J. M., Brasher, P. M. A., & Marra, C. A. (2014) · Trials · doi:10.1186/1745-6215-15-488 · Cited by 20
Rönnby, S., Lundberg, O., Fagher, K., Jacobsson, J., Tillander, B. M., Gauffin, H., Hansson, P.-O., Dahlström, Ö., & Timpka, T. (2018) · JMIR mhealth and uhealth · doi:10.2196/10270 · Cited by 21
“In any setting where novel technology is incorporated with established practices, knowledge of user acceptance is important [10].”
Gray, C. S., Gravesande, J., Hans, P. K., Nie, J. X., Sharpe, S., Loganathan, M., Lyons, R., & Cott, C. A. (2019) · JMIR Formative Research · doi:10.2196/11950 · Cited by 19
Christie, H. L., Boots, L. M. M., Hermans, I., Govers, M., Tange, H. J., Verhey, F. R. J., & de Vugt, M. E. (2021) · JMIR Aging · doi:10.2196/24724 · Cited by 21
Takian, A. (2012) · Studies in health technology and informatics · doi:10.3233/978-1-61499-101-4-901 · Cited by 23
Harvey, J., Avery, A. J., Hibberd, R. E., & Barber, N. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-16 · Cited by 26
“Consequently, there needs to be a continuous engagement between the system and the social environment to render a product usable [15].”
BackgroundWorth, A., Pinnock, H., Fletcher, M. J., Hoskins, G., Levy, M. L., & Sheikh, A. (2010) · Primary Care Respiratory Journal · doi:10.4104/pcrj.2010.00070 · Cited by 20
Self-citationIs there and do we need evidence on eHealth interventions?
Al-Shorbaji, N. (2013) · IRBM · doi:10.1016/j.irbm.2012.11.001 · Cited by 17
Cresswell, K. M., Smith, P. A., Swainson, C. P., Timoney, A., & Sheikh, A. (2016) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v23i3.842 · Cited by 16
Self-citationHackett, J., Allsop, M. J., Taylor, S., Bennett, M., & Bewick, B. M. (2020) · Health Informatics Journal · doi:10.1177/1460458220906289 · Cited by 16
Cenci, A., Ilskov, S. J., Andersen, N. S., & Chiarandini, M. (2023) · AI and Ethics · doi:10.1007/s43681-023-00274-9 · Cited by 17
“…innovation (an exception is Maathuis et al., [84]) where ethical reflection is urgently needed to figure out which values can be ascribed in similar tools and how they would shape future technical developments and actual uses (on the need of ethical insights, see [27, 44, 82, 83, 110].”
Inpatient Clinical Information Systems
Cresswell, K. M., & Sheikh, A. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-809523-2.00002-9 · Cited by 17
Self-citationTahsin, F., Tracy, S., Chau, E., Harvey, S., Loganathan, M., McKinstry, B., Mercer, S. W., Nie, J. X., Ramsay, T., Thavorn, K., et al. (2021) · Digital Health · doi:10.1177/20552076211045579 · Cited by 17
“31 Consistent with recommendations on how to evaluate and test complex interventions using multiple data sources, 29,32,33 the following data sources were used to answer the research question of this study: (1) Device generated data to measure actual usage; (2) The 2002 version of PSSUQ 34 to…”
MethodOrsolini, L., Bellagamba, S., Marchetti, V., Menculini, G., Valenta, S. T., Salvi, V., & Volpe, U. (2022) · Healthcare · doi:10.3390/healthcare10020390 · Cited by 15
Pirtle, B., & Chandra, A. (2011) · American Journal of Health Sciences (AJHS) · doi:10.19030/ajhs.v2i2.6627 · Cited by 15
Talmon, J. L., Ammenwerth, E., Brender, J., Rigby, M., Nykänen, P., & de Keizer, N. F. (2011) · Methods of Information in Medicine · doi:10.3414/me10-01-0072 · Cited by 15
Telepsychiatry in the developing world: Whither promised joy?
Chakrabarti, S., & Shah, R. (2016) · Indian Journal of Social Psychiatry · doi:10.4103/0971-9962.193200 · Cited by 13
Scantlebury, A., Sheard, L., Fedell, C., & Wright, J. (2021) · Digital Health · doi:10.1177/20552076211010033 · Cited by 15
Slattery, B. W., Haugh, S., Francis, K., O’Connor, L. L., Barrett, K., Dwyer, C. P., O’Higgins, S., Egan, J., & McGuire, B. E. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0414-x · Cited by 13
Crawford, F., Bekker, H., Young, M. E., & Sheikh, A. (2010) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v18i4.781 · Cited by 16
Self-citationeHealth: advantages, disadvantages and guiding principles for the future. (Preprint)Preprint
Soriano, B. J. N., Duarte, S. G., Fernández‐Alonso, A. M., Perales, A. B., & Carreño, T. P. (2019) · doi:10.2196/preprints.15366 · Cited by 13
Verweij, L., Metsemakers, S. J. J. P. M., Ector, G. I. C. G., Rademaker, P., Bekker, C. L., van Vlijmen, B., van der Reijden, B. A., Blijlevens, N. M. A., & Hermens, R. (2023) · Journal of Medical Internet Research · doi:10.2196/45259 · Cited by 14
La santé numérique : un levier pour améliorer l’accessibilité aux soins de santé au Québec
François, J., & Audrain‐Pontevia, A. (2020) · Revue Organisations & territoires · doi:10.1522/revueot.v29n3.1196 · Cited by 15
Anaraki, N. R., Mukhopadhyay, M., Jewer, J., Patey, C., Norman, P., Hurley, O., Etchegary, H., & Asghari, S. (2024) · BMC Health Services Research · doi:10.1186/s12913-024-11345-w · Cited by 13
“…are generally reported to have a positive impact on patient care [21] wherein its impact ranges from an increase in the availability of patient information, enhanced communication between healthcare workers, improved healthcare accessibility, and reduced patient wait times [13, 22, 23].”
BackgroundBhattacharjya, S., Lenker, J., & Ghosh, R. (2022) · Assistive Technology · doi:10.1080/10400435.2022.2028936 · Cited by 14
Naaldenberg, J., & Aarts, N. (2020) · BMJ Global Health · doi:10.1136/bmjgh-2020-003858 · Cited by 11
Lucivero, F. (2017) · The AMA Journal of Ethic · doi:10.1001/journalofethics.2017.19.4.stas1-1704 · Cited by 11
Schuit, A. S., van Zwieten, V., Holtmaat, K., Cuijpers, P. J., Eerenstein, S. E. J., Leemans, C. R., Vergeer, M. R., Voortman, J., Karagozoglu, K. H., van Weert, S., et al. (2021) · European Journal of Cancer Care · doi:10.1111/ecc.13497 · Cited by 13
Smith, M. Y., Frise, S., Feron, J., & Marshall, R. (2022) · Drug Safety · doi:10.1007/s40264-022-01165-4 · Cited by 12
A Clinician’s View of Next-Generation Remote Healthcare System
Puddu, P. E., D’Ambrosi, A., Scarparo, P., Centaro, E., Torromeo, C., Schiariti, M. S., Fedele, F., & Gensini, G. F. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8842-2_1 · Cited by 13
Chung, J., Chaudhuri, S., Le, T., Chi, N., Thompson, H. J., & Demiris, G. (2015) · Educational Gerontology · doi:10.1080/03601277.2015.1071586 · Cited by 11
“In addition, with the attention to patient-centered care, there is an increasing demand for using consumer e-health applications for older adults in various daily settings such as home, community, work-places, or recreational facilities (Catwell & Sheikh, 2009).”
BackgroundKhoumbati, K., Abbasi, M. A., Shah, S. G. S., & Stergioulas, L. K. (2018) · Advances in theory and practice of emerging markets · doi:10.1007/978-3-319-75013-2_18 · Cited by 10
Technical evaluation of an e-health platform.
Lo, O., Lü, F., Buchanan, W. J., & Thüemmler, C. (2012) · Edinburgh Napier Research Repository (Edinburgh Napier University) · Cited by 12
Using ANT to Understand Key Issues for Successful E-Health Solutions
Muhammad, I., Zwicker, M., & Wickramasinghe, N. S. (2013) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2013.588 · Cited by 12
van der Ven, J., van den Bemt, B. J. F., van Dijk, L., Opdam, M., Haegens, L. L., Vriezekolk, J. E., & Verhoef, L. M. (2023) · JMIR Human Factors · doi:10.2196/44885 · Cited by 10
“This might foster the efficiency and patient-centeredness of health care, for instance, through intensification of home monitoring and tailoring information to personal needs [4-6].”
BackgroundSchoen, D. E., Glance, D., & Thompson, S. (2015) · Journal of Foot and Ankle Research · doi:10.1186/s13047-015-0128-z · Cited by 10
Salisbury, C., O’Cathain, A., Thomas, C., Edwards, L., Montgomery, A. A., Hollinghurst, S. P., Large, S., Nicholl, J. P., Pope, C. J., Rogers, A. E., et al. (2017) · Programme Grants for Applied Research · doi:10.3310/pgfar05010 · Cited by 9
E-game in Healthcare: As an E-intervention to Promote Public Health.
Karami, M., & Hafizi, N. (2016) · PubMed · Cited by 8
Tailored Web-Based Smoking Interventions and Reduced Attrition: Systematic Review and Meta-Analysis
Shah, A., Chaiton, M., Baliunas, D. O., & Schwartz, R. M. (2020) · Journal of Medical Internet Research · doi:10.2196/16255 · Cited by 8
Chapel, B., Alexandre, F., Héraud, N., Ologeanu‐Taddei, R., Cases, A.-S., Bughin, F., & Hayot, M. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08370-y · Cited by 9
Qualitative Evaluation of Quality Measurement Within Emergency Clinician–Staffed Telehealth Programs
Whitehead, D. C., Jaffe, T. A., Hayden, E. M., & Zachrison, K. S. (2022) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2022.05.026 · Cited by 8
How Using ANT Can Assist to Understand Key Issues for Successful e-Health Solutions
Muhammad, I., Zwicker, M., & Wickramasinghe, N. S. (2013) · International Journal of Actor-Network Theory and Technological Innovation · doi:10.4018/jantti.2013070105 · Cited by 9
“…environment of healthcare service delivery not to mention the inherent complexities of healthcare operations (Ammenwerth, Iller, & Mahler, 2006; Catwell & Sheikh, 2009; Cresswell, Worth, & Sheikh, 2010; Cresswell, Worth, & Sheikh, 2011; Lorenzi et al., 2009; Lubitz & Wickramasinghe, 2006).”
BackgroundHandwritten to Electronic Prescriptions: Emerging Views and Practices, Saudi Arabia
Qureshi, N. A. (2014) · Journal of Advances in Medicine and Medical Research · doi:10.9734/bjmmr/2014/10962 · Cited by 8
“Financial incentives to providers for implementing EPS in USA have the use of e-prescribing, which has also increased the likelihood of prescribing generic medications that has considerably decreased medication costs [101].”
BackgroundQuantitative metrics for evaluating the phased roll-out of clinical information systems
Wong, D., Wu, N., & Watkinson, P. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.06.007 · Cited by 8
“In the event of an unexpectedly high number of transitions, the approach here would allow a change of strategy mid-deployment as espoused by Catwell [26].”
MethodHuynh, T., Kroh, J., & Schultz, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0293550 · Cited by 8
Cenci, A. (2025) · AI and Ethics · doi:10.1007/s43681-024-00636-x · Cited by 8
“Similarly, people with cognitive/mental problems, including dementia, barely participate to the design of eHealth/mHealth technologies ([1, 22, 25, 37, 41, 99, 100, 122] inter alia ).”
The Need for a Socio-Technical Analysis in E-Health
Muhammad, I., Teoh, S. Y., & Wickramasinghe, N. S. (2013) · International Journal of E-Health and Medical Communications · doi:10.4018/jehmc.2013040105 · Cited by 8
“…environments are not just technical in nature but are multidimensional and include organisational, cultural, legal and social considerations as well (Ammenwerth et al., 2006; Catwell & Sheikh, 2009; Cresswell et al., 2011; Lorenzi et al., 2009; DesRoches et al., 2008; André et al., 2008).”
BackgroundChavannes, N. H., Puy, R. S. D., & Bai, C. (2015) · npj Primary Care Respiratory Medicine · doi:10.1038/npjpcrm.2015.45 · Cited by 7
A Parental mHealth Resource Targeting Emergent Literacy: An Experimental Study
Scheepers, C., Eccles, R., Abdoola, S., Graham, M. A., & van der Linde, J. (2020) · Early Childhood Education Journal · doi:10.1007/s10643-020-01052-6 · Cited by 7
BUSINESS MODELS FOR SUSTAINED EHEALTH IMPLEMENTATION: LESSONS FROM TWO CONTINENTS
Dyk, L. V., Wentzel, M. J., van Limburg, A. H. M., van Gemert‐Pijnen, J. E. W. C., & Schutte, C. (2012) · University of Twente Research Information · Cited by 8
Understanding Stakeholder Interests and Perspectives in Evaluations of Health IT
Lee, L., & Sheikh, A. (2016) · Studies in health technology and informatics · doi:10.3233/978-1-61499-635-4-53 · Cited by 8
Self-citationManaging patients’ data with clinical decision support systems: a factual assessment
Razmak, J., Bélanger, C. H., & Farhan, W. (2018) · Journal of Decision System · doi:10.1080/12460125.2018.1533159 · Cited by 10
“In evaluating the bene fi ts of implementing new technologies in healthcare, scholars must explore the relationship between the literature reporting on those bene fi ts and the reality of technology application in actual settings (Blumenthal & Tavenner, 2010; Catwell & Sheikh, 2009).”
BackgroundBlasi, F., Caiani, E. G., Cereda, M. G., Donetti, D., Montorsi, M., Panella, V., Panina, G., Pelagalli, F., & Speroni, E. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.876625 · Cited by 8
Kayser, L., Furstrand, D., Rasmussen, E. N., Monberg, A.-C., & Karnoe, A. (2022) · Informatics · doi:10.3390/informatics9030069 · Cited by 6
Gaddi, A. V., & Capello, F. (2013) · Springer eBooks · doi:10.1007/978-88-470-5253-6_1 · Cited by 8
Jones, E. B., Swain, M. J., Patel, V., & Furukawa, M. F. (2014) · Healthcare · doi:10.1016/j.hjdsi.2013.12.015 · Cited by 6
Muhammad, I., & Wickramasinghe, N. S. (2014) · Advances in human and social aspects of technology book series · doi:10.4018/978-1-4666-6126-4.ch002 · Cited by 7
Maierhöfer, S., Waltering, I., Jacobs, M., Würthwein, G., Appelrath, M., Koling, S., & Hempel, G. (2022) · Journal of Pharmaceutical Policy and Practice · doi:10.1186/s40545-022-00495-z · Cited by 6
“The use of health information technology, such as clinical decision support systems (CDSS), is a promising strategy in this regard and is promoted by public policy in several countries [15, 16].”
BackgroundA Multimodal Emotion-Focused e-health Monitoring Support System
Chen, J., Hu, B., Li, N., Mao, C., & Moore, P. T. (2013) · ... International Conference on Complex, Intelligent and Software Intensive Systems · doi:10.1109/cisis.2013.92 · Cited by 6
Gjestsen, M. T., Wiig, S., & Testad, I. (2019) · Journal of Medical Internet Research · doi:10.2196/12797 · Cited by 7
Haugh, S., O’Connor, L. L., Slattery, B. W., Hanlon, M. L. D., Flynn, J., Quinn, S., Jennings, C., & McGuire, B. E. (2019) · HRB Open Research · doi:10.12688/hrbopenres.12953.1 · Cited by 6
Pezaro, S., & Lilley, L. (2015) · Women and Birth · doi:10.1016/j.wombi.2015.04.008 · Cited by 6
Approaches to Assess E-Health Programs: A Scoping Review.
Joshi, N. K., Bhardwaj, P., Saxena, D. B., Suthar, P., & Joshi, V. (2021) · Indian Journal of Community Medicine · doi:10.4103/ijcm.ijcm_340_20 · Cited by 3
“[13] They also emphasized that continuous systematic evaluations of e‑Health interventions are needed.”
BackgroundMethodInfluentialEvaluation of a web-based telehealth system
Dhillon, J. S., Wünsche, B. C., & Lutteroth, C. (2012) · International Conference of the NZ Chapter of the ACM's Special Interest Group on Human-Computer Interaction (CHINZ) · doi:10.1145/2379256.2379260 · Cited by 6
“Any healthcare system ideally should be evaluated while it is being designed and developed to maximise benefits and minimise risks of deployment at the real setting [16].”
BackgroundAdopting Technologies in Medical and Healthcare Systems
Mahendrakar, N. (2025) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2025.73297 · Cited by 5
O’Connor, S., Tilston, G., Jones, O., Sharma, A., Ormesher, L., Quinn, B., Wilson, A. J., Myers, J., Peek, N. B., & Palin, V. (2024) · BMC Medicine · doi:10.1186/s12916-024-03489-7 · Cited by 5
“EHRs are often used by clinical teams in hospital to record maternal and neonatal health events, support clinical decision-making and care delivery [18–20].”
Scherrens, A., Jacobs, A., Beernaert, K., Pardon, K., Raemdonck, E., Fallon, M., Cresswell, K. M., Farič, N., Williams, R., Lundeby, T., et al. (2024) · Palliative Care and Social Practice · doi:10.1177/26323524241296143 · Cited by 7
“Together this maximises the chances of successful adoption and implementation, 48,70,71 whilst minimising chances of introducing new risks.”
Developing a patient‐led electronic feedback system for quality and safety within Renal PatientView
Giles, S. J., Reynolds, C., Heyhoe, J., & Armitage, G. (2016) · Journal of Renal Care · doi:10.1111/jorc.12186 · Cited by 7
Metadata
- Authors
- Catwell, L., & Sheikh, A.
- Affiliations
- University of Edinburgh (GB)
- Corresponding author
- Sheikh, Aziz
- ORCID
- Sheikh, Aziz
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2009-08-18
- Volume
- 6
- Issue
- 8
- Pages
- e1000126
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC2719100
- OpenAlex
- W2052843612
- Cited by
- 307
- References
- 21
- Citations by year
- 2026: 2 · 2025: 10 · 2024: 16 · 2023: 11 · 2022: 25 · 2021: 17 · 2020: 17 · 2019: 22 · 2018: 19 · 2017: 25 · 2016: 19 · 2015: 20 · 2014: 18 · 2013: 32 · 2012: 21
- FWCI
- 31.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Funders
- British Lung Foundation
- Asthma and Lung UK
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Keywords
- digital health interventions, continuous assessment, systemic evaluation, digital health
- MeSH terms
- Evaluation Studies as Topic, Humans, Medical Informatics, Research Design, Telecommunications, Internet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
