Evaluating eHealth: How to Make Evaluation More Methodologically Robust
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.
Abstract
In the third in a series of articles on evaluating eHealth, Richard Lilford and colleagues consider the evaluation of health IT systems as they are rolled out following preimplementation testing.
Overview
CoChat Summary
TL;DR
This essay argues that eHealth evaluations should be methodologically pluralistic, using both quantitative and qualitative methods, and should usually assess systems at organizational/workgroup level rather than only at the individual patient level. It also argues that external summative evaluation should be kept distinct from internal formative feedback, because mixing them can reduce the generalisability of findings.
Key findings
- The authors state that evaluation of IT systems often requires a mixed-methods approach and that there is “no” consensus yet comparable to randomized trials for drugs.
- They argue that the primary unit of analysis for many IT systems is likely the organizational/workgroup level (e.g., wards, hospitals, practices), because systems may not be restrictable to individuals and “contamination” can dilute effects.
- They distinguish formative assessment as timely feedback to implementers and summative assessment as generalisable knowledge for decision makers beyond the life cycle of a particular application.
- They recommend external assessments because they can be independent, use masked assessment of endpoints, include contemporaneous controls, and use before-and-after controlled designs to reduce bias.
Methods
- Not stated in the paper.
- The paper is a narrative essay/review drawing on the authors’ experience with National Programme for Information Technology (NPFIT) evaluation in England and on cited literature.
- The main “analysis” is conceptual and argumentative: it synthesizes methodological principles for evaluating eHealth, including mixed methods, organizational-level observation, and separation of formative versus summative evaluation.
Limitations
- The paper notes a tension between formative and summative evaluation: feedback used to improve implementation can alter summative results and reduce generalisability.
- It states that distinctions between formative and summative assessment are “not watertight,” so these categories overlap in practice.
- Evident from the text: this is a position essay rather than an empirical study, so it does not present original outcome data or a systematic test of the proposed methods.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
0 Valid · 0 Not valid · 12 Not checked · 27 Own results and interpretations
In the paper
“little evidence is available on the effect of multifunctional commercially developed systems”
§ Evaluating eHealth: How to Make Evaluation More Methodologically Robust > eHealth in the UK
What the sources say
- Not checked yetcites [3] B Chaudhry, 2006, Systematic review: impact of health information technology on quality, efficiency, and costs of medical care., Ann Intern Med source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“they often need to be studied at the organisational level [17].”
§ Evaluating eHealth: How to Make Evaluation More Methodologically Robust > Observations at Patient and System Level
What the sources say
- Not checked yetcites [17] J Hippisley-Cox, 2003, The electronic patient record in primary care–regression or progression? A cross sectional study., BMJ source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“External assessment can provide expertise in the measurement of endpoints (for example, error rates/quality) where special expertise is needed.”
§ Evaluating eHealth: How to Make Evaluation More Methodologically Robust > External and Internal Assessments
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The primary unit of analysis in evaluation of IT systems is therefore likely to be at the organisational/workgroup level (e.g., wards, hospitals, practices).”
§ Evaluating eHealth: How to Make Evaluation More Methodologically Robust > Observations at Patient and System Level
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“if the results of formative research are fed back into implementations, this may influence summative results.”
§ Evaluating eHealth: How to Make Evaluation More Methodologically Robust > Formative and Summative Assessment
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
10 more claims
Create a free account to see all 39 claims and to verify the 12 references still open.
- Probably not a substantial claim. 19
- It may state two separate claims as one. 8
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Finally, our formative work might have had an impact on the findings from our summative research, and this could therefore be seen by some as a limitation. We, however, view this as a strength as it provided an opportunity to share with hospitals any early lessons learnt.98”
The cited source says what the claim attributes to it.
In the citing paper
“Several authors [1,2,4,6,62,76-87] note that a broader view is needed to assess the overall impact on health care.”
From A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies (2011) · cites this paper as [62]
In the citing paper
“This is unsurprising given that only limited national-scale implementations of EHR systems have taken place to date, and there is as a consequence little published evidence to inform larger scale evaluations [28].”
In the citing paper
“In our view, the conditions for implementation must be taken into account right from the start (contextual inquiry and value specification; see Figure 1) [62].”
From A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies (2011) · cites this paper as [62]
Citations
Cites
Chaudhry, B., Wang, J., Wu, S., Maglione, M., Mojica, W., Roth, E., Morton, S. C., & Shekelle, P. G. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-10-200605160-00125 · Cited by 3,159
Information Technology and Organizational Change: Causal Structure in Theory and Research
Markus, M. L., & Robey, D. (1988) · Management Science · doi:10.1287/mnsc.34.5.583 · Cited by 2,108
Developing and Evaluating Complex Interventions: New Guidance
Craig, P., Dieppe, P. A., MacIntyre, S., Michie, S., Nazareth, I., & Petticrew, M. P. (2006) · UCL Discovery (University College London) · Cited by 1,974
Poissant, L. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1700 · Cited by 901
Lilford, R., Mohammed, M. A., Spiegelhalter, D., & Thomson, R. (2004) · The Lancet · doi:10.1016/s0140-6736(04)15901-1 · Cited by 448
Self-citationKaplan, B. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00184-8 · Cited by 380
Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation
Catwell, L., & Sheikh, A. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000126 · Cited by 307
A controlled trial of smart infusion pumps to improve medication safety in critically ill patients*
Rothschild, J. M., Keohane, C. A., Cook, E. F., Orav, E. J., Burdick, E., Thompson, S., Hayes, J., & Bates, D. W. (2005) · Critical Care Medicine · doi:10.1097/01.ccm.0000155912.73313.cd · Cited by 301
The Use of Electronic Medical Records: Communication Patterns in Outpatient Encounters
Makoul, G., Curry, R. H., & Tang, P. C. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080610 · Cited by 240
STARE-HI—Statement on reporting of evaluation studies in Health Informatics
TALMON, J., AMMENWERTH, E., BRENDER, J., DEKEIZER, N., NYKANEN, P., & RIGBY, M. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.09.002 · Cited by 234
Research works printed but not linked: 1
- LB Benko, 2003, Back to the drawing board., Mod Healthc, 33, 12
Cited by
A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126
Sheikh, A., Cornford, T., Barber, N., Avery, A., Takian, A., Lichtner, V., Petrakaki, D., Crowe, S., Marsden, K., Robertson, A., et al. (2011) · BMJ · doi:10.1136/bmj.d6054 · Cited by 314
Greenhalgh, T., Stramer, K., Bratan, T., Byrne, E., Russell, J., & Potts, H. W. W. (2010) · BMJ · doi:10.1136/bmj.c3111 · Cited by 275
“However, this standard is not accepted by some healthcare information systems researchers, who define robustness in terms of experimental or quasi-experimental studies of the “deployment” of a technology and its “impact” on predefined outcomes.”
BackgroundWhy Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles
Greenhalgh, T., & Russell, J. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000360 · Cited by 273
“has published three papers offering a “robust” and “scientific” approach to eHealth evaluation”
BackgroundWestbrook, J. I., Reckmann, M., Li, L., Runciman, W. B., Burke, R., Lo, C., Baysari, M. T., Braithwaite, J., & Day, R. O. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001164 · Cited by 216
Metadata
- Authors
- Lilford, R. J., Foster, J., & Pringle, M.
- Affiliations
- University of Birmingham (GB)
- University of Nottingham (GB)
- Corresponding author
- Lilford, Richard James
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2009-11-24
- Volume
- 6
- Issue
- 11
- Pages
- e1000186
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC2777393
- OpenAlex
- W2023814934
- Cited by
- 79
- References
- 32
- Citations by year
- 2026: 1 · 2025: 1 · 2023: 5 · 2022: 2 · 2021: 3 · 2020: 11 · 2019: 6 · 2018: 4 · 2017: 8 · 2016: 6 · 2015: 5 · 2014: 5 · 2013: 6 · 2012: 5
- FWCI
- 11.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- eHealth evaluation, implementation evaluation, method robustness, health information technology
- MeSH terms
- Evaluation Studies as Topic, United Kingdom, Humans, Medical Informatics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating eHealth: How to Make Evaluation More Methodologically Robust. https://app.cochat.ai/papers/10.1371/journal.pmed.1000186
- The paper
- Lilford, R. J., Foster, J., & Pringle, M. (2009). Evaluating eHealth: How to Make Evaluation More Methodologically Robust. PLoS Medicine. https://doi.org/10.1371/journal.pmed.1000186
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.