CoChat

Evaluating eHealth: How to Make Evaluation More Methodologically Robust

Open access

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; none of the 12 sources its claims cite is checked for support yet.

Abstract

In the third in a series of articles on evaluating eHealth, Richard Lilford and colleagues consider the evaluation of health IT systems as they are rolled out following preimplementation testing.

Overview

CoChat Summary

TL;DR

This essay argues that eHealth evaluations should be methodologically pluralistic, using both quantitative and qualitative methods, and should usually assess systems at organizational/workgroup level rather than only at the individual patient level. It also argues that external summative evaluation should be kept distinct from internal formative feedback, because mixing them can reduce the generalisability of findings.

Key findings

  • The authors state that evaluation of IT systems often requires a mixed-methods approach and that there is “no” consensus yet comparable to randomized trials for drugs.
  • They argue that the primary unit of analysis for many IT systems is likely the organizational/workgroup level (e.g., wards, hospitals, practices), because systems may not be restrictable to individuals and “contamination” can dilute effects.
  • They distinguish formative assessment as timely feedback to implementers and summative assessment as generalisable knowledge for decision makers beyond the life cycle of a particular application.
  • They recommend external assessments because they can be independent, use masked assessment of endpoints, include contemporaneous controls, and use before-and-after controlled designs to reduce bias.

Methods

  • Not stated in the paper.
  • The paper is a narrative essay/review drawing on the authors’ experience with National Programme for Information Technology (NPFIT) evaluation in England and on cited literature.
  • The main “analysis” is conceptual and argumentative: it synthesizes methodological principles for evaluating eHealth, including mixed methods, organizational-level observation, and separation of formative versus summative evaluation.

Limitations

  • The paper notes a tension between formative and summative evaluation: feedback used to improve implementation can alter summative results and reduce generalisability.
  • It states that distinctions between formative and summative assessment are “not watertight,” so these categories overlap in practice.
  • Evident from the text: this is a position essay rather than an empirical study, so it does not present original outcome data or a systematic test of the proposed methods.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pmed.1000186

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Figures

Figure 1 Causal chain showing levels where IT may impact. The potential impact of IT at different levels in a health care organisation. These boxes show endpoints that can be measured at different stages of the causal pathway. These endpoints include system effects (operational effects), effects on mediating variables, and endpoints at the patient level such as clinical errors and their sequelae.
Figure 1 Causal chain showing levels where IT may impact. The potential impact of IT at different levels in a health care organisation. These boxes show endpoints that can be measured at different stages of the causal pathway. These endpoints include system effects (operational effects), effects on mediating variables, and endpoints at the patient level such as clinical errors and their sequelae.CC BY · PubMed Central
Figure 2 Diagrammatic representation of development and deployment of IT systems. Diagrammatic representation of the development and deployment of IT systems to demonstrate the ideas of internal versus external assessment on the one hand and formative versus summative evaluation on the other.
Figure 2 Diagrammatic representation of development and deployment of IT systems. Diagrammatic representation of the development and deployment of IT systems to demonstrate the ideas of internal versus external assessment on the one hand and formative versus summative evaluation on the other.CC BY · PubMed Central

Verification

Claims the paper makes

0 Valid · 0 Not valid · 12 Not checked · 27 Own results and interpretations

10 more claims

Create a free account to see all 39 claims and to verify the 12 references still open.

See all claims — free account
  • Probably not a substantial claim. 19
  • It may state two separate claims as one. 8

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 32 references

Research works printed but not linked: 1

  1. LB Benko, 2003, Back to the drawing board., Mod Healthc, 33, 12

Cited by

See all 79

Metadata

Authors
Lilford, R. J., Foster, J., & Pringle, M.
Affiliations
  • University of Birmingham (GB)
  • University of Nottingham (GB)
Corresponding author
Lilford, Richard James
Venue
PLoS Medicine
Published
2009-11-24
Volume
6
Issue
11
Pages
e1000186
ISSN
  • 1549-1277 (ISSN-L)
  • 1549-1676
OpenAlex
W2023814934
Cited by
79
References
32
Citations by year
2026: 1 · 2025: 1 · 2023: 5 · 2022: 2 · 2021: 3 · 2020: 11 · 2019: 6 · 2018: 4 · 2017: 8 · 2016: 6 · 2015: 5 · 2014: 5 · 2013: 6 · 2012: 5
FWCI
11.28
Type
journal article
Language
en
Open access
Locations
  • PLoS Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Social Sciences › Public Administration › Evaluation and Performance Assessment
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
Keywords
eHealth evaluation, implementation evaluation, method robustness, health information technology
MeSH terms
Evaluation Studies as Topic, United Kingdom, Humans, Medical Informatics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluating eHealth: How to Make Evaluation More Methodologically Robust. https://app.cochat.ai/papers/10.1371/journal.pmed.1000186
The paper
Lilford, R. J., Foster, J., & Pringle, M. (2009). Evaluating eHealth: How to Make Evaluation More Methodologically Robust. PLoS Medicine. https://doi.org/10.1371/journal.pmed.1000186

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.