CoChat
Sign Up
Cited by 52Electronic Health Records SystemsInformation Systems Theories and ImplementationEvaluation and Performance Assessment

Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems

Open access
Takian, Amirhossein, Petrakaki, Dimitra, Cornford, Tony, Sheikh, Aziz, Barber, Nicholas·2012·BMC Health Services Research
DOI 10.1186/1472-6963-12-105PMC PMC3469374
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: A commitment to Electronic Health Record (EHR) systems now constitutes a core part of many governments' healthcare reform strategies. The resulting politically-initiated large-scale or national EHR endeavors are challenging because of their ambitious agendas of change, the scale of resources needed to make them work, the (relatively) short timescales set, and the large number of stakeholders involved, all of whom pursue somewhat different interests. These initiatives need to be evaluated to establish if they improve care and represent value for money. METHODS: Critical reflections on these complexities in the light of experience of undertaking the first national, longitudinal, and sociotechnical evaluation of the implementation and adoption of England's National Health Service's Care Records Service (NHS CRS). RESULTS/DISCUSSION: We advance two key arguments. First, national programs for EHR implementations are likely to take place in the shifting sands of evolving sociopolitical and sociotechnical and contexts, which are likely to shape them in significant ways. This poses challenges to conventional evaluation approaches which draw on a model of baseline operations → intervention → changed operations (outcome). Second, evaluation of such programs must account for this changing context by adapting to it. This requires careful and creative choice of ontological, epistemological and methodological assumptions. SUMMARY: New and significant challenges are faced in evaluating national EHR implementation endeavors. Based on experiences from this national evaluation of the implementation and adoption of the NHS CRS in England, we argue for an approach to these evaluations which moves away from seeing EHR systems as Information and Communication Technologies (ICT) projects requiring an essentially outcome-centred assessment towards a more interpretive approach that reflects the situated and evolving nature of EHR seen within multiple specific settings and reflecting a constantly changing milieu of policies, strategies and software, with constant interactions across such boundaries.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper argues that evaluations of national electronic health record (EHR) programs must adapt to shifting sociopolitical and sociotechnical contexts rather than rely on conventional before-after or outcome-centric designs. Drawing on a real-time evaluation of England’s NHS Care Records Service, it advocates interpretive, case-based, reflexive approaches that study EHR “in-the-making.”

Key findings

  • National EHR implementations are inherently political and dynamic, so the presence/absence of technology is a misleading basis for evaluation; the authors argue for studying “changing” as it occurs rather than only measuring achieved change.
  • The NHS CRS was a $19.6 (£12.7) billion program; implementation clusters were reduced from five to four (2005) and then to three (2006), with only two Local Service Providers active after 2008.
  • The authors’ initial before-during-after and stepped-wedge plans were adapted to linked case studies and an interpretive approach because local strategies, software configurations, and political changes made direct comparisons across sites, regions, or software difficult.
  • They conclude that non-use, partial adoption, and workarounds should not be seen only as failure but as different enactments of technology-in-use that may chart a path to successful national EHR service.

Methods

  • Critical reflections based on the first national, longitudinal, sociotechnical evaluation of the implementation and adoption of England’s NHS Care Records Service (NHS CRS).
  • The evaluation was commissioned as a 30-month (September 2008–March 2011) formative and summative real-time evaluation; initial plans proposed up to five “early adopter” sites in each of three implementation clusters, purposively including teaching, non-teaching, foundation, acute, mental health, and district hospitals.
  • The original design used a stepped-wedge/before-during-after logic and a sociotechnical framework combining Donabedian’s structure, process, and outcome concepts with system functions, human perspectives, and health care system dimensions; the team later shifted toward linked case-based studies, multiple narratives, and cross-site dialogue.

Limitations

  • The paper states that the 30-month timeframe could reflect only part of implementation and miss longer-term organizational changes; different software systems and releases provided different functionalities, rendering comparisons between sites, regions, or software difficult.
  • The authors state that their plans assumed clearly distinguishable absence/presence periods, but shifting contexts, distributed implementation ecology, and political changes meant the evaluation was inevitably partial—both incomplete and biased—and that they were often present where the action was not.
  • The paper notes that direct functionalities for assessing error, safety, and quality of care were mostly unimplemented in studied sites within the timeframe, so only availability of medical records in outpatient clinics was pursued in some sites; it also acknowledges that case-study findings are context-specific and shaped by researchers’ backgrounds and interpretive choices.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/1472-6963-12-105

Cites

87 of 87 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 42 cited sources checked · 0 supported · 0 flagged

  • Findingreasoning

    Implementation of NHS CRS was highly context-bound.

    “Clearly implementation was highly context-bound.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    In the NME cluster, implementation team members and clinical staff had to become involved in substantial ongoing activities of development, design and testing of the software in order to tailor the growing software to their needs.

    “In the NME cluster, in particular, implementation team members and clinical staff had to become involved in substantial ongoing activities of development, design and testing of the software in order to tailor the growing software to their (and by implication the wider NHS’s) needs.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    The introduction of NHS CRS into early adopter hospitals proved more challenging than initially anticipated.

    “The introduction of NHS CRS as an idea, let alone an operational software, into early adopter hospitals proved more challenging than initially anticipated.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    The NHS CRS came into being through the intervention of a number of institutions including DH, NHS CFH, LSPs, software providers, hospital and community Trusts, and other stakeholders.

    “The third challenge we needed to reflect was the distributed ecology of the NHS CRS as it came into being through the intervention of a number of institutions including the Department of Health (DH), NHS CFH (the agency managing the delivery), the LSPs, software providers, hospital and community Trusts, and many other professional, regulatory- and industry-based stakeholders.”
    Quote in the paperSubstantial claim
  • Findingown data

    In practice, the evaluation encountered many complex processes concerned not so much with 'using' these systems more or less as they were delivered, but of 'making' these systems as they were adapted and applied to the work at hand and the goals of the particular Trust.

    “Thus, in contrast to our initial assumption that comparisons could be made on the basis of absence and then presence of NHS CRS systems in the hospitals under investigation, in practice we encountered many complex processes concerned not so much with ‘using’ these systems more or less as they were delivered, but of ‘making’ these systems as they were adapted and applied to the work at hand and the”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The research plans assumed there would be clearly distinguishable periods defined by the absence and then presence of NHS CRS in hospitals, allowing before-during-after comparisons.
  • May combine two claims. 23
  • Classified as probably not a substantial claim. 32

  • May combine two claims.

    Findingreasoning

    Few NHS Trusts saw the national vision as the principal driving force for their own engagement; many found the national vision fundamentally unconvincing.

    “Few saw the national vision as the principal driving force for their own engagement; indeed many found the national vision fundamentally unconvincing.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    The political nature of the NHS CRS was understood by most stakeholders and was what they wanted to talk about most.

    “This political nature of the NHS CRS was understood by most stakeholders – often it was what they wanted to talk about most [40] – but this constituted a significant methodological challenge to account for such influences.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [40] Sheikh A, Cornford T, Barber N, Avery A, Takian A, Lichtner V, Petrakaki D, Crowe S, Marsden K, Robertson A, Morrison Z, Klecun E, Prescott R, Quinn C, Jani Y, Ficociello M, Voutsina K, Paton J, Fernando B, Jacklin A, Cresswell K: Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in "early adopter" hospitals. Br Med J. 2011, 343: d6054-10.1136/bmj.d6054. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    NHS CRS was a political project from the start, constituting an important part of the then New Labour government health agenda.

    “NHS CRS was a political project from the start, constituting an important part of the then [New Labour] government health agenda.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The software packages embodying the NHS CRS were neither self-contained nor did they tend to directly satisfy local requirements.

    “The software packages embodying the NHS CRS were neither self-contained nor did they tend to directly satisfy local requirements.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Delays in delivering systems, different levels of preparedness of early adopters, and sometimes opposing and conflicting views within NHS Trusts about what the NHS CRS was or should be resulted in hospitals developing a range of strategies and approaches to their own implementation of EHR systems.

    “Delays in delivering systems, different levels of preparedness of ‘early adopters’, and sometimes opposing and conflicting views within NHS Trusts about what the NHS CRS was or should be, resulted in hospitals developing a range of strategies and approaches to their own implementation of EHR systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Although the NHS CRS started from a single vision of a nationally shared EHR, this vision was translated into multiple local visions, leading to distinct local strategies for implementation and adoption (or non-adoption/dis-adoption).

    “Although the NHS CRS started from a single vision of a nationally shared EHR, this vision was translated into multiple local visions. This in turn led to the development of a number of distinct local strategies for its implementation and adoption (or in some cases non-adoption, or even dis-adoption).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The three key NHS CRS software systems studied - Millennium, RiO and Lorenzo - provided different mixes of functionalities in different releases and for different sites, rendering comparisons between sites, regions or software difficult.

    “Further, the three key NHS CRS software systems studied – Millennium, RiO and Lorenzo – would provide different mixes of functionalities in different releases and for different sites, rendering comparisons between sites, regions or software difficult.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    The NHS CRS was formed, translated and reproduced in various sites and settings and embodied different meanings for different people at different times and locations.

    “In this way we were able to understand how the NHS CRS was formed (i.e. how it was ‘per-formed’), translated and reproduced in various sites and settings [55] and the different meanings it embodied for different people, at different times and locations [56].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [55] Gherardi S: Practice? It’s a matter of taste?. Management Learning. 2009, 40: 535-10.1177/1350507609340812. source

      Not checked in the free run.

    • cites [56] Mol A, Law J: Regions. networks and fluids: anaemia and social topology. Soc Stud Sci. 1994, 24 (4): 641-671. 10.1177/030631279402400402. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Findingnone given

    The authors' approach worked; they provided formative feedback to a number of Trusts and to NHS CFH at the time it was most needed.

    “Our approach worked. We provided formative feedback to a number of Trusts and to NHS CFH at the time that it was most needed.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingcitation

    The dimensions described in the paper were to some degree outcomes of the value the researchers attributed to (or inscribed into) EHR systems, depending on each researcher's background, expertise and beliefs.

    “Indeed, the dimensions we have described here were to some degree outcomes of the value we (as researchers) attributed to (or inscribed into) EHR systems [67], depending on each researcher’s background, expertise and beliefs.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [67] Weber M: The Methodology Of The Social Sciences. 1949, First Edition, Free Press

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    EHR programs are inherently political and exist within a dynamic environment.

    “• EHR programs are inherently political and exist within a dynamic environment”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Any EHR program has a malleable character as it is shaped by contextual forces and reinterpreted by various interest groups and people.

    “First, we recognize the malleable character of any EHR program as it is shaped by contextual forces and is reinterpreted by various interest groups and people.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Evaluators need to draw upon alternative perspectives and understandings of technology and the possible role of information and data in changing work practices and organizational structures and hence potential to effect specific outcomes.

    “Second, we take from this the need for evaluators to draw upon alternative perspectives and understandings of technology and the possible role of information and data in changing work practices and organizational structures and hence potential to effect specific outcomes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Evaluation of national EHR programs must account for changing context by adapting to it.

    “Second, evaluation of such programs must account for this changing context by adapting to it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    EHR is understood by studying what people do, and in particular how, and to what extent, they 'work to make it work'.

    “· EHR is understood by studying what people do, and in particular how, and to what extent, they ‘work to make it work’”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    National programs for EHR implementations are likely to take place in evolving sociopolitical and sociotechnical contexts, which are likely to shape them in significant ways.

    “First, national programs for EHR implementations are likely to take place in the shifting sands of evolving sociopolitical and sociotechnical and contexts, which are likely to shape them in significant ways.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Conducting meaningful research in a dynamic environment requires methodological reflection and adaptation.

    “We conclude from this experience that conducting meaningful research in a dynamic environment requires methodological reflection and adaptation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The evaluation was inevitably partial (both incomplete and biased) because the evaluators could not capture all stages of NHS CRS implementation nor always be in the right place at the right time.

    “Rather, we had to accept that we were often present in places where the action was not [47] and thus that the evaluation was inevitably partial (both in the sense of being incomplete and of being biased).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [47] Law J: Organizing Modernity: Social Order and Social Theory. 1993, Wiley Blackwell

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Before-after evaluation designs may miss vital information about the change process, and assume that there is a clear definition of 'after'.

    “• Before-after evaluation designs may miss vital information about the change process, and assume that there is a clear definition of ‘after’”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    EHR is a sociotechnical intervention that is given meaning through the activities of its implementation and adoption; EHR is performed.

    “• EHR is a sociotechnical intervention that is given meaning through the activities of its implementation and adoption. EHR is performed”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Only limited national-scale implementations of EHR systems have taken place to date, and consequently there is little published evidence to inform larger scale evaluations.

    “This is unsurprising given that only limited national-scale implementations of EHR systems have taken place to date, and there is as a consequence little published evidence to inform larger scale evaluations [28].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [28] Lilford RJ, Foster J, Pringle M: Evaluating ehealth: how to make evaluation more methodologically robust. PLoS Med. 2009, 6 (11): e1000186-10.1371/journal.pmed.1000186. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Some of the authors' evaluation findings may not have particularly direct relevance for other programs and settings with their own challenges and distinct characteristics.

    “Some of our evaluation findings may not have a particularly direct relevance for other programs and settings with their own challenges and distinct characteristics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    The interpretive approach allows researchers to comprehend and reflect on the context and the means by which implementation and change occurs or not.

    “The interpretive approach in particular allows researchers to comprehend and reflect on the context and the means by which implementation and change occurs or not.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The authors were able to bring to the fore the intricate set of interlocking changes in practice that an EHR implies, a more formative view than the image of discrete even radical change that is the language of broad policy prescription and of technology vendors.

    “We were able to bring to the fore the intricate set of interlocking changes in practice that an EHR implies, a more formative view than the image of discrete even radical change that is the language of broad policy prescription and of technology vendors.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    There is no single, or standard way of best implementing national and large-scale EHR systems, and no predefined and prescriptive strategy to evaluate them.

    “For these reasons we suggest there is no single, or standard way of best implementing national and large-scale EHR systems, and so too there is no predefined and prescriptive strategy to evaluate them.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Evaluations that are focused on changing narrate the system change and tell the story of EHR implementation and adoption through multiple voices.

    “• Evaluations that are focused on changing narrate the system change and tell the story of EHR implementation and adoption through multiple voices.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    In this work the authors adopted the interpretive position that trades off notions of sampling, significance and generalizability in favor of depth of enquiry, contextualization, abstraction, dialogue and critical reflection.

    “Rather, in this work we adopted the interpretive position that trades off notions of sampling, significance and generalizability in favor of depth of enquiry, contextualization, abstraction, dialogue and critical reflection [30,60,70].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [30] Greenhalgh T, Russell J: Why do evaluation of ehealth programs fail? An alternative set of guiding principles. PLoS Med. 2010, 7 (11): e1000360-10.1371/journal.pmed.1000360. source

      Not checked in the free run.

    • cites [60] Klein K, Myers M: A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly. 1999, 23 (1): 67-94. 10.2307/249410. source

      Not checked in the free run.

    • cites [70] Wallerstein N: Power between evaluator and community: research relationships within New Mexico's healthier communities. Soc Sci Med. 1999, 49: 459-468. 10.1016/S0277-9536(99)00133-1. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    This constitutes a detailed stock of knowledge that can inform policy makers and managers at all levels, support learning within healthcare organizations, and provide transferable insights of relevance beyond the focus of the evaluation.

    “This constitutes a detailed stock of knowledge that can inform policy makers and managers at all levels, support learning within healthcare organizations, and provide transferable insights of relevance beyond the focus of our evaluation [72,73].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [72] Mays N, Pope C: Quality in qualitative health research. Qualitative research in health care. 1999, London: BMJ Publication Group, 1

      Not checked in the free run.

    • cites [73] Corbin J, Strauss A: Strategies for qualitative data analysis. Basics of Qualitative Research. Techniques and procedures for developing grounded theory. 2008, Thousand Oaks: Sage Publications Inc, 3 source

      Not checked in the free run.

    • Rests on an assumption: The authors' experience evaluating the NHS CRS is sufficiently similar to other national EHR evaluations for lessons to transfer.

  • May combine two claims.

    Premisecitation

    The infrequent use of RCTs in EHR evaluations has been due to multiple concerns including ethics, difficulty to control for effect mediators, costs and time-span, impracticalities of randomizing parts of a hospital system, and difficulties in measuring effects across diverse outcomes.

    “This has been due to multiple concerns including: the ethics of such approaches; difficulty to ‘control for’ potentially important effect mediators; associated costs and time-span [17,74]; impracticalities of randomizing parts of a hospital system; and difficulties in measuring the effects across a diverse array of outcomes [13,75,76].”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [17] Heathfielad HA, Pitty D, Hanka R: Evaluating information technology in healthcare: barriers and challenges. Br Med J. 1998, 316: 1959-1961. 10.1136/bmj.316.7149.1959. source

      Not checked in the free run.

    • cites [74] Stoop AP, Heathfield H, De Mul M, Berg M: Evaluation of Patient Care Information Systems: Theory and Practice. Health Information Management: Integrating Information and Communication Technology in Health Care Work. Edited by: Berg M. 2004, London: Routledge, 206-29.

      Not checked in the free run.

    • cites [13] Shepperd S, Lewin S, Straus S, Clarke M, Eccles MP, Fitzpatrick R, Wong G, Sheikh A: Can we systematically review studies that evaluate complex interventions?. PLoS Med PLoS Med. 2009, 6 (8): e1000086. source

      Not checked in the free run.

    • cites [75] Brender J: Handbook of evaluation methods for health informatics. 2006, London: Elsevier Academic Press

      Not checked in the free run.

    • cites [76] Kaplan B, Shaw NT: People, organisational, and social issues: evaluation as an exemplar. Yearbook of Medical Informatics. Edited by: Haux R, Kulikowski C. 2002, Stuttgart: Shattauer, 71-88.

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The authors' moves from comparative before-after methodology towards case study methodology and interpretive approach changed the initial plans and limited the evaluation's scope.

    “Our moves from a comparative before-after methodology towards a case study methodology and an interpretive approach certainly changed the initial plans. We recognize too that these moves also limited the evaluation’s scope.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    Any evaluations of large-scale or national EHR initiatives are likely to take place within changing political, economic and technical contexts, which will shape both the EHR and the evaluation in significant ways.

    “Any evaluations of large-scale or national EHR initiatives are likely to take place within changing political, economic and technical contexts, which will shape both the EHR and the evaluation in significant ways.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    It is fundamental to understanding health IT that periods of change and changing should be studied carefully.

    “Beyond this, we see it as fundamental to understanding health IT that we should study carefully periods of change and changing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    There are tensions among significant stakeholders as they balance parallel assumptions and expectations about technology and about their work.

    “Rather, it is understood, there are tensions among significant stakeholders as they balance parallel assumptions and expectations about technology and about their work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Greater insights could be gained from approaches that sought to 'tell the whole story' not just the ending.

    “Rather we saw that greater insights could be gained from approaches that sought to ‘tell the whole story’ not just the ending [59].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [59] Greenhalgh T, Stones R: Theorising big IT programmes in healthcare: strong structuration theory meets actor network theory. Soc Sci Med. 2010, 70: 1285-1294. 10.1016/j.socscimed.2009.12.034. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    In-situ and real-time evaluations benefit from an interpretive perspective focused on process if they are to inform policy at the time that it is most needed.

    “But we do argue that in-situ and real-time evaluations benefit from an interpretive perspective focused on process if they are to inform policy at the time that it most needed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisereasoning

    Policy makers, implementers and other stakeholders are not naïve and seldom perceive implementation as just a linear process that starts with finding the 'right' technology and putting it in place so that intended users use it as planned with expected outcomes achieved.

    “Policy makers, implementers and other stakeholders are not naïve and seldom perceive implementation as just a linear process that starts off with finding the ‘right’ technology and putting it in place so that the intended users use it as planned with the expected outcomes being achieved.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The authors' sociotechnical lens focused on how things were 'made-to-work' rather than on how well or not the EHR systems functioned.

    “Our sociotechnical lens was in particular focused on the specific question of how things were ‘made-to-work’ rather than on how well or not the EHR systems functioned [51].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [51] Petrakaki D, Cornford T, Klecun E: Sociotechnical Changing in Healthcare. Information technology in health care: socio-technical approaches - from safe systems to patient safety. Edited by: Aarts J, Nohr C, Aarts J, Aarhus NC. 2010, Ios Press, ISBN [ISBN: 978-1-60750-568-6]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Evaluations in this style can speak of outcomes in terms of concepts such as changed expectations, recognition of benefits, assessments of risk, perceptions of consequence, processes of learning.

    “• Evaluations in this style can speak of outcomes in terms of concepts such as changed expectations, recognition of benefits, assessments of risk, perceptions of consequence, processes of learning”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Such findings can be of real use for health services managers and policy makers, and help address pro-innovation bias and techno-centric dreams; they can also facilitate organizational learning.

    “• Such findings can be of real use for health services managers and policy makers, and help address pro-innovation bias and techno-centric dreams. It can also facilitate organizational learning”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    National EHR implementations engage with multiple stakeholders who have different understandings of EHR and of its potential purposes and different strategies for accommodating it (or not).

    “They engage with multiple stakeholders who have different understandings of EHR and of its potential purposes and different strategies for accommodating it (or not).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Encouraging research respondents to narrate their own stories drawing from past experiences, present understandings and future projections builds a rich picture of the many processes at work, not all of which align directly with the innovation or technology in question.

    “Encouraging research respondents to narrate their own stories drawing from their past experiences, present understandings and future projections, builds a rich picture of the many processes that are at work, not all of which align directly with the innovation or technology in question.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Case study-based evaluations can probe deeper and can address EHR systems within dynamic or distinctive socio-cultural environments.

    “• Case study-based evaluations can probe deeper and can address EHR systems within dynamic or distinctive socio-cultural environments”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    This type of evaluation will inevitably be contingent.

    “This type of evaluation will inevitably be contingent and hence we emphasize the need for reflexivity among researchers and adaptation of perspectives as have other authors writing on similar studies [66].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [66] Greenhalgh T, Russell J, Ashcroft RE, Parsons W: Why national ehealth programs need dead philosophers: wittgensteinian reflections on policymakers’ reluctance to learn from history. Milbank Q. 2011, 89 (4): 533-563. 10.1111/j.1468-0009.2011.00642.x. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    National EHR programs cannot be dissociated from the national social and political context, nor from the local contexts where they engage with managerial and clinical practice and changing occurs.

    “Just as national EHR programs cannot be dissociated from the national social and political context, neither can they be disassociated from the local contexts, the place where they come engaged with managerial and clinical practice and changing occurs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationcitation

    The evaluation is the result of processes of reduction, simplification and interpretation and is fundamentally subjective and a 'partial truth'.

    “This evaluation is thus like any other the result of processes of reduction, simplification and interpretation [47] and fundamentally subjective and a ‘partial truth’ [68].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [47] Law J: Organizing Modernity: Social Order and Social Theory. 1993, Wiley Blackwell

      Not checked in the free run.

    • cites [68] Seale PC: Researching Society and Culture. 2004, Sage Publications: Second Edition

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Multiple coordinated case studies allow an insightful cross-site dialogue that can reveal common themes and distinct experiences.

    “• Multiple coordinated case studies allow an insightful cross-site dialogue that can reveal common themes and distinct experiences”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Using the presence or absence of EHR technology as the fundamental basis of evaluation research design (before-after) is misleading and inadequate.

    “This suggests that using the presence or absence of EHR technology as the fundamental basis of evaluation research design (before-after) is misleading and inadequate.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    The use of a socio-technical language focused on practice can help see technology-led programs of change as contingent and malleable - something in the making or being built - rather than something essential and fixed with defined and time-invariant causal properties.

    “In presenting analysis and results, the use of a socio-technical language focused on practice can help us to see technology-led programs of change as being contingent and malleable – something in the making or being built – rather than something essential and fixed with defined and time-invariant causal properties [78].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [78] Rabinow P: Anthropos today: reflections on modern equipment. 2003, Princeton: Princeton University Press

      Not checked in the free run.

48 of 48

  • The authors state this as an assumption.

    The research plans assumed there would be clearly distinguishable periods defined by the absence and then presence of NHS CRS in hospitals, allowing before-during-after comparisons.

    “our research plans were predicated on the belief that there would be clearly distinguishable periods defined by the absence and then presence of NHS CRS in hospitals. This would allow us to make comparisons based on before (absence), during (implementation) and after (presence).”
    “in contrast to our initial assumption that comparisons could be made on the basis of absence and then presence of NHS CRS systems in the hospitals under investigation”

  • The authors state this as an assumption.

    The authors assumed from the outset that the implementation of NHS CRS would lead to diverse outcomes reflecting distinct contexts.

    “We assumed from the outset that this would lead to diverse outcomes that reflected these distinct contexts.”

  • The paper relies on this without support.

    The authors' experience evaluating the NHS CRS is sufficiently similar to other national EHR evaluations for lessons to transfer.

    “Building the house: implications of our experience for other national EHR evaluations”
    “This constitutes a detailed stock of knowledge that can inform policy makers and managers at all levels, support learning within healthcare organizations, and provide transferable insights of relevance beyond the focus of our evaluation [72,73].”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • That prior study demonstrated the importance of considering the sociotechnical context of EHR implementation, but the UK investigators did not apply a formal framework to assess patient safety until now.

    “That study demonstrated the importance of considering the sociotechnical context of EHR implementation, although the UK investigators did not apply a formal framework to assess patient safety until now. 30”

    From Exploring the sociotechnical intersection of patient safety and electronic health record implementation (2014) · cites this paper as [30]

Cites

Flagged first, then by citations

3 of the 90 references couldn’t be matched to a known paper.

Research works printed but not linked: 6

  1. Institute of Medicine: Health and human sciences in the 21st century: charting a new course for a healthier America. 2009, New York: National Academies Press
  2. Shekelle PG, Goldzweig CL: Costs and benefits of health information technology: an updated systematic review. 2009, RAND corporation, London: Health foundation for Southern California evidence-based practice center
  3. Brennan S: THE NHS IT PROJECT the biggest computer programme in the world..ever!. Radcliffe. 2005
  4. National Audit Office: The National Programme for IT in the NHS: Progress since 2006. 2008, [ www.nao.org.uk/publications/0708/the_national_programme_for_it.aspx ]
  5. Pawson R, Tilley N: Realistic Evaluation. 2004, London: SAGE Publications
  6. Law J: Organizing Modernity: Social Order and Social Theory. 1993, Wiley Blackwell

Cited by

Sorted by citations

Related

Authors
Takian, A., Petrakaki, D., Cornford, T., Sheikh, A., & Barber, N.
Affiliations
  • Brunel University of London (GB)
  • University of Sussex (GB)
  • London School of Economics and Political Science (GB)
  • University of Edinburgh (GB)
  • UCL School of Pharmacy (GB)
Corresponding author
Takian, Amirhossein
ORCID
Venue
BMC Health Services Research
Publisher
Springer Science and Business Media LLC
Published
2012-04-30
Volume
12
Issue
1
Pages
105-105
ISSN
1472-6963 (ISSN-L)
DOI
10.1186/1472-6963-12-105
PMC
PMC3469374
OpenAlex
W2104058658
Cited by
51
References
90
Citations by year
2025: 2 · 2024: 3 · 2023: 1 · 2020: 3 · 2019: 3 · 2018: 5 · 2017: 1 · 2016: 7 · 2015: 4 · 2014: 12 · 2013: 7 · 2012: 3
FWCI
13.48
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Health Services Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Durham Research Online (Durham University) (acceptedVersion, other-oa)
  • Durham Research Online (Durham University) (submittedVersion, cc-by)
  • Brunel University Research Archive (BURA) (Brunel University London) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Funders
  • London School of Economics and Political Science
  • National Institute for Health and Care Research
  • University of Nottingham
  • University of Edinburgh
  • School of Pharmacy, University College London
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
Keywords
electronic health records, national health information systems, National Health Service, implementation evaluation, technology adoption, evaluation methodology, context sensitivity, health care reform, interpretive evaluation, health policy
MeSH terms
Computer Communication Networks, Diffusion of Innovation, England, Health Plan Implementation, Health Policy, Humans, Interinstitutional Relations, National Health Programs, Social Change, Program Evaluation, Models, Organizational, Delivery of Health Care, Integrated, Capacity Building, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review