Implementing online consultations in primary care: a mixed-method evaluation extending normalisation process theory through service co-production
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Objectives: To examine patient and staff views, experiences and acceptability of a UK primary care online consultation system and ask how the system and its implementation may be improved. Design: Mixed-method evaluation of a primary care e-consultation system. Setting: Primary care practices in South West England. Methods: Qualitative interviews with 23 practice staff in six practices. Patient survey data for 756 e-consultations from 36 practices, with free-text survey comments from 512 patients, were analysed thematically. Anonymised patients’ records were abstracted for 485 e-consultations from eight practices, including consultation types and outcomes. Descriptive statistics were used to analyse quantitative data. Analysis of implementation and the usage of the e-consultation system were informed by: (1) normalisation process theory, (2) a framework that illustrates how e-consultations were co-produced and (3) patients’ and staff touchpoints. Results: We found different expectations between patients and staff on how to use e-consultations ‘appropriately’. While some patients used the system to try and save time for themselves and their general practitioners (GPs), some used e-consultations when they could not get a timely face-to-face appointment. Most e-consultations resulted in either follow-on phone (32%) or face-to-face appointments (38%) and GPs felt that this duplicated their workload. Patient satisfaction of the system was high, but a minority were dissatisfied with practice communication about their e-consultation. Conclusions: Where both patients and staff interact with technology, it is in effect ‘co-implemented’. How patients used e-consultations impacted on practice staff’s experiences and appraisal of the system. Overall, the e-consultation system studied could improve access for some patients, but in its current form, it was not perceived by practices as creating sufficient efficiencies to warrant financial investment. We illustrate how this e-consultation system and its implementation can be improved, through mapping the co-production of e-consultations through touchpoints.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies reported low asynchronous telemedicine use before the pandemic; up to seven consultations per 1000 patients per month. 31–33,35”
From Effectiveness and safety of asynchronous telemedicine consultations in general practice: a systematic review (2024) · cites this paper as [33]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Many were telephone or face-to-face, (55.3–74%), which were more likely when patients had new or complex problems, 31–33 or required physical examination. 34,35”
From Effectiveness and safety of asynchronous telemedicine consultations in general practice: a systematic review (2024) · cites this paper as [33]
The cited source says what the claim attributes to it.
In the citing paper
“Concerns included digitally excluded patients and reinforcing health inequities. 31–33,43,48”
From Effectiveness and safety of asynchronous telemedicine consultations in general practice: a systematic review (2024) · cites this paper as [33]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Three studies reported patient use of asynchronous telemedicine increased gradually before the pandemic 26,31–33,40 and in 2016 20% of UK general practices planned to introduce it. 49”
From Effectiveness and safety of asynchronous telemedicine consultations in general practice: a systematic review (2024) · cites this paper as [33]
Experimental — accuracy not yet measuredIn the citing paper
“One quantitative survey followed up 756 e-consultations in general practice and found that most generated either a telephone (32%) or face-to-face (38%) consultation (108)”
From Planning and Evaluating Remote Consultation Services: A New Conceptual Framework Incorporating Complexity and Practical Ethics (2021) · cites this paper as [108]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
A conceptual model of service quality and its implications for future research
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1985) · Journal of Marketing · doi:10.1177/002224298504900403 · Cited by 21,645
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887
Implementing, Embedding, and Integrating Practices: An Outline of Normalization Process Theory
May, C., & Finch, T. (2009) · Sociology · doi:10.1177/0038038509103208 · Cited by 2,115
Murray, E., Treweek, S., Pope, C., MacFarlane, A., Ballini, L., Dowrick, C., Finch, T., Kennedy, A., Mair, F., O'Donnell, C., et al. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-63 · Cited by 1,478
Service Blueprinting: A Practical Technique for Service Innovation
Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008) · California Management Review · doi:10.2307/41166446 · Cited by 1,194
Co-Production and the Co-Creation of Value in Public Services: A suitable case for treatment?
Osborne, S. P., Radnor, Z., & Strokosch, K. (2016) · Public Management Review · doi:10.1080/14719037.2015.1111927 · Cited by 1,126
Coproduction of healthcare service
Batalden, M., Batalden, P., Margolis, P., Seid, M., Armstrong, G., Opipari-Arrigan, L., & Hartung, H. (2016) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004315 · Cited by 1,078
Ross, J., Stevenson, F., Lau, R., & Murray, E. (2016) · Implementation Science · doi:10.1186/s13012-016-0510-7 · Cited by 1,011 · cited 3×
“Our results align with other studies that highlight potential barriers to technological implementation including that: the clinical data the system was designed to generate from patients were sometimes incomplete(46); the system was not fully interoperable with other IT systems, and costs prohibited long-term usage.(1…”
BackgroundResultInfluentialA New Theory for Public Service Management? Toward a (Public) Service-Dominant Approach
Osborne, S. P., Radnor, Z., & Nasi, G. (2013) · The American Review of Public Administration · doi:10.1177/0275074012466935 · Cited by 848
Individual determinants of research utilization by nurses: a systematic review update
Squires, J. E., Estabrooks, C. A., Gustavsson, P., & Wallin, L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-1 · Cited by 798
Triangulation and integration: processes, claims and implications
Moran-Ellis, J., Alexander, V. D., Cronin, A., Dickinson, M., Fielding, J., Sleney, J., & Thomas, H. (2006) · Qualitative Research · doi:10.1177/1468794106058870 · Cited by 771
Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices
Prasad, V., & Ioannidis, J. P. (2014) · Implementation Science · doi:10.1186/1748-5908-9-1 · Cited by 723
Brandsen, T., & Honingh, M. (2016) · Public Administration Review · doi:10.1111/puar.12465 · Cited by 625 · cited 1×
“This article examines co-production 'in the implementation of core services’ where ‘citizens are actively engaged in the implementation, but not the design, of an individual serv ice’ (Brandsen et al, p433).(22) We develop NPT to analyse patients’ implementation roles, using service co-production theory to understand…”
MethodHobbs, F. D. R., Bankhead, C., Mukhtar, T., Stevens, S., Perera-Salazar, R., Holt, T., & Salisbury, C. (2016) · The Lancet · doi:10.1016/s0140-6736(16)00620-6 · Cited by 601
Self-citationMair, F. S., May, C., O’Donnell, C., Finch, T., Sullivan, F., & Murray, E. (2012) · Bulletin of the World Health Organization · doi:10.2471/blt.11.099424 · Cited by 553 · cited 1×
“specifically examine how e-health services affect clinical interactions with patients.(43) 45 It shows how e-health”
BackgroundOsborne, S. P., & Strokosch, K. (2013) · British Journal of Management · doi:10.1111/1467-8551.12010 · Cited by 534
Purtle, J., Peters, R., & Brownson, R. C. (2015) · Implementation Science · doi:10.1186/s13012-015-0367-1 · Cited by 446
Rajski, M., Zanetti-Dällenbach, R., Vogel, B., Herrmann, R., Rochlitz, C., & Buess, M. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-1 · Cited by 355
Why is it difficult to implement e-health initiatives? A qualitative study
Murray, E., Burns, J., May, C., Finch, T., O'Donnell, C., Wallace, P., & Mair, F. (2011) · Implementation Science · doi:10.1186/1748-5908-6-6 · Cited by 270 · cited 1×
“of adverse effects on workload(43) and poor interactional workability of technology which can impede adoption within primary care.(44) e-Consultations supported efficiencies for straightforward GP queries, but less so complex ones, showing that how patients use technology can affect its implementation.”
Background(2007) · International Journal of Health Care Quality Assurance · doi:10.1108/ijhcqa.2007.06220eae.001 · Cited by 267
Co-Production, Interdependence and Publicness: Extending public service-dominant logic
Alford, J. (2016) · Public Management Review · doi:10.1080/14719037.2015.1111659 · Cited by 250 · cited 1×
“29 33 Understanding this process helps to analyse service users’ roles as a co-producer of a service.(26) 34 NPT constructs(14) 16 and service co-production processes(23) can be integrated together and used to analyse staff and patients’ initial expectations, interactions with and experiences of e-consultations, and t…”
BackgroundService co-production and value co-creation: The case for a service-oriented architecture (SOA)
Ordanini, A., & Pasini, P. (2008) · European Management Journal · doi:10.1016/j.emj.2008.04.005 · Cited by 237
Operationalizing Co-Production in Public Services Delivery: The contribution of service blueprinting
Radnor, Z., Osborne, S. P., Kinder, T., & Mutton, J. (2014) · Public Management Review · doi:10.1080/14719037.2013.848923 · Cited by 236
Integrating telecare for chronic disease management in the community: What needs to be done?
May, C. R., Finch, T. L., Cornford, J., Exley, C., Gately, C., Kirk, S., Jenkings, K. N., Osbourne, J., Robinson, A. L., Rogers, A., et al. (2011) · BMC Health Services Research · doi:10.1186/1472-6963-11-131 · Cited by 220 · cited 1×
“e-Consultations supported efficiencies for straightforward GP queries, but less so complex ones, showing that how patients use technology can affect its implementation.(45) Our results align with other studies that highlight potential barriers to technological implementation including that: the clinical data the syste…”
ResultThe organising vision for telehealth and telecare: discourse analysis
Greenhalgh, T., Procter, R., Wherton, J., Sugarhood, P., & Shaw, S. (2012) · BMJ Open · doi:10.1136/bmjopen-2012-001574 · Cited by 199
Wherton, J., Sugarhood, P., Procter, R., Hinder, S., & Greenhalgh, T. (2015) · Implementation Science · doi:10.1186/s13012-015-0271-8 · Cited by 135
Brant, H., Atherton, H., Ziebland, S., McKinstry, B., Campbell, J. L., & Salisbury, C. (2016) · British Journal of General Practice · doi:10.3399/bjgp16x685597 · Cited by 120
Self-citationEdwards, H. B., Marques, E., Hollingworth, W., Horwood, J., Farr, M., Bernard, E., Salisbury, C., & Northstone, K. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-016901 · Cited by 110
Self-citationUse of an electronic consultation system in primary care: a qualitative interview study
Banks, J., Farr, M., Salisbury, C., Bernard, E., Northstone, K., Edwards, H., & Horwood, J. (2018) · British Journal of General Practice · doi:10.3399/bjgp17x693509 · Cited by 108
Self-citationConnell, L. A., McMahon, N. E., Harris, J. E., Watkins, C. L., & Eng, J. J. (2014) · Implementation Science · doi:10.1186/s13012-014-0090-3 · Cited by 99
Sopcak, N., Aguilar, C., O’Brien, M. A., Nykiforuk, C., Aubrey-Bassler, K., Cullen, R., Grunfeld, E., & Manca, D. P. (2016) · Implementation Science · doi:10.1186/s13012-016-0525-0 · Cited by 90
Clarke, D. J., Godfrey, M., Hawkins, R., Sadler, E., Harding, G., Forster, A., McKevitt, C., Dickerson, J., & Farrin, A. (2013) · Implementation Science · doi:10.1186/1748-5908-8-96 · Cited by 88
Boaz, A., Robert, G., Locock, L., Sturmey, G., Gager, M., Vougioukalou, S., Ziebland, S., & Fielden, J. (2016) · Journal of Health Organization and Management · doi:10.1108/jhom-02-2015-0027 · Cited by 81
Hanna, L., May, C., & Fairhurst, K. (2012) · Family Practice · doi:10.1093/fampra/cmr087 · Cited by 63
Kennedy, A., Vassilev, I., James, E., & Rogers, A. (2015) · Implementation Science · doi:10.1186/s13012-016-0384-8 · Cited by 63
Atherton, H., & Ziebland, S. (2016) · DIGITAL HEALTH · doi:10.1177/2055207616675559 · Cited by 45
Holt, T. A., Fletcher, E., Warren, F., Richards, S., Salisbury, C., Calitri, R., Green, C., Taylor, R., Richards, D. A., Varley, A., et al. (2016) · British Journal of General Practice · doi:10.3399/bjgp16x684001 · Cited by 42 · cited 1×
“Other interventions designed to improve efficiency and access in primary care highlight potential workload issues; for example, nurse-led telephone triage may reduce GP contact time, but increase overall clinician contact time.(42) Previous e-health studies that use NPT highlight barriers Table 3 Nature of e-consultat…”
BackgroundSelf-citationVideo Consultation Use by Australian General Practitioners: Video Vignette Study
Jiwa, M., & Meng, X. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2638 · Cited by 37 · cited 1×
“For more efficient implementation of e-consultations, further consideration may be needed of when it is appropriate to use technology, for example, for less complex tasks,(9) and when face-toface interaction is essential, such as in the diagnosis of complex symptoms.(41) Other interventions designed to improve efficie…”
MethodHanna, L., May, C., & Fairhurst, K. (2011) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v19i1.789 · Cited by 25
Gibson, J., Lightbody, E., McLoughlin, A., McAdam, J., Gibson, A., Day, E., Fitzgerald, J., May, C., Price, C., Emsley, H., et al. (2016) · Health Expectations · doi:10.1111/hex.12333 · Cited by 25
Electronic consultations offer few benefits for GP practices, says study
Wise, J. (2017) · BMJ · doi:10.1136/bmj.j5141 · Cited by 7
Cited by
Murphy, M., Scott, L. J., Salisbury, C., Turner, A., Scott, A., Denholm, R., Lewis, R., Iyer, G., Macleod, J., & Horwood, J. (2021) · British Journal of General Practice · doi:10.3399/bjgp.2020.0948 · Cited by 486
Self-citationErku, D., Khatri, R., Endalamaw, A., Wolka, E., Nigatu, F., Zewdie, A., & Assefa, Y. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20196854 · Cited by 177
“Furthermore, e-consultations did not substitute for, but were, rather, an ‘add on’ to, face-to-face consultations, further duplicating their workload [59].”
Greenhalgh, T., Rosen, R., Shaw, S. E., Byng, R., Faulkner, S., Finlay, T., Grundy, E., Husain, L., Hughes, G., Leone, C., et al. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.726095 · Cited by 173
“One quantitative survey followed up 756 e-consultations in general practice and found that most generated either a telephone (32%) or face-to-face (38%) consultation (108).”
BackgroundHuddlestone, L., Turner, J., Eborall, H., Hudson, N., Davies, M. J., & Martin, G. P. (2020) · BMC Family Practice · doi:10.1186/s12875-020-01107-y · Cited by 170
Smith, H. J. J., Budworth, L., Grindey, C., Hague, I., Hamer, N., Kislov, R., van der Graaf, P., & Langley, J. (2022) · Health Research Policy and Systems · doi:10.1186/s12961-022-00838-x · Cited by 169
Metadata
- Authors
- Farr, M., Banks, J., Edwards, H. B., Northstone, K., Bernard, E., Salisbury, C., & Horwood, J.
- Affiliations
- University Hospitals Bristol NHS Foundation Trust (GB)
- Corresponding author
- Farr, Michelle
- Venue
- BMJ Open
- Publisher
- BMJ
- Published
- 2018-03
- Volume
- 8
- Issue
- 3
- Pages
- e019966
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC5875620
- OpenAlex
- W2792425939
- Cited by
- 106
- References
- 42
- Citations by year
- 2026: 4 · 2025: 13 · 2024: 12 · 2023: 23 · 2022: 13 · 2021: 18 · 2020: 15 · 2019: 7 · 2018: 1
- FWCI
- 9.76
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMJ Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- Funders
- National Institute for Health Research
- One Care Consortium Ltd
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Healthcare Systems and Technology
- Keywords
- teleconsultation, e-consultation, primary care, Normalisation Process Theory, service co-production, patient and staff experiences, acceptability, general practitioner workload, patient satisfaction, health care access, implementation evaluation, touchpoints
- MeSH terms
- Adult, Appointments and Schedules, Communication, England, Female, Humans, Interviews as Topic, Male, Online Systems, Primary Health Care, Surveys and Questionnaires, Referral and Consultation, Workload, Patient Satisfaction, Process Assessment, Health Care, Telemedicine, Qualitative Research, General Practitioners
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.