Evaluating diverse electronic consultation programs with a common framework
Open accessAbstract
BACKGROUND: Electronic consultation is an emerging mode of specialty care delivery that allows primary care providers and their patients to obtain specialist expertise without an in-person visit. While studies of individual programs have demonstrated benefits related to timely access to specialty care, electronic consultation programs have not achieved widespread use in the United States. The lack of common evaluation metrics across health systems and concerns related to the generalizability of existing evaluation efforts may be hampering further growth. We sought to identify gaps in knowledge related to the implementation of electronic consultation programs and develop a set of shared evaluation measures to promote further diffusion. METHODS: Using a case study approach, we apply the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) and the Quadruple Aim frameworks of evaluation to examine electronic consultation implementation across diverse delivery systems. Data are from 4 early adopter healthcare delivery systems (San Francisco Health Network, Mayo Clinic, Veterans Administration, Champlain Local Health Integration Network) that represent varied organizational structures, care for different patient populations, and have well-established multi-specialty electronic consultation programs. Data sources include published and unpublished quantitative data from each electronic consultation database and qualitative data from systems' end-users. RESULTS: Organizational drivers of electronic consultation implementation were similar across the systems (challenges with timely and/or efficient access to specialty care), though unique system-level facilitators and barriers influenced reach, adoption and design. Effectiveness of implementation was consistent, with improved patient access to timely, perceived high-quality specialty expertise with few negative consequences, garnering high satisfaction among end-users. Data about patient-specific clinical outcomes are lacking, as are policies that provide guidance on the legal implications of electronic consultation and ideal remuneration strategies. CONCLUSION: A core set of effectiveness and implementation metrics rooted in the Quadruple Aim may promote data-driven improvements and further diffusion of successful electronic consultation programs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Widespread dissemination of eConsult-like programmes will occur only when their positive impact is obvious and the quality of care delivery is acceptable to all populations, including vulnerable ones.
“These objectives are in line with previous research suggesting that a widespread dissemination of eConsult-like programmes will occur only when their positive impact is obvious and the quality of care delivery is acceptable to all populations, including the vulnerable ones [31].”
From Key factors for national spread and scale-up of an eConsult innovation (2020) · cites this paper as [31]
eConsult's successful spread and scale-up will depend on the collection and publication of data to comprehensively demonstrate its value in terms of reach, effectiveness, and adoption, and should include the consumer/patient perspective.
“The authors further propose that eConsult’s successful spread and scale-up will depend on the collection and publication of data to comprehensively demonstrate its value in terms of reach, effectiveness, and adoption and should include the consumer/patient perspective.”
From Key factors for national spread and scale-up of an eConsult innovation (2020) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
From Triple to Quadruple Aim: Care of the Patient Requires Care of the Provider
Bodenheimer, T., & Sinsky, C. (2014) · The Annals of Family Medicine · doi:10.1370/afm.1713 · Cited by 3,462 · see citing works
Glasgow, R. E. (2006) · Health Education Research · doi:10.1093/her/cyl081 · Cited by 565 · see citing works
Disruptive Innovation In Health Care Delivery: A Framework For Business-Model Innovation
Hwang, J., & Christensen, C. M. (2008) · Health Affairs · doi:10.1377/hlthaff.27.5.1329 · Cited by 500 · see citing works
Vimalananda, V. G., Gupte, G., Seraj, S. M., Orlander, J., Berlowitz, D., Fincke, B. G., & Simon, S. R. (2015) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15582108 · Cited by 437 · see citing works
Self-citationLiddy, C., Drosinis, P., & Keely, E. (2016) · Family Practice · doi:10.1093/fampra/cmw024 · Cited by 195 · see citing works
Self-citationeReferral — A New Model for Integrated Care
Chen, A. H., Murphy, E. J., & Yee, H. F. (2013) · New England Journal of Medicine · doi:10.1056/nejmp1215594 · Cited by 173 · see citing works
Self-citationKim, Y., Chen, A. H., Keith, E., Yee, H. F., & Kushel, M. (2009) · Journal of General Internal Medicine · doi:10.1007/s11606-009-0955-3 · Cited by 150 · see citing works
Self-citationLiddy, C., Afkham, A., Drosinis, P., Joschko, J., & Keely, E. (2015) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2015.03.140255 · Cited by 140 · see citing works
Self-citationOlayiwola, J. N., Anderson, D., Jepeal, N., Aseltine, R., Pickett, C., Yan, J., & Zlateva, I. (2016) · The Annals of Family Medicine · doi:10.1370/afm.1869 · Cited by 117 · see citing works
Barnett, M. L., Yee, H. F., Mehrotra, A., & Giboney, P. (2017) · Health Affairs · doi:10.1377/hlthaff.2016.1283 · Cited by 115 · see citing works
A Safety-Net System Gains Efficiencies Through ‘eReferrals’ To Specialists
Chen, A. H., Kushel, M. B., Grumbach, K., & Yee, H. F. (2010) · Health Affairs · doi:10.1377/hlthaff.2010.0027 · Cited by 98 · see citing works
Self-citationTuot, D. S., Leeds, K., Murphy, E. J., Sarkar, U., Lyles, C. R., Mekonnen, T., & Chen, A. H. (2015) · BMC Health Services Research · doi:10.1186/s12913-015-1233-1 · Cited by 98 · see citing works
Self-citationWootton, R., Geissbuhler, A., Jethwani, K., Kovarik, C., Person, D., Vladzymyrskyy, A., Zanaboni, P., & Zolfo, M. (2012) · Bulletin of the World Health Organization · doi:10.2471/blt.11.099143 · Cited by 74 · see citing works
Implementation of an electronic referral system for outpatient specialty care.
Straus, S. G., Chen, A. H., Yee, H. F., Kushel, M., & Bell, D. S. (2011) · PubMed Central · Cited by 73 · see citing works
Self-citationGupte, G., Vimalananda, V., Simon, S. R., DeVito, K., Clark, J., & Orlander, J. D. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.4801 · Cited by 63 · see citing works
Self-citationVeteran, Primary Care Provider, and Specialist Satisfaction With Electronic Consultation
Rodriguez, K. L., Burkitt, K. H., Bayliss, N. K., Skoko, J. E., Switzer, G. E., Zickmund, S. L., Fine, M. J., & Macpherson, D. S. (2015) · JMIR Medical Informatics · doi:10.2196/medinform.3725 · Cited by 63 · see citing works
Follow-up Actions on Electronic Referral Communication in a Multispecialty Outpatient Setting
Singh, H., Esquivel, A., Sittig, D. F., Murphy, D., Kadiyala, H., Schiesser, R., Espadas, D., & Petersen, L. A. (2011) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1501-z · Cited by 57 · see citing works
Emerging Trends Could Exacerbate Health Inequities In The United States
Arcaya, M. C., & Figueroa, J. F. (2017) · Health Affairs · doi:10.1377/hlthaff.2017.0011 · Cited by 57 · see citing works
Electronic consultation system demonstrates educational benefit for primary care providers
Kwok, J., Olayiwola, J. N., Knox, M. J., Murphy, E., & Tuot, D. S. (2017) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x17711822 · Cited by 56 · see citing works
Self-citationAdoption and impact of an eConsult system in a fee-for-service setting
Gleason, N. J., Prasad, P. A., Ackerman, S., Ho, C. K., Monacelli, J., Wang, M., Collado, D., & Gonzales, R. (2016) · Healthcare · doi:10.1016/j.hjdsi.2016.05.005 · Cited by 54 · see citing works
Lee, B. J., & Forbes, K. (2009) · BMJ · doi:10.1136/bmj.b2395 · Cited by 52 · see citing works
Longer-term impact of cardiology e-consults
Wasfy, J. H., Rao, S. K., Kalwani, N. M., Chittle, M. D., Richardson, C. A., Gallen, K. M., Isselbacher, E. M., Kimball, A. B., & Ferris, T. G. G. (2015) · American Heart Journal · doi:10.1016/j.ahj.2015.11.019 · Cited by 52 · see citing works
Provider Satisfaction With Virtual Specialist Consultations in a Family Medicine Department
Angstman, K. B., Adamson, S. C., Furst, J. W., Houston, M. S., & Rohrer, J. E. (2009) · The Health Care Manager · doi:10.1097/hcm.0b013e318196def8 · Cited by 50 · see citing works
Impact of a national specialty e-consultation implementation project on access.
Kirsh, S. R., Carey, E., Aron, D. C., Cardenas, O., Graham, G. D., Jain, R., Au, D. H., Tseng, C., Franklin, H., & Ho, P. M. (2015) · PubMed · Cited by 50 · see citing works
Keely, E., Traczyk, L., & Liddy, C. E. (2015) · Canadian Journal of Diabetes · doi:10.1016/j.jcjd.2014.12.010 · Cited by 48 · see citing works
Self-citationProviding Specialty Consultant Expertise to Primary Care: An Expanding Spectrum of Modalities
Kirsh, S. R., Ho, P. M., & Aron, D. C. (2014) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2014.04.016 · Cited by 47 · see citing works
Leveraging an electronic referral system to build a medical neighborhood
Tuot, D. S., Murphy, E. J., McCulloch, C. E., Leeds, K., Chan, E. C. Y., & Chen, A. H. (2015) · Healthcare · doi:10.1016/j.hjdsi.2015.04.001 · Cited by 44 · see citing works
Self-citationClinical Consultations Using Store-and-Forward Telemedicine Technology
Houston, M. S., Myers, J. D., Levens, S. P., McEvoy, M. T., Smith, S. A., Khandheria, B. K., Shen, W.-K., Torchia, M. E., & Berry, D. J. (1999) · Mayo Clinic Proceedings · doi:10.4065/74.8.764 · Cited by 43 · see citing works
Erin, K., Paul, D., Amir, A., & Clare, L. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-505-0-38 · Cited by 35 · see citing works
Patterson, V., & Wootton, R. (2013) · Global Health Action · doi:10.3402/gha.v6i0.21465 · Cited by 34 · see citing works
Impact of Question Content on e-Consultation Outcomes
Tran, C., Liddy, C., Pinto, N., & Keely, E. (2016) · Telemedicine and e-Health · doi:10.1089/tmj.2015.0081 · Cited by 29 · see citing works
Self-citationNorth, F., Uthke, L. D., & Tulledge-Scheitel, S. M. (2015) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15572204 · Cited by 27 · see citing works
Self-citationTransforming the Endocrine Consult: Asynchronous Provider Consultations
Cruz, M. L., Gleason, N., Wang, M., Wrenn, K., & Gonzales, R. (2015) · Endocrine Practice · doi:10.4158/ep14297.or · Cited by 27 · see citing works
Early e-consultation face-to-face conversions
Pecina, J. L., & North, F. (2016) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x15602634 · Cited by 24 · see citing works
Self-citationClare, L., Catherine, D. A., Paul, D., Ferdinand, M.-Y., Amir, A., & Erin, K. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-505-0-67 · Cited by 21 · see citing works
ADVANCING THE PRIMARY/SPECIALTY CARE INTERFACE THROUGH ECONSULTS AND ENHANCED REFERRALS
Davis, A. K., Gilchrist, V. J., Grumbach, K., James, P., Kallenberg, R., & Shipman, S. A. (2015) · The Annals of Family Medicine · doi:10.1370/afm.1829 · Cited by 21 · see citing works
Liddy, C., Deri Armstrong, C., McKellips, F., & Keely, E. (2016) · Journal of the American Association of Nurse Practitioners · doi:10.1002/2327-6924.12266 · Cited by 20 · see citing works
Self-citationPrevention of delayed referrals through the Champlain BASE eConsult service.
Liddy, C. E., Drosinis, P., Fogel, A., & Keely, E. (2017) · PubMed Central · Cited by 20 · see citing works
Self-citationUlloa, J. G., Russell, M. D., Chen, A. H., & Tuot, D. S. (2017) · BMC Health Services Research · doi:10.1186/s12913-017-2375-0 · Cited by 15 · see citing works
Self-citationWootton, R., Liu, J., & Bonnardot, L. (2014) · Frontiers in Public Health · doi:10.3389/fpubh.2014.00247 · Cited by 13 · see citing works
Pecina, J. L., Frank, J. M., & North, F. (2016) · SAGE Open Medicine · doi:10.1177/2050312116682127 · Cited by 12 · see citing works
Self-citationOlayiwola, J. N., Knox, M. J., Dubé, K., Lu, E. C., Woldeyesus, T., James, I. E., Willard‐Grace, R., & Tuot, D. S. (2017) · Health Services Research · doi:10.1111/1475-6773.12776 · Cited by 12 · see citing works
Self-citationA retrospective analysis of medical record use in e-consultations
Pecina, J. L., & North, F. (2016) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x16654607 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- [23]Liddy C Rowan MS Afkham A Maranger J Keely E Building access to specialist care through e-consultation Open Med 2013 7 1 e1 e8 23687533 PMC3654501
- [40]Vital signs Core metrics for health and health care progress 2015 Washington The National Academies Press 26378329
- [42]Gleason N Prasad PA Ackerman S Ho C Monacelli J Wang M Adoption and impact of an eConsult system in a fee-for-service setting Healthc (Amst) 2017 5 1–2 40 45 10.1016/j.hjdsi.2016.05.005 27469441 · DOI
Cited by
Sorted by citations
Osman, M. A., Schick-Makaroff, K., Thompson, S., Bialy, L., Featherstone, R., Kurzawa, J., Zaidi, D., Okpechi, I., Habib, S., Shojai, S., et al. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2019-001629 · Cited by 108 · see citing works
“This is particularly salient when studying a topic such as eConsult, which is a complex intervention in an emerging field(18) with working concepts that are just evolving.(19) Moreover, a scoping review is appropriate when the literature includes a variety of study designs (quantitative, qualitative and mixed methods).”
Self-citationLópez Seguí, F., Vidal-Alaball, J., Sagarra Castro, M., García-Altés, A., & García Cuyàs, F. (2020) · Journal of Medical Internet Research · doi:10.2196/14478 · Cited by 42 · see citing works
Key factors for national spread and scale-up of an eConsult innovation
Moroz, I., Archibald, D., Breton, M., Cote-Boileau, E., Crowe, L., Horsley, T., Hyseni, L., Johar, G., Keely, E., Burns, K. K., et al. (2020) · Health Research Policy and Systems · doi:10.1186/s12961-020-00574-0 · Cited by 24 · see citing works
Self-citation
- Authors
- Tuot, D. S., Liddy, C., Vimalananda, V. G., Pecina, J., Murphy, E. J., Keely, E., Simon, S. R., North, F., Orlander, J. D., & Chen, A. H.
- Affiliations
- San Francisco General Hospital (US)
- University of Ottawa (CA)
- Boston University (US)
- Mayo Clinic (US)
- Harvard University (US)
- Corresponding author
- Tuot, Delphine S.
- ORCID
- Venue
- BMC Health Services Research
- Published
- 2018-10-24
- Volume
- 18
- Issue
- 1
- Pages
- 814-814
- ISSN
- 1472-6963 (ISSN-L)
- PMC
- PMC6201558
- OpenAlex
- W2898407995
- Cited by
- 45
- References
- 43
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 4 · 2023: 4 · 2022: 10 · 2021: 7 · 2020: 7 · 2019: 6 · 2017: 1
- FWCI
- 8.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Health Services Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- uO Research (University of Ottawa) (submittedVersion, public-domain)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- BlueCross BlueShield of South Carolina Foundation (P-1606-08494)
- Topics
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Healthcare Systems and Technology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- e-consultation, telemedicine, access to specialist care, program implementation, RE-AIM framework, Quadruple Aim, evaluation metrics, primary care, patient satisfaction
- MeSH terms
- Adult, Ambulatory Care Facilities, Delivery of Health Care, Diffusion of Innovation, Female, Health Personnel, Humans, Male, San Francisco, Specialization, United States, United States Department of Veterans Affairs, Remote Consultation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
