CoChat
Sign Up
Cited by 6Healthcare Systems and TechnologyTelemedicine and Telehealth ImplementationInnovations in Medical Education

eConsult specialist quality of response (eSQUARE): a novel tool to measure specialist correspondence via electronic consultation

Open access
Tran, Christopher, Archibald, Douglas, Humphrey-Murto, Susan, Wood, Timothy J, Dudek, Nancy, Liddy, Clare, Keely, Erin·2022·Journal of Telemedicine and Telecare
DOI 10.1177/1357633x21998216PMC PMC9066665
Open this paper in CoChat — free account

Abstract

High-quality correspondence between healthcare providers is critical for effective patient care. We developed an assessment tool to measure the quality of specialist correspondence to primary care providers (PCPs) via electronic consultation (eConsult), where specialists provide advice without specialist-patient interactions. We incorporated fourteen previously described features of high-quality eConsult correspondence into an assessment tool named the eConsult Specialist Quality of Response (eSQUARE). Six PCPs and two specialists applied the 10-item eSQUARE tool to 30 eConsults of varying quality as informed by PCP survey data. Content, response process, and internal structure validity evidence was gathered. Psychometric properties were calculated using descriptive statistics and generalizability analyses. Mean total score for low-quality eConsults (M = 24 ± 5.6) was significantly lower than moderate-quality eConsults (M = 38 ± 4.7; p<0.001) which was significantly lower than high-quality eConsults (M = 46 ± 3.0; p = 0.002). Reliability measures were high, including generalizability coefficient (0.96), inter-item (≥0.55) and item-total correlations (≥0.68). A decision study demonstrated that a single rater was adequate to achieve a reliability measure of ≥0.70. This study demonstrates initial validity evidence including multiple reliability measures for the eSQUARE. A single rater is adequate to achieve reliability measures for formative feedback. Future studies can apply the eSQUARE when planning educational initiatives aiming to improve specialist-to-PCP correspondence via eConsult.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1177/1357633x21998216

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 41 · Cited by 6

Research works printed but not linked: 40

  1. [1]O’Malley A Reschovsky J Saiontz-Martinez C. Interspecialty communication supported by health information technology associated with lower hospitalization rates for ambulatory care-sensitive conditions. J Am Board Fam Med 2015; 28: 404– 417. 25957373
  2. [2]Vermeir P Vandijck D Degroote S, et al. Communication in healthcare: a narrative review of the literature and practical recommendations. Int J Clin Pract 2015; 69: 1257– 1267. 26147310 PMC4758389
  3. [3]O’Malley A Reschovsky J. Referral and consultation communication between primary care and specialist physicians. Arch Intern Med 2011; 171: 56– 65. 21220662
  4. [4]Vimalananda V Gupte G Seraj S, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare 2015; 21: 323– 330. 25995331 PMC4561452
  5. [5]Olayiwola J Potapov A Gordon A, et al. Electronic consultation impact from the primary care clinician perspective: outcomes from a national sample. J Telemed Telecare 2019; 25: 493– 498. 29991315
  6. [6]Kirsh S Carey E Aron D, et al. Impact of a national specialty e-consultation implementation project on access. Am J Manag Care 2015; 21: e648– 654. 26760427
  7. [7]Tuot D Murphy E McCulloch C, et al. Leveraging an electronic referral system to build a medical neighborhood. Healthcare 2015; 3: 202– 208. 26699344
  8. [8]Joschko J Keely E Grant R, et al. Electronic consultation services worldwide: environmental scan. J Med Internet Res 2018; 20: e11112. 30578187 PMC6320413
  9. [9]Keely E Liddy C. Transforming the specialist referral and consultation process in Canada. Can Med Assoc J 2019; 191: E408– 409. 30988040 PMC6464882
  10. [10]Keely E Drosinis P Afkham A, et al. Perspectives of Champlain BASE specialist physicians: their motivation, experiences and recommendations for providing eConsultations to primary care providers. Stud Health Technol Inform 2015; 209: 38– 45. 25980703
  11. [11]Keely E Williams R Epstein G, et al. Specialist perspectives on Ontario provincial electronic consultation services. Telemed J E Health 2019; 25: 3– 10. 29746210
  12. [14]Lee M Ray K Mehrotra A, et al. Primary care practitioners’ perceptions of electronic consult systems. JAMA Intern Med 2018; 178: 782– 789. 29801079 PMC6145753
  13. [15]Rash A Sheldon R Donald M, et al. Valued components of a consultant letter from referring physicians’ perspective: a systematic literature synthesis. J Gen Intern Med 2018; 33: 948– 954. 29508258 PMC5975156
  14. [16]Russell C Sandu V Moroz I, et al. Key components of traditional consultation letters and their relevance to electronic consultation replies: a systematic review. Telemed J E Health 2020; 26: 689– 699. 31596680
  15. [17]Fox A Palmer R Crossley K, et al. Improving the quality of outpatient clinic letters using the Sheffield Assessment Instrument for Letters (SAIL). Med Educ 2004; 38: 852– 858. 15271046
  16. [18]Keely E Myers K Dojeiji S, et al. Peer assessment of outpatient consultation letters – feasibility and satisfaction. BMC Med Educ 2007; 7: 13. 17519024 PMC1890286
  17. [19]Tuot D Sehgal N Neeman N, et al. Enhancing quality of trainee-written consultation notes. Am J Med 2012; 125: 649– 652. 22560812 PMC3832626
  18. [20]Tran C Archibald D Humphrey-Murto S, et al. What makes a high-quality electronic consultation (eConsult)? A nominal group study. J Telemed Telecare 2020; 26: 239– 247. 30717627
  19. [21]Cook D Beckman T. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006; 119: 166.e7–16. 16443422
  20. [22]Halman S Dudek N Wood T, et al. Direct observation of clinical skills feedback scale: development and validity evidence. Teach Learn Med 2016; 28: 385– 394. 27285377
  21. [23]Bierman J Hufmeyer K Liss D, et al. Promoting responsible electronic documentation: validity evidence for a checklist to assess progress notes in the electronic health record. Teach Learn Med 2017; 29: 420– 432. 28497983
  22. [24]Weber D Held J Jandarov R, et al. Development and establishment of initial validity evidence for a novel tool for assessing trainee admission notes. J Gen Intern Med 2020; 35: 1078– 1083. 31993944 PMC7174454
  23. [25]Stansfield R Gruppen L. Power analyses: planning, conducting and evaluating education research. In: Cleland J Durning SJ (eds) Research medical education. Oxford: John Wiley, 2015, pp. 43– 50.
  24. [28]Crossley J Russell J Jolly B, et al. ‘I’m pickin’ up good regressions’: the governance of generalisability analyses. Med Educ 2007; 41: 926– 934. 17908111
  25. [32]Vispoel W Morris C Kilinc M. Practical applications of generalizability theory for designing, evaluating, and improving psychological assessments. J Pers Assess 2018; 100: 53– 67. 28418721
  26. [33]Norman G. Likert scales, levels of measurement and the ‘laws’ of statistics. Adv Health Sci Educ 2010; 15: 625– 632. 20146096
  27. [34]Willits F Theodori G Luloff A. Another look at Likert scales. J Rural Soc Sci 2016; 31: 126– 139.
  28. [35]Downing S. Reliability: on the reproducibility of assessment data. Med Educ 2004; 38: 1006– 1012. 15327684
  29. [36]Edwards M Slagle A Rubright J, et al. Fit for purpose and modern validity theory in clinical outcomes assessment. Qual Life Res 2018; 27: 1711– 1720. 28688085
  30. [37]Lineberry M. Validity and quality. Assessment in health professions education. New York: Routledge, 2019.
  31. [38]Reeves T Marbach-Ad G. Contemporary test validity in theory and practice: a primer for discipline-based education researchers. CBE Life Sci Educ 2016; 15: 1– 9. PMC4803101 26903498
  32. [39]Park Y. Reliability. Assessment in health professions education. New York: Routledge, 2019.
  33. [40]Crossley J Howe A Newble D, et al. Sheffield Assessment Instrument for Letters (SAIL): performance assessment using outpatient letters. Med Educ 2001; 35: 1115– 1124. 11895235
  34. [41]Williams R Klamen D McGaghie W. Cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med 2003; 15: 270– 292. 14612262
  35. [42]Stille C Mazor K Meterko V, et al. Development and validation of a tool to improve paediatric referral/consultation communication. BMJ Qual Saf 2011; 20: 692– 697. 21339312
  36. [43]Cook D Dupras D Beckman T, et al. Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial. J Gen Intern Med 2009; 24: 74– 79. 19002533 PMC2607488
  37. [44]Streiner D Norman G Cairney J. Biases in responding. Health measurement scales. Oxford: Oxford University Press, 2015, pp. 100– 130.
  38. [45]Colquhoun H Brehaut J Sales A, et al. A systematic review of the use of theory in randomized controlled trials of audit and feedback. Implement Sci 2013; 8: 66. 23759034 PMC3702512
  39. [46]Keely E Archibald D Tuot D, et al. Unique educational opportunities for PCPs and specialists arising from electronic consultation services. Acad Med 2017; 92: 45– 51. 28030423
  40. [47]Sewell J Day L Tuot D, et al. A brief, low-cost intervention improves the quality of ambulatory gastroenterology consultation notes. Am J Med 2013; 126: 732– 738. 23791206 PMC3730533

Authors
Tran, C., Archibald, D., Humphrey-Murto, S., Wood, T. J., Dudek, N., Liddy, C., & Keely, E.
Affiliations
University of Ottawa (CA)
Corresponding author
Tran, Christopher
ORCID
Venue
Journal of Telemedicine and Telecare
Publisher
SAGE Publications
Published
2021-03-03
Volume
28
Issue
4
Pages
280-290
ISSN
  • 1357-633X (ISSN-L)
  • 1758-1109
DOI
10.1177/1357633x21998216
PMC
PMC9066665
OpenAlex
W3134946996
Cited by
6
References
41
Citations by year
2026: 3 · 2023: 3
FWCI
0.73
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Telemedicine and Telecare (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Healthcare Systems and Technology
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
Keywords
e-consultation, primary care providers, scale development, psychometric properties, generalizability, inter-rater reliability, formative feedback, interprofessional communication
MeSH terms
Health Services Accessibility, Humans, Primary Health Care, Referral and Consultation, Specialization, Reproducibility of Results, Remote Consultation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review