A Systematic Review of Asynchronous, Provider-to-Provider, Electronic Consultation Services to Improve Access to Specialty Care Available Worldwide

In the paper

“It was additionally shown that many eConsult services were primarily limited to single specialty services, particularly dermatology, thus generalizability of impact across other specialty areas was not established.”

What the sources say

• ?Not checked — no full text of the sourcecites [5] 5. Liddy C, Drosinis P, Keely E. Electronic consultation systems: Worldwide prevalence and their impact on patient care-A systematic review. Fam Pract 2016;33:274-285. source
• Disputed

  The Results (Study characteristics; page unavailable) say that the three sites accounting for most reviewed articles offered multiple specialties; only five of 27 articles described single-specialty programs. The Methods (Eligibility criteria; page unavailable) say image-based dermatology consultations were excluded. The Discussion (Limitations; page unavailable) limits generalizability because studies were concentrated in three health systems, not because most services were dermatology-only. (LLM, conf=0.88)

  Checked by a reasoning model — may be wrong.

  cites [6] 6. Vimalananda VG, Gupte G, Seraj SM, Orlander J, Berlowitz D, Fincke BG, et al. Electronic consultations (e-consults) to improve access to specialty care: A systematic review and narrative synthesis. J Telemed Telecare 2015;21:323-330. sourceExperimental — accuracy not yet measured

In the paper

“Previous reviews evaluating Consult's use and impact demonstrated its overall feasibility in application and its ability to improve timely access to specialty care.5,6”

What the sources say

• ?Not checked — no full text of the sourcecites [5] 5. Liddy C, Drosinis P, Keely E. Electronic consultation systems: Worldwide prevalence and their impact on patient care-A systematic review. Fam Pract 2016;33:274-285. source
• Supported

  The Discussion’s Summary of findings explicitly concludes that e-consults are feasible in a variety of settings and flexible in their application. The Abstract’s Discussion repeats that conclusion, supporting the attributed claim of overall feasibility. (LLM, conf=0.94)

  cites [6] 6. Vimalananda VG, Gupte G, Seraj SM, Orlander J, Berlowitz D, Fincke BG, et al. Electronic consultations (e-consults) to improve access to specialty care: A systematic review and narrative synthesis. J Telemed Telecare 2015;21:323-330. source

In the paper

“This framework proposes that the optimization of the performance of the health system should involve simultaneously pursuing four interdependent objectives: population health, experience of receiving care (patients), experience of providing care (providers), and per capita costs.”

What the sources say

• Supported

  The opening section identifies population health, patient experience, and per capita cost as the three dimensions pursued simultaneously. The Abstract recommends adding providers’ work life as a fourth aim, and the Conclusion describes its importance to achieving population health. The supplied passages do not give page numbers. (LLM, conf=0.93)

  cites [7] 7. Bodenheimer T, Sinsky C. From triple to quadruple aim: Care of the patient requires care of the provider. Ann Fam Medicine 2014;12:573-576. source

In the paper

“Other eConsult services are independent of individual health records and use a secure web-based portals to link providers, such as the Champlain BASE in Canada, [20][21][22][23][24][25][26][27][28][29][30][31][32] TNMG in Brazil, [33][34][35][36] and KSYOS in The Netherlands, 37 or harness secure intranet e-mail as the means of communication, such as Army Knowledge Online (AKO).38”

What the sources say

• ?Not checked — no full text of the sourcecites [20] 20. Fogel A, Khamisa K, Afkham A, Liddy C, Keely E. Ask the eConsultant: Improving access to haematology expertise using an asynchronous eConsult system. J Telemed Telecare 2016;23:421-427. source
• Supported

  The Abstract identifies Champlain BASE as a secure, web-based tool linking primary care providers with specialists. The eConsult Service section describes providers logging into the system, uploading information from electronic medical records, and optionally downloading the completed discussion into a patient’s record. This supports the Champlain BASE example to which this citation is attached. (LLM, conf=0.91)

  cites [21] 21. Liddy C, Afkham A, Drosinis P, Joschko J, Keely E. Impact and satisfaction with a new eConsult service: A mixed methods study of primary care providers. J Am Board Fam Med 2015;28:394-403. source
• ?Not checked — no full text of the sourcecites [22] 22. Liddy C, Deri Armstrong C, Drosinis P, Mito-Yobo F, Afkham A, Keely E. What are the costs of improving access to specialists through eConsultation? The Champlain BASE experience. Stud Health Technol Inform 2015;209:75-83. source
• Partly supported

  Methods > The eConsult service (page not provided) identifies Champlain BASE as a service in Canada and says primary care providers use a secure web-based platform to contact specialists. It does not say whether the platform is independent of individual health records. This source is cited for Champlain BASE, not for the separately cited services. (LLM, conf=0.84)

  cites [23] 23. Liddy C, Drosinis P, Deri Armstrong C, McKellips F, Afkham A, Keely E. What are the cost savings associated with providing access to specialist care through the Champlain BASE eConsult service? A costing evaluation. BMJ Open 2016; 6:e010920. sourceExperimental — accuracy not yet measured
• Partly supported

  The Introduction identifies Champlain BASE as an eConsult service launched in Ottawa, Canada, and Method > The eConsult Service describes its secure web-based platform linking primary care providers with specialists. Our copy does not establish that the platform is independent of individual health records. The citations following TNMG, KSYOS, and AKO attach those examples to other sources, not this paper. (LLM, conf=0.84)

  cites [24] 24. Liddy C, Drosinis P, Joschko J, Keely E. Improving access to specialist care for an aging population. Gerontol Geriatr Med 2016;2:1-7. sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [25] 25. Johnston DL, Murto K, Kurzawa J, Liddy C, Keely E, Lai L. Use of electronic consultation system to improve access to care in pediatric hematology/ oncology. J Pediatr Hematol Oncol 2017;39:e367-e369. source
• ?Not checked — no full text of the sourcecites [26] 26. Tran C, Liddy C, Pinto N, Keely E. Impact of question content on e-consultation outcomes. Telemed J E Health 2016;22:216-222. source
• Partly supported

  The Methods section (page not provided) describes Champlain BASE in Canada as a secure web-based service through which primary care providers send questions to specialists. Our copy does not say that it is independent of individual health records. This citation is attached to Champlain BASE, not to the separately cited TNMG, KSYOS, or AKO examples. (LLM, conf=0.84)

  cites [27] 27. Murthy R, Rose G, Liddy C, Afkham A, Keely E. eConsultations to infectious disease specialists: Questions asked and impact on primary care providers' behavior. Open Forum Infect Dis 2017;4:0fx030. sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [28] 28. Shehata F, Posner G, Afkham A, Liddy C, Keely E. Evaluation of an electronic consultation service in obstetrics and gynecology in Ontario. Obstetr Gynecol 2016;127:1033-1038. source
• ?Not checked — no full text of the sourcecites [29] 29. Liddy C, Smyth C, Poulin PA, Joschko J, Rebelo M, Keely E. Improving access to chronic pain services through econsultation: A cross-sectional study of the Champlain BASE eConsult Service. Pain Med 2016;17:1049-1057. source

In the paper

“In Brazil, specialists described the referral process as unorganized, lacking patient triaging, and consisting of inadequate communication between providers.4”

What the sources say

• Supported

  In Results §3.2 (page not provided), specialists describe patient lists without logical organization and overcrowding because there is no prior triage or screening. A participant also says referrals and return referrals often fail to reach the other physician because there is no direct link between providers. (LLM, conf=0.94)

  cites [4] 4. Juliani C, MacPhee M, Spiri W. Brazilian specialists perspectives on the patient referral process. Healthcare (Basel) 2017;5:4. source

In the paper

“In other studies, when asked about satisfaction with eConsult, 75-100% of providers reported being satisfied, 33,47 and most providers reported that they would use eConsult again in the future.13”

What the sources say

• ?Not checked — no full text of the sourcecites [33] 33. Marcolino M, Minelli Figueira R, Pereira Afonso dos Santos J, Silva Cardoso C, Luiz Ribeiro A, Alkmim MB. The experience of a sustainable large scale Brazilian telehealth network. Telemed J E Health 2016;22:899-908. source
• ?Not checked — no full text of the sourcecites [47] 47. Segura BT, Bustabad S. A new form of communication between rheumatology and primary care: The virtual consultation. Reumatol Clin 2016;12:11-14. source
• Supported

  Citation 13 is attached to the claim about future use, not the 75–100% satisfaction figure, which has separate citations. In Results > Intent to Use e-Consult in the Future, the paper reports that 93% of PCPs and 100% of specialists intended to use e-consults again. (LLM, conf=0.95)

  cites [13] 13. Rodriguez KL, Burkitt KH, Bayliss NK, Skoko JE, Switzer GE, Zickmund SL, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform 2015;3:e5. source

In the paper

“Furthermore, patients identified communication (effective communication with PCPs or providers) as the domain that was most important regarding their overall satisfaction with the eConsult process and unanimously indicated that they intended to use eConsults in the future based on the quality of care and the timeliness of care.13”

What the sources say

• Supported

  The Results (Overall Satisfaction) say patients identified communication, often with PCPs or other providers, as the most important domain for their satisfaction. The Results (Intent to Use e-Consult in the Future) report that 100% of interviewed patients intended to use e-consults again, with their reasons focused primarily on quality of care (n=4) and timeliness (n=1). (LLM, conf=0.92)

  cites [13] 13. Rodriguez KL, Burkitt KH, Bayliss NK, Skoko JE, Switzer GE, Zickmund SL, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform 2015;3:e5. source

In the paper

“The results showed that those who had an eConsult were significantly more likely (1.5 times) to receive a consultation with a cardiologist and had lower rates of emergency department utilization than those who had traditional face-toface consultations.39”

What the sources say

• Partly supported

  Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  cites [39] 39. Olayiwola JN, Anderson D, Jepeal N, Aseltine R, Pickett C, Yan J, et al. Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: A cluster-randomized controlled trial. Ann Fam Med 2016;14:133-140. sourceExperimental — accuracy not yet measured

In the paper

“They also rated the service high (median ratings of 4) for quality of care, timeliness, improved access, and safety.”

What the sources say

• Supported

  The Results sections for Quality, Time, Access, and Safety each report a veteran satisfaction rating of 4.0 for the e-consult program. The paper measures a broader time domain rather than timeliness alone, but its Discussion describes patients as equally satisfied across all four areas. (LLM, conf=0.91)

  cites [13] 13. Rodriguez KL, Burkitt KH, Bayliss NK, Skoko JE, Switzer GE, Zickmund SL, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform 2015;3:e5. source

In the paper

“Alternatively, another study estimated monthly savings for the military (using a cost minimization analysis) by using eConsult to be $140,907 (yearly savings were $467,181); where civilian monthly savings were estimated as $28,260 (yearly savings were $105,400).46”

What the sources say

• Partly supported

  The Results report projected monthly savings of $140,907 from prevented medical evacuations and $28,260 from prevented civilian referrals, based on regression of the first-year data. Table 1 gives $467,181 in first-year evacuation savings; the Methods’ estimate of $1,054 per civilian referral and the 100 prevented referrals reported in the abstract yield $105,400. The citing sentence does not distinguish the projected monthly figures from the observed first-year totals, and the source describes regression analysis rather than a cost-minimization analysis. (LLM, conf=0.88)

  cites [46] 46. Lin AH, Cole JH, Chin JC, Mahnke CB. The Health Experts onLine at Portsmouth (HELP) system: One-year review of adult and Pediatric Asynchronous Telehealth Consultations. SAGE Open Med 2016;4: 2050312115626433. sourceExperimental — accuracy not yet measured

In the paper

“A retrospective analysis of dermatology cases referred by MSF field doctors revealed that only 10 of 65 cases (15%) had patient follow-up data with two cases of patient death.41”

What the sources say

• Supported

  The Materials and Methods section describes a retrospective analysis of dermatological cases referred by MSF field doctors. The Results (Case characteristics) report 65 cases, of which only 10 (15%) had follow-up data; two reported the patient's death. (LLM, conf=0.95)

  cites [41] 41. Delaigue S, Morand JJ, Olson D, Wootton R, Bonnardot L. Teledermatology in low-resource settings: The MSF experience with a multilingual tele-expertise platform. Front Public Health 2014;14;2:233. source

In the paper

“The lowest was 7.4% 43 and the highest 78%, 36 with most studies reporting between 22% and 68% reductions in face-to-face specialist visits.”

What the sources say

• Supported

  The Abstract and Discussion report that 36 of 487 patients (7.4%) did not need a urology clinic visit. This supports the 7.4% figure attributed to this paper; the 78% figure is attached to a different citation. (LLM, conf=0.94)

  cites [43] 43. McGeady JB, Blaschko SD, Brajtbord JS, Sewell JL, Chen AH, Breyer BN. Electronic preconsultation as a method of quality improvement for urological referrals. Urolog Pract 2014;1:172-175. source
• ?Not checked — no full text of the sourcecites [36] 36. Marcolino MS, Alkmim MB. Teleconsultations to provide support for primary care practitioners and improve quality of care-The experience of a large scale telehealth service in Brazil. Stud Health Technol Inform 2015; 216:987. source

In the paper

“Issues in communication refer to unclear specialist responses or expectations, which may be the result of unclear questions and expectations posed, 15 or from an unclear understanding on how to use the program.41”

What the sources say

• Partly supported

  The Results (Qualitative Findings; page unavailable) report that two infrequent users were unfamiliar with the e-consult process, and leaders said substantial education was needed. The Discussion (Principal Findings; page unavailable) calls for better communication about the groups’ needs and expectations. The paper does not establish that unfamiliarity with the program caused unclear specialist responses or expectations. (LLM, conf=0.82)

  cites [15] 15. Gupte G, Vimalananda V, Simon SR, DeVito K, Clark J, Orlander JD. Disruptive innovation: Implementation of electronic consultations in a Veterans Affairs Health Care System. JMIR Med Inform 2016;4:e6. sourceExperimental — accuracy not yet measured
• Supported

  Citation 41 follows the clause about unclear understanding of how to use the program. In “Weaknesses and recommendations” (page unavailable), the paper recommends giving users clear information about how and when to use the system; its survey also records a complaint about insufficient support in using it. These passages support the cited clause, without establishing it as a proven cause of unclear specialist responses. (LLM, conf=0.86)

  cites [41] 41. Delaigue S, Morand JJ, Olson D, Wootton R, Bonnardot L. Teledermatology in low-resource settings: The MSF experience with a multilingual tele-expertise platform. Front Public Health 2014;14;2:233. source

In the paper

“Some challenges associated with eConsult were those relating to unclear directions from specialists, an occasional lack of information or pertinent questions delivered to specialist by the PCP, and lack of patient follow-up.41”

What the sources say

• Partly supported

  The Results (Survey results; page unavailable) identify poor clinical information from referrers and lack of patient follow-up feedback as concerns. Table 3 (Materials and Methods, Survey; page unavailable) also records one complaint about a specialist answer that was not well adapted, but it does not establish that specialist directions were unclear or that referrers failed to ask pertinent questions. The paper concerns MSF teledermatology rather than eConsult generally. (LLM, conf=0.85)

  cites [41] 41. Delaigue S, Morand JJ, Olson D, Wootton R, Bonnardot L. Teledermatology in low-resource settings: The MSF experience with a multilingual tele-expertise platform. Front Public Health 2014;14;2:233. sourceExperimental — accuracy not yet measured

In the paper

“Fort et al. reported that many of the specialists participating in the Safety Net Specialty Care eConsult Program in Denver Metropolitan Area believed there was room to expand the program beyond the current volume of eConsults, although this perception varied by department, with high-volume or lower-staff-ratio departments expressing interest in maintaining the program at its current size.52”

What the sources say

• Supported

  Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  cites [52] 52. Fort MP, Namba LM, Dutcher S, Copeland T, Bermingham N, Fellenz C, et al. Implementation and evaluation of the safety net specialty care program in the Denver Metropolitan area. Permanente J 2017;21:16-22. source

In the paper

“study determined the median time for patient notification after eConsult and found it to be 3 days.18”

What the sources say

• Supported

  The Results report a median of 3 days from e-consultation completion to patient notification (IQR, 1–6 days). This was measured for e-consultations with documented patient notification. (LLM, conf=0.95)

  cites [18] 18. Pecina JL, Frank JM, North F. A retrospective study on how primary care providers manage specialists' recommendations after an e-consultation. SAGE Open Med 2016;4:2050312116682127. source

In the paper

“While an RCT demonstrated the potential of eConsult for reducing adverse cardiovascular outcomes, 39 2 retrospective studies reported potential for harm due to lack of patient follow-up.40,41”

What the sources say

• Supported

  The Results report fewer emergency department visits for possible cardiac symptoms among e-consultation patients than controls over six months. The Discussion describes a potential reduction in emergency department use, while noting that the trial was not powered to detect statistical differences. This supports the citing sentence’s cautious claim of potential benefit; the separate retrospective-study claim is attributed to other citations. (LLM, conf=0.86)

  cites [39] 39. Olayiwola JN, Anderson D, Jepeal N, Aseltine R, Pickett C, Yan J, et al. Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: A cluster-randomized controlled trial. Ann Fam Med 2016;14:133-140. source
• ?Not checked — no full text of the sourcecites [40] 40. Price EL, Sewell JL, Chen AH, Sarkar U. Minding the gaps: Assessing communication outcomes of electronic preconsultation exchange. Jt Comm J Qual Patient Saf 2016;42:341-354. source
• Partly supported

  The Materials and Methods section (Confidentiality and security; page unavailable) identifies this work as a retrospective chart review. Results (Case characteristics; page unavailable) report that only 10 cases had follow-up data, and the Discussion (Limitations of the study; page unavailable) says the missing information prevented assessment of the system’s overall impact. The paper does not report that lack of patient follow-up caused harm; the RCT finding is attached to a different citation. (LLM, conf=0.86)

  cites [41] 41. Delaigue S, Morand JJ, Olson D, Wootton R, Bonnardot L. Teledermatology in low-resource settings: The MSF experience with a multilingual tele-expertise platform. Front Public Health 2014;14;2:233. sourceExperimental — accuracy not yet measured

In the paper

“Study designs varied with most studies (40%) classified as retrospective and cross-sectional (16%), and smaller proportion of mixed method designs (7%) and randomized controlled trials (RCTs; 7%).”

In the paper

“eConsult services were implemented on a variety of technological platforms (Table 4).”

In the paper

“Only 7 (16%) studies described a single specialty service, which included services in the following specialty groups: dermatology, cardiology, and pulmonology.”

In the paper

“Table 2 shows that the majority of studies were conducted in the United States (44%) and Canada (28%), with some in Brazil (9%), Europe (Spain, Italy, Austria, The Netherlands; 7%), and Australia (2%).”

In the paper

“A search of target databases on May 11, 2017 revealed 1,021 citations.”

In the paper

“Figure 2 shows the distribution of top specialties accessed through eConsult, with endocrinology being the most frequently accessed at 19%, followed by dermatology (16%), hematology (9%), cardiology (9%), gastroenterology 8%), and neurology (7%).”

In the paper

“The breadth of specialty services offered has greatly expanded beyond dermatology, with most studies focused on multispecialty service (84%), ranging from 8 to 86 specialties.”

In the paper

“The number of eConsults reported ranged from 65 to 217,014, and number of participating providers ranged from 17 to 4,260 PCPs and 4 to 479 specialists.”

In the paper

“There were multiple articles describing six eConsult services, which accounted for 67% (n = 29) of the articles: 30% (n = 13) from the Champlain BASE Ô eConsult service, 9% (n = 4) from the TNMG (Telehealth Network of Minas Gerais) in Brazil, 7% (n = 3) from the Mayo Clinic, 7% (n = 3) from San Francisco General Hospital, 5% (n = 2) from the University of California San Francisco (UCSF), 5% (n = 2”

In the paper

“Most articles described eConsult for PCP-specialist communication, with only one reporting on specialist-to-specialist communication.10”

What the sources say

• ?Not checked — no full text of the sourcecites [10] 10. North F, Uthke LD, Tulledge-Scheitel SM. Internal e-consultations in an integrated multispecialty practice: A retrospective review of use, content, and outcomes. J Telemed Telecare 2015;21:151-159. source

In the paper

“eConsult's educational value was noted in several studies, with 90% of providers reported having learned from eConsult, 37 whereas in others 74% reported their questions were answered, 35 and 89% thought eConsult results were conclusive.47”

What the sources say

• ?Not checked — no full text of the sourcecites [37] 37. Thijssing L, van der Heijden JP, Melissant CF, Chavannes NH, Witkamp L, Jaspers MW. Telepulmonology and telespirometry. In: Lovis C, et al., eds. e-Health-For continuity of care. Amsterdam: IOS Press; 2014:211-215. source
• ?Not checked — no full text of the sourcecites [35] 35. Alkmim MB, Marcolino MS, Maia JX, Pessoa CG, Machado E, Sousa L. Clinical quality control of a large-scale teleconsultation service. Stud Health Technol Inform 2015;216:988. source
• ?Not checked — no full text of the sourcecites [47] 47. Segura BT, Bustabad S. A new form of communication between rheumatology and primary care: The virtual consultation. Reumatol Clin 2016;12:11-14. source

In the paper

“Providers also reported that psychiatric eConsults increased access to specialty consultations for mental health and improved support for diagnosis and treatment.48”

What the sources say

• ?Not checked — no full text of the sourcecites [48] 48. Golberstein E, Kolvenbach S, Carruthers H, Druss B, Goering P. Effects of electronic psychiatric consultations on primary care provider perceptions of mental health care: Survey results from a randomized evaluation. Healthc (Amst) 2018;6:17-22. source

In the paper

“Finally, another study suggested a potential for millions in transportation saving by implementing eConsult to minimize face-toface consultations.12”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [12] 12. Kirsh S, Carey E, Aron DC, Cardenas O, Graham G, Jain R, et al. Impact of a national specialty e-consultation implementation project on access. Am J Manag Care 2015;21:e648-e654.

In the paper

“Within referrals considered resolved (where the referral complaints have been resolved or addressed), only nine patients (9%) received unplanned care while awaiting scheduling decisions, five of whom experienced harm that was related to referral complaints, although authors noted that scheduling of appointments may not have avoided this harm.”

What the sources say

• ?Not checked — no full text of the sourcecites [40] 40. Price EL, Sewell JL, Chen AH, Sarkar U. Minding the gaps: Assessing communication outcomes of electronic preconsultation exchange. Jt Comm J Qual Patient Saf 2016;42:341-354. source

In the paper

“The authors concluded that the few adverse outcomes that were observed were not due to communication lapses in the referral process, but rather to being unintentionally left unscheduled with evidence for discontinuity of care and lack of patient or provider follow-up.”

What the sources say

• ?Not checked — no full text of the sourcecites [40] 40. Price EL, Sewell JL, Chen AH, Sarkar U. Minding the gaps: Assessing communication outcomes of electronic preconsultation exchange. Jt Comm J Qual Patient Saf 2016;42:341-354. source

In the paper

“| Alkmim 34 | 2015 | Brazil | Multispecialty | Cross-sectional | 1,954 | Factors associated with | Weak |”

In the paper

“| Olayiwola 39 | 2016 | United States | Single specialty | Cluster-randomized | 19 PCPs (control), 17 | Specialist response time, | Moderate |”

In the paper

“While a wider breadth of specialty services are now being offered by many eConsult services worldwide, which may enable a wider population to obtain access to specialty care, there are still significant gaps in understanding the impact of the service on population health using traditional measures of mortality, morbidity, and clinical outcomes.”

In the paper

“| Nami 50 | 2015 | Italy and Austria | Single specialty | Clinical trial | 391 Patients | Concordance between | Weak |”

In the paper

“| Cruz 17 | 2015 | United States | Multispecialty | Program evaluation 158 eConsults | Response time changes in | Weak |”

In the paper

“| Bonnardot 45 | 2014 | Global (MSF) | Multispecialty | Retrospective | 1,039 Telemedicine | Response time, provider | Weak |”

In the paper

“| Delaigue 41 | 2014 | Multicountry (Sudan, | Single specialty | Retrospective | 65 Teledermatology | Response time, areas for | Weak |”

In the paper

“| Lin 46 | 2016 | Global (NMCP) | Multispecialty | Teleconsultation | 585 Teleconsultations Referral/medevac | Weak |”

In the paper

“| Marcolino 36 | 2015 | Brazil | Multispecialty | Observational | 63,975 | Referral avoidance, | Weak |”

In the paper

“| Rodriguez 13 | 2015 | United States | Multispecialty | Quality improvement | 15 Veteran patients, | Provider satisfaction, | Strong |”

In the paper

“| Scheibe 51 | 2015 | United States | Multispecialty | Retrospective chart | 2,105 eReferrals | Volume/appropriateness | Moderate |”

In the paper

“| Fort 52 | 2017 | United States | Multispecialty | Mixed methods | 10 Patients, 18 safety | Patient satisfaction, | Weak |”

In the paper

“In general, these studies yield promising findings in terms of cost savings to the healthcare systems, return of investment, and societal savings.”

In the paper

“| Segura 47 | 2016 | Spain | Multispecialty | Observational | 938 Virtual consults, | Avoidance of referrals, | Weak |”

In the paper

“In our own recent study, we interviewed patients who have had an eConsult done on their behalf by their PCP and reported that all patients expressed acceptance for eConsult as a model for improving access to specialist care, had largely positive experiences with it as a model of care delivery, and supported its use in their future care.55”

What the sources say

• ?Not checked — no full text of the sourcecites [55] 55. Joschko J, Liddy C, Moroz I, Reiche M, Crowe L, Afkham A, et al. Just a click away: Exploring patients' perspectives on receiving care through the Champlain BASE Ô eConsult service. Fam Pract 2018;35:93-98. source

In the paper

“| Fogel 20 | 2016 | Canada | Multispecialty | Cross-sectional | 436 Hematology | Avoidance of referrals, | Weak |”

In the paper

“Open-ended input responses revealed that most providers appreciated the service, finding it safe, timely, easy to use, beneficial to patient care, and capable of improving communication and facilitating provider education.25”

What the sources say

• ?Not checked — no full text of the sourcecites [25] 25. Johnston DL, Murto K, Kurzawa J, Liddy C, Keely E, Lai L. Use of electronic consultation system to improve access to care in pediatric hematology/ oncology. J Pediatr Hematol Oncol 2017;39:e367-e369. source

In the paper

“| Pecina 18 | 2016 | United States | Multispecialty | Retrospective study 1,041 eConsults | Follow-up of | Weak |”

In the paper

“While the majority of new research examined outcomes related to patient and provider experience, the population health and cost measures continue to be limited and require further development.”

In the paper

“| Kirsh 12 | 2015 | United States | Multispecialty | Observational cohort 217,014 eConsults | Growth of eConsults by | Moderate |”

In the paper

“| Gupte 15 | 2016 | United States | Multispecialty | Observational quality | 7,097 eConsults | Provider experience, | Weak |”

In the paper

“The results to date indicate that eConsult systems have expanded significantly internationally and are largely multiservice in nature.”

In the paper

“| Liddy 22 | 2015 | Canada | Multispecialty | Economic analysis | 235 PCPs, 27 | Cost savings | Weak |”

In the paper

“Moreover, although the few studies which assessed specialist experiences found them to be positive (i.e., in terms of perceived impact on patient, ease of use, improving access, usefulness), one identified challenge was the potential for an increase in workload for specialists, along with the potential for issues in communication.30”

What the sources say

• ?Not checked — no full text of the sourcecites [30] 30. Keely E, Drosinis P, Afkham A, Liddy C. Perspectives of Champlain BASE specialist physicians: Their motivation, experiences and recommendations for providing eConsultations to primary care providers. Stud Tech Health Inform 2015;209:38-45. source

In the paper

“| Byrom 42 | 2016 | Australia | Single specialty | Retrospective case | 406 Telederm cases | Response time, | Weak |”

In the paper

“| Maddry 38 | 2014 | United States (note: | Multispecialty | Retrospective | 99 Teleconsultations | Response time, type, and | Weak |”

In the paper

“| Golberstein 48 | 2017 | United States | Multispecialty | Cluster-randomized | 72 Control PCPs, 70 | Effect of eConsult on PCP | Weak |”

In the paper

“| Liddy 24 | 2016 | Canada | Multispecialty | Cross-sectional | 1,796 eConsults | Response time, impact of | Weak |”

In the paper

“| Murthy 27 | 2017 | Canada | Multispecialty | Retrospective | 224 Infectious | eConsult response time, | Weak |”

In the paper

“| Price 40 | 2016 | United States | Multispecialty | Chart review | 266 eReferrals | Reasons for not | Weak |”

In the paper

“| Liddy 29 | 2016 | Canada | Multispecialty | Cross-sectional | 93 PCPs | Response time, provider | Weak |”

In the paper

“| McGeady 43 | 2014 | United States | Multispecialty | Retrospective chart | 1,705 eReferrals, 487 | Avoidance of referrals, | Weak |”

In the paper

“| Liddy 23 | 2016 | Canada | Multispecialty | Costing evaluation | 3,487 eConsults | Costs and potential | Weak |”

In the paper

“Similarly, the impact of eConsult services on specialists, although deemed fairly positive by a few articles, is another understudied area in need of further attention, especially since there is a potential for increases in workload, which could affect quality of care.30”

What the sources say

• ?Not checked — no full text of the sourcecites [30] 30. Keely E, Drosinis P, Afkham A, Liddy C. Perspectives of Champlain BASE specialist physicians: Their motivation, experiences and recommendations for providing eConsultations to primary care providers. Stud Tech Health Inform 2015;209:38-45. source

In the paper

“A majority of studies assessed patient experience of care from a PCP perspective, which points to a need to measure patient experience directly.”

In the paper

“| Pecina 19 | 2016 | United States | Multispecialty | Retrospective | 5,115 eConsults | Reason for face-to-face | Weak |”

In the paper

“| Datta 14 | 2015 | United States | Single specialty | Randomized clinical | 196 Patients | Total and per participant | Weak |”

In the paper

“| Shehata 28 | 2016 | Canada | Multispecialty | Retrospective | 394 eConsults 151 | Avoidance of traditional | Weak |”

In the paper

“| Liddy 31 | 2016 | Canada | Multispecialty | Mixed methods | 4,260 eConsults | eConsult patterns and | Weak |”

In the paper

“| Marcolino 33 | 2016 | Brazil | Multispecialty | Ecological study | 73,698 | Provider satisfaction, | Weak |”

In the paper

“| Alkmim 35 | 2015 | Brazil | Multispecialty | Randomized | 640 Teleconsultations Objectivity, courtesy, | Weak | |”

In the paper

“| Nelson 49 | 2016 | United States | Single specialty | Prospective study | 196 Teledermatology | Impact on diagnosis and | Weak |”

In the paper

“| Johnston 25 | 2017 | Canada | Multispecialty | Cross-sectional | 85 eConsults | Provider satisfaction, | Weak |”

In the paper

“| Liddy 21 | 2015 | Canada | Multispecialty | Mixed-methods | 2,052 eConsults | Satisfaction, education, | Weak |”

In the paper

“| Keely 30 | 2015 | Canada | Multispecialty | Survey | 34 Specialists | Provider experience/ | Weak |”

In the paper

“In contrast to the previous review, we have found an increased international presence of eConsult and noted that the breadth of specialty services offered has greatly expanded beyond dermatology, with most studies (84%) focused on multispecialty and only a handful of studies addressing a single specialty service.”

In the paper

“| North 10 | 2015 | United States | Multispecialty | Retrospective review 3,008 eConsults, 2,885 | Conversion of eConsults to | Weak |”

In the paper

“| Barnett 44 | 2017 | United States | Multispecialty | Retrospective | 3,060 PCPs | Access, patterns of use, | Weak |”

In the paper

“C.L. and I.M. performed a second abstract screen to further eliminate (or include) any articles for full screening.”

In the paper

“A total of 43 studies were used for data extraction after limiting publication dates to 2014-present, excluding articles not published in English or French, and removing duplicates and ineligible articles (Fig. 1).”

In the paper

“Two reviewers (A.M. and N.N.) screened the abstracts for relevancy, with a third reviewer (I.M.) verifying 15 of the screened articles for rigor.”

In the paper

“In Northern Ireland, statistics from the first 6 months of 2017 revealed that 71.6% of patients waited longer than 9 weeks to see a specialist and 24.2% waited more than 52 weeks.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [3] 3. UK Department of Health. Northern Ireland waiting time statistics: Outpatient waiting times quarter ending June 2017. London: UK Department of Health, 2017. www.health-ni.gov.uk/sites/default/files/publications/health/hs-niwts- outpatient-waiting-times-q1-17-18.pdf (last accessed March 21, 2018).

In the paper

“The protocol was registered on PROSPERO.2017,CRD42017069139.”

In the paper

“On May 11, 2017 we conducted a search through OVID of Medline (from 1946), EMBASE (from 1947), and ERIC (1965 to April 2017) and through EBSCO of CINAHL.”

In the paper

“A modified version of the Effective Public Health Practice Project (EPHPP) Quality Assessment Tool for Quantitative Studies by Thomas et al. 8 was used to assess the quality of articles.5”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Thomas et al. 8

In the paper

“Full articles were then screened for relevancy by N.N. and A.M.”

In the paper

“In Canada, 56% of patients waited at least 4 weeks to see a specialist, a trend that has remained consistent since 2010 and is significantly higher than the Commonwealth Fund average of 36%.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [1] 1. Canadian Institute for Health Information. How Canada compares: Results from the Commonwealth Fund's 2016 International Health Policy Survey of Adults in 11 countries. Ottawa: CIHI, 2017. www.cihi.ca/sites/default/files/document/text- alternative-version-2016-cmwf-en-web.pdf (last accessed March 21, 2018).

In the paper

“A recent wait time report in England showed that in the first 7 months of 2017, 89.4% of patients waited up to 18 weeks to see a specialist.This result failed to meet their standard of 92% of patients waiting no more than 18 weeks for their consult.2”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [2] 2. NHS England. Statistical Press Notice: NHS referral to treatment (RTT) waiting times data August 2017. London: NHS, 2017. www.england.nhs.uk/statistics/ wp-content/uploads/sites/2/2017/06/RTT-statistical-press-notice-PDF-300K- 51515.pdf (last accessed March 21, 2018).

In the paper

“We did not exclude any articles based on the quality assessment.”

In the paper

“Similarly, the impact of eConsult services on specialists, although deemed fairly positive by a few articles, is another understudied area in need of further attention, especially since there is a potential for increases in workload, which could affect quality of care.30”

What the sources say

• ?Not checked — no full text of the sourcecites [30] 30. Keely E, Drosinis P, Afkham A, Liddy C. Perspectives of Champlain BASE specialist physicians: Their motivation, experiences and recommendations for providing eConsultations to primary care providers. Stud Tech Health Inform 2015;209:38-45. source

In the paper

“First, the paucity of RCTs reflects the early, emerging nature of this health technology.”

In the paper

“Comparing these wait times to those typically experienced with a traditional referral pathway for face-to-face visits with a specialist, which are at least 4 weeks in Canada, 1 eConsults provide a promising alternative.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [1] 1. Canadian Institute for Health Information. How Canada compares: Results from the Commonwealth Fund's 2016 International Health Policy Survey of Adults in 11 countries. Ottawa: CIHI, 2017. www.cihi.ca/sites/default/files/document/text- alternative-version-2016-cmwf-en-web.pdf (last accessed March 21, 2018).

In the paper

“eConsult services in this review were shown to be effective in terms of providing faster access to specialists advice, with short response times of the participating specialists and resulting in substantial avoidance of face-to-face referral visits.”

In the paper

“Unlike other e-health technology solutions, eConsult services show sustained use and spread, high adoption, and little pushback from the providers who use them, as evidenced by high provider satisfaction ratings.25”

What the sources say

• ?Not checked — no full text of the sourcecites [25] 25. Johnston DL, Murto K, Kurzawa J, Liddy C, Keely E, Lai L. Use of electronic consultation system to improve access to care in pediatric hematology/ oncology. J Pediatr Hematol Oncol 2017;39:e367-e369. source

In the paper

“Furthermore, reports of provider education, quality of responses/feedback, and impact on healthcare support and workload, 37 suggest the provider experience is generally positive in these domains and point out that eConsult has the potential to improve overall job satisfaction, retention in remote communities, and provide educational/continuing professional development opportunities.”

What the sources say

• ?Not checked — no full text of the sourcecites [37] 37. Thijssing L, van der Heijden JP, Melissant CF, Chavannes NH, Witkamp L, Jaspers MW. Telepulmonology and telespirometry. In: Lovis C, et al., eds. e-Health-For continuity of care. Amsterdam: IOS Press; 2014:211-215. source

In the paper

“| Shipherd 11 | 2016 | United States | Multispecialty | Feasibility program | 303 eConsults, 230 | Time spent responding to | Weak |”

In the paper

“| Thijssing 37 | 2014 | The Netherlands | Single specialty | Descriptive study | 227 Telespirometry | Referral avoidance, impact | Weak |”

In the paper

“| Tran 26 | 2016 | Canada | Multispecialty | Cross-sectional | 1,055 eConsults | Impact of question type | Weak |”

In the paper

“| Tran 32 | 2016 | Canada | Multispecialty | Cross-sectional | 180 PCPs, 464 | Provider satisfaction, | Moderate |”