Methodologies for assessing telemedicine: A systematic review of reviews
Open accessAbstract
Background and objectives Previous reviews have expressed concerns about the quality of telemedicine studies. There is debate about shortcomings and appropriate methodologies. The aim of this review of systematic reviews of telemedicine is to summarize methodologies used in telemedicine research, discuss knowledge gaps and recommendations and suggest methodological approaches for further research.Methods We conducted a review of systematic reviews of telemedicine according to a protocol listing explicit methods, selection criteria, data collection and quality assessment procedures. We included reviews where authors explicitly addressed and made recommendations for assessment methodologies. We did a qualitative analysis of the reviews included, sensitized by two broad methodological positions; positivist and naturalistic approaches. The analysis focused on methodologies used in the primary studies included in the reviews as reported by the review authors, and methodological recommendations made by the review authors.Results We identified 1593 titles/abstracts. We included 50 reviews that explicitly addressed assessment methodologies. One group of reviews recommended larger and more rigorously designed controlled studies to assess the impacts of telemedicine; a second group proposed standardisation of populations, and/or interventions and outcome measures to reduce heterogeneity and facilitate meta-analysis; a third group recommended combining quantitative and qualitative research methods; and others applying different naturalistic approaches including methodologies addressing mutual adaptations of services and users; politically driven action research and formative research aimed at collaboration to ensure capacity for improvement of services in natural settings.Conclusions Larger and more rigorous studies are crucial for the production of evidence of effectiveness of unambiguous telemedicine services for pre defined outcome measures. Summative methodologies acknowledging telemedicine as complex innovations and outcomes as partly contingent on values, meanings and contexts are also important. So are formative, naturalistic methodologies that acknowledge telemedicine as ongoing collaborative achievements and engage with stakeholders, including patients to produce and conceptualise new and effective telemedicine innovations.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The large heterogeneity found in the present review is not unusually large when compared with similar reviews.
“Hence, the large heterogeneity found in the present review is not unusually large when compared with similar reviews.”
From The Effectiveness of Telemedicine Solutions for the Management of Type 2 Diabetes: A Systematic Review, Meta-Analysis, and Meta-Regression (2023) · cites this paper as [285]
Cites
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · see citing works
Qualitative research methods in health technology assessment: a review of the literature.
Murphy, Dingwall, Greatbatch, Parker, & Watson (1998) · Health Technology Assessment · doi:10.3310/hta2160 · Cited by 1,107 · see citing works
What Is eHealth (3): A Systematic Review of Published Definitions
Oh, H., Jadad, A., Rizo, C., Enkin, M., Powell, J., & Pagliari, C. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e1 · Cited by 1,015 · see citing works
Griffiths, F., Lindenmeyer, A., Powell, J., Lowe, P., & Thorogood, M. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.2.e10 · Cited by 820 · cited 2× · see citing works
“[58] F.”
BackgroundInteractive Health Communication Applications for people with chronic disease
Murray, E., Burns, J., See Tai, S., Lai, R., & Nazareth, I. (2005) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004274.pub4 · Cited by 722 · see citing works
Lewin, S., Glenton, C., & Oxman, A. D. (2009) · BMJ · doi:10.1136/bmj.b3496 · Cited by 678 · see citing works
Internet-administered cognitive behavior therapy for health problems: a systematic review
Cuijpers, P., van Straten, A., & Andersson, G. (2008) · Journal of Behavioral Medicine · doi:10.1007/s10865-007-9144-1 · Cited by 555 · see citing works
Currell, R., Urquhart, C., Wainwright, P., & Lewis, R. (2000) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002098 · Cited by 463 · see citing works
Systematic review of evidence for the benefits of telemedicine
Hailey, D., Roine, R., & Ohinmaa, A. (2002) · Journal of Telemedicine and Telecare · doi:10.1258/1357633021937604 · Cited by 400 · see citing works
Internet-based physical activity interventions: a systematic review of the literature.
van den Berg, M. H., Schoones, J. W., & Vliet Vlieland, T. P. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.3.e26 · Cited by 377 · see citing works
Kaltenthaler, E., Brazier, J., De Nigris, E., Tumur, I., Ferriter, M., Beverley, C., Parry, G., Rooney, G., & Sutcliffe, P. (2006) · Health Technology Assessment · doi:10.3310/hta10330 · Cited by 357 · see citing works
Home telehealth for diabetes management: a systematic review and meta‐analysis
Polisena, J., Tran, K., Cimon, K., Hutton, B., McGill, S., & Palmer, K. (2009) · Diabetes, Obesity and Metabolism · doi:10.1111/j.1463-1326.2009.01057.x · Cited by 239 · see citing works
Clinical outcomes resulting from telemedicine interventions: a systematic review
Hersh, W. R., Helfand, M., Wallace, J., Kraemer, D., Patterson, P., Shapiro, S., & Greenlick, M. (2001) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-1-5 · Cited by 234 · see citing works
Reger, M. A., & Gahm, G. A. (2009) · Journal of Clinical Psychology · doi:10.1002/jclp.20536 · Cited by 234 · see citing works
Bewick, B. M., Trusler, K., Barkham, M., Hill, A. J., Cahill, J., & Mulhern, B. (2008) · Preventive Medicine · doi:10.1016/j.ypmed.2008.01.005 · Cited by 233 · see citing works
Griffiths, K. M., & Christensen, H. (2006) · Clinical Psychologist · doi:10.1080/13284200500378696 · Cited by 210 · see citing works
Psychotherapy mediated by remote communication technologies: a meta-analytic review
Bee, P. E., Bower, P., Lovell, K., Gilbody, S., Richards, D., Gask, L., & Roach, P. (2008) · BMC Psychiatry · doi:10.1186/1471-244x-8-60 · Cited by 189 · cited 1× · see citing works
“Bee et al suggested that future research in tele-psychotherapy might overcome existing methodological shortcomings by conducting large-scale trials that incorporated both clinical outcome and process-oriented measures [71].”
BackgroundHome telemonitoring of patients with diabetes: a systematic assessment of observed effects
Jaana, M., & Paré, G. (2007) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2006.00686.x · Cited by 178 · see citing works
Cost Comparison Between Telemonitoring and Usual Care of Heart Failure: A Systematic Review
Seto, E. (2008) · Telemedicine and e-Health · doi:10.1089/tmj.2007.0114 · Cited by 156 · cited 1× · see citing works
“need for evidence of both indirect and direct costs [34].”
BackgroundE-Therapy for Mental Health Problems: A Systematic Review
Postel, M. G., de Haan, H. A., & De Jong, C. A. J. (2008) · Telemedicine and e-Health · doi:10.1089/tmj.2007.0111 · Cited by 125 · see citing works
Sintchenko, V., Magrabi, F., & Tipper, S. (2007) · Medical Informatics and the Internet in Medicine · doi:10.1080/14639230701447701 · Cited by 47 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Effectiveness of telemedicine: A systematic review of reviews
Ekeland, A. G., Bowes, A., & Flottorp, S. (2010) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2010.08.006 · Cited by 1,162 · see citing works
Self-citationBarriers and facilitators to the implementation of eHealth services: systematic literature analysis
Schreiweis, B., Pobiruchin, M., Strotbaum, V., Suleder, J., Wiesner, M., & Bergh, B. (2019) · Journal of Medical Internet Research · doi:10.2196/14197 · Cited by 354 · see citing works
A Review of Telehealth Service Implementation Frameworks
Van Dyk, L. (2014) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph110201279 · Cited by 300 · see citing works
Telemedicine for Developing Countries
Combi, C., Pozzani, G., & Pozzi, G. (2016) · Applied Clinical Informatics · doi:10.4338/aci-2016-06-r-0089 · Cited by 286 · see citing works
Interventions to increase patient portal use in vulnerable populations: a systematic review
Grossman, L. V., Masterson Creber, R. M., Benda, N. C., Wright, D., Vawdrey, D. K., & Ancker, J. S. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz023 · Cited by 251 · see citing works
Timpel, P., Oswald, S., Schwarz, P. E. H., & Harst, L. (2020) · Journal of Medical Internet Research · doi:10.2196/16791 · Cited by 243 · see citing works
The use of telemedicine to address access and physician workforce shortages
COMMITTEE ON PEDIATRIC WORKFORCE, Rimsza, M. E., Hotaling, A. J., Keown, M. E., Marcin, J. P., Moskowitz, W. B., Sigrest, T. D., & Simon, H. K. (2015) · Pediatrics · doi:10.1542/peds.2015-1253 · Cited by 214 · see citing works
Irving, M., Stewart, R., Spallek, H., & Blinkhorn, A. (2018) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x16686776 · Cited by 182 · see citing works
Telemedicine across borders: A systematic review of factors that hinder or support implementation
Saliba, V., Legido-Quigley, H., Hallik, R., Aaviksoo, A., Car, J., & McKee, M. (2012) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2012.08.003 · Cited by 173 · see citing works
Monitoring strategies in children with asthma: a randomised controlled trial
Voorend-van Bergen, S., Vaessen-Verberne, A. A., Brackel, H. J., Landstra, A. M., van den Berg, N. J., Hop, W. C., de Jongste, J. C., Merkus, P. J., & Pijnenburg, M. W. (2015) · Thorax · doi:10.1136/thoraxjnl-2014-206161 · Cited by 66 · see citing works
Hangaard, S., Laursen, S. H., Andersen, J. D., Kronborg, T., Vestergaard, P., Hejlesen, O., & Udsen, F. W. (2023) · Journal of Diabetes Science and Technology · doi:10.1177/19322968211064633 · Cited by 55 · see citing works
“Real-World” Practical Evaluation Strategies: A Review of Telehealth Evaluation
Agboola, S., Hale, T. M., Masters, C., Kvedar, J., & Jethwani, K. (2014) · JMIR Research Protocols · doi:10.2196/resprot.3459 · Cited by 49 · see citing works
“However, current telehealth literature is largely heterogeneous consisting of many pilots and mixed quality trials with diverse outcomes, which makes it difficult to estimate pooled effects of telehealth [12].”
InfluentialTønning, M. L., Kessing, L. V., Bardram, J. E., & Faurholt-Jepsen, M. (2019) · Journal of Medical Internet Research · doi:10.2196/15362 · Cited by 40 · see citing works
“It is necessary to accept and incorporate technological developments and changes in the design of a specific RCT in line with suggestions from similar but older scientific fields as telehealth in general [64].”
BackgroundOrlandoni, P., Jukic Peladic, N., Spazzafumo, L., Venturini, C., Cola, C., Sparvoli, D., Giorgini, N., Basile, R., & Fagnani, D. (2016) · Geriatrics & Gerontology International · doi:10.1111/ggi.12551 · Cited by 36 · see citing works
Combi, C., Pozzani, G., & Pozzi, G. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.103 · Cited by 6 · see citing works
- Authors
- Ekeland, A. G., Bowes, A., & Flottorp, S.
- Affiliations
- University Hospital of North Norway (NO)
- University of Stirling (GB)
- Nasjonalt Kunnskapssenter for Helsetjenesten (NO)
- Corresponding author
- Ekeland, Anne G.
- Venue
- International Journal of Medical Informatics
- Publisher
- Elsevier BV
- Published
- 2012-01
- Volume
- 81
- Issue
- 1
- Pages
- 1-11
- ISSN
- 1386-5056 (ISSN-L)
- OpenAlex
- W1988331818
- Cited by
- 187
- References
- 21
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 3 · 2023: 6 · 2022: 16 · 2021: 16 · 2020: 18 · 2019: 12 · 2018: 12 · 2017: 19 · 2016: 14 · 2015: 19 · 2014: 24 · 2013: 13 · 2012: 5
- FWCI
- 2.07
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- International Journal of Medical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Stirling Online Research Repository (University of Stirling) (submittedVersion)
- NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
- Funders
- European Commission DG Information Society
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- telemedicine, assessment methods, methodological quality, naturalistic approach, positivist approach, formative research, action research, stakeholder engagement
- MeSH terms
- Evaluation Studies as Topic, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
