CoChat
Sign Up
Cited by 605Usability and User Interface DesignMobile Health and mHealth ApplicationsHealth, Technology, Consumer Behavior

The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index.

Methods of usability testing in the development of eHealth applications: A scoping review

Open access
Maramba, Inocencio, Chatterjee, Arunangsu, Newman, Craig·2019·International Journal of Medical Informatics
DOI 10.1016/j.ijmedinf.2019.03.018
Open this paper in CoChat — free accountOpen PDF

Abstract

Background The number of eHealth applications has exponentially increased in recent years, with over 325,000 health apps now available on all major app stores. This is in addition to other eHealth applications available on other platforms such as PC software, web sites and even gaming consoles. As with other digital applications, usability is one of the key factors in the successful implementation of eHealth apps. Reviews of the literature on empirical methods of usability testing in eHealth were last published in 2015. In the context of an exponentially increasing rate of App development year on year, an updated review is warranted.Objective To identify, explore, and summarize the current methods used in the usability testing of eHealth applications.Methods A scoping review was conducted on literature available from April 2014 up to October 2017. Four databases were searched. Literature was considered for inclusion if it was (1) focused on an eHealth application (which includes websites, PC software, smartphone and tablet applications), (2) provided information about usability of the application, (3) provided empirical results of the usability testing, (4) a full or short paper (not an abstract) published in English after March 2014. We then extracted data pertaining to the usability evaluation processes described in the selected studies.Results 133 articles met the inclusion criteria. The methods used for usability testing, in decreasing order of frequency were: questionnaires (n = 105), task completion (n = 57), 'Think-Aloud' (n = 45), interviews (n = 37), heuristic testing (n = 18) and focus groups (n = 13). Majority of the studies used one (n = 45) or two (n = 46) methods of testing. The rest used a combination of three (n = 30) or four (n = 12) methods of testing usability. None of the studies used automated mechanisms to test usability. The System Usability Scale (SUS) was the most frequently used questionnaire (n = 44). The ten most frequent health conditions or diseases where eHealth apps were being evaluated for usability were the following: mental health (n = 12), cancer (n = 10), nutrition (n = 10), child health (n = 9), diabetes (n = 9), telemedicine (n = 8), cardiovascular disease (n = 6), HIV (n = 4), health information systems (n = 4) and smoking (n = 4). Further iterations of the app were reported in a minority of the studies (n = 41). The use of the 'Think-Aloud' (Pearson Chi-squared test: χ2 = 11.15, p < 0.05) and heuristic walkthrough (Pearson Chi-squared test: χ2 = 4.48, p < 0.05) were significantly associated with at least one further iteration of the app being developed.Conclusion Although there has been an exponential increase in the number of eHealth apps, the number of studies that have been published that report the results of usability testing on these apps has not increased at an equivalent rate. The number of digital health applications that publish their usability evaluation results remains only a small fraction. Questionnaires are the most prevalent method of evaluating usability in eHealth applications, which provide an overall measure of usability but do not pinpoint the problems that need to be addressed. Qualitative methods may be more useful in this regard. The use of multiple evaluation methods has increased. Automated methods such as eye tracking have not gained traction in evaluating health apps. Further research is needed into which methods are best suited for the different types of eHealth applications, according to their target users and the health conditions being addressed.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ijmedinf.2019.03.018

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 34 references

Research works printed but not linked: 3

  1. I.S.O. DIS, 2009, 9241-210: 2010, Ergon. Hum. Syst. Interact., 210
  2. Hartzler, 2016, Acceptability of a team-based mobile health (mHealth) application for lifestyle self-management, Conf. Proc. … Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf. 2016, 3277
  3. Franzese, 2017, The promise of voice: connecting drug delivery through voice-activated technology, ONdrugDelivery, 2017, 34

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Maramba, I., Chatterjee, A., & Newman, C.
Affiliations
University of Plymouth (GB)
Corresponding author
Maramba, Inocencio
ORCID
Venue
International Journal of Medical Informatics
Publisher
Elsevier BV
Published
2019-06
Volume
126
Pages
95-104
ISSN
1386-5056 (ISSN-L)
DOI
10.1016/j.ijmedinf.2019.03.018
OpenAlex
W2934166592
Cited by
602
References
34
Citations by year
2026: 76 · 2025: 76 · 2024: 121 · 2023: 103 · 2022: 113 · 2021: 77 · 2020: 28 · 2019: 8
FWCI
5.01
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Medical Informatics (publishedVersion)
  • PubMed (publishedVersion)
  • PEARL (University of Plymouth) (submittedVersion, cc-by)
Funders
  • eHealth Productivity and Innovation in Cornwall
  • Isles of Scilly
  • European Regional Development fund
  • EPIC Project Team
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
Keywords
e-health applications, usability testing, usability evaluation methods, mobile health apps, System Usability Scale, think-aloud, task completion, heuristic evaluation, questionnaire-based evaluation, digital health
MeSH terms
Smartphone, Diabetes Mellitus, Humans, Mental Health, Surveys and Questionnaires, Smoking, User-Computer Interface, Telemedicine, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review