CoChat
Sign Up
Cited by 605Usability and User Interface DesignMobile Health and mHealth ApplicationsHealth, Technology, Consumer Behavior

Methods of usability testing in the development of eHealth applications: A scoping review

Open access
Maramba, Inocencio, Chatterjee, Arunangsu, Newman, Craig·2019·International Journal of Medical Informatics
DOI 10.1016/j.ijmedinf.2019.03.018
Open this paper in CoChat — free accountOpen PDF

Abstract

Background The number of eHealth applications has exponentially increased in recent years, with over 325,000 health apps now available on all major app stores. This is in addition to other eHealth applications available on other platforms such as PC software, web sites and even gaming consoles. As with other digital applications, usability is one of the key factors in the successful implementation of eHealth apps. Reviews of the literature on empirical methods of usability testing in eHealth were last published in 2015. In the context of an exponentially increasing rate of App development year on year, an updated review is warranted.Objective To identify, explore, and summarize the current methods used in the usability testing of eHealth applications.Methods A scoping review was conducted on literature available from April 2014 up to October 2017. Four databases were searched. Literature was considered for inclusion if it was (1) focused on an eHealth application (which includes websites, PC software, smartphone and tablet applications), (2) provided information about usability of the application, (3) provided empirical results of the usability testing, (4) a full or short paper (not an abstract) published in English after March 2014. We then extracted data pertaining to the usability evaluation processes described in the selected studies.Results 133 articles met the inclusion criteria. The methods used for usability testing, in decreasing order of frequency were: questionnaires (n = 105), task completion (n = 57), 'Think-Aloud' (n = 45), interviews (n = 37), heuristic testing (n = 18) and focus groups (n = 13). Majority of the studies used one (n = 45) or two (n = 46) methods of testing. The rest used a combination of three (n = 30) or four (n = 12) methods of testing usability. None of the studies used automated mechanisms to test usability. The System Usability Scale (SUS) was the most frequently used questionnaire (n = 44). The ten most frequent health conditions or diseases where eHealth apps were being evaluated for usability were the following: mental health (n = 12), cancer (n = 10), nutrition (n = 10), child health (n = 9), diabetes (n = 9), telemedicine (n = 8), cardiovascular disease (n = 6), HIV (n = 4), health information systems (n = 4) and smoking (n = 4). Further iterations of the app were reported in a minority of the studies (n = 41). The use of the 'Think-Aloud' (Pearson Chi-squared test: χ2 = 11.15, p < 0.05) and heuristic walkthrough (Pearson Chi-squared test: χ2 = 4.48, p < 0.05) were significantly associated with at least one further iteration of the app being developed.Conclusion Although there has been an exponential increase in the number of eHealth apps, the number of studies that have been published that report the results of usability testing on these apps has not increased at an equivalent rate. The number of digital health applications that publish their usability evaluation results remains only a small fraction. Questionnaires are the most prevalent method of evaluating usability in eHealth applications, which provide an overall measure of usability but do not pinpoint the problems that need to be addressed. Qualitative methods may be more useful in this regard. The use of multiple evaluation methods has increased. Automated methods such as eye tracking have not gained traction in evaluating health apps. Further research is needed into which methods are best suited for the different types of eHealth applications, according to their target users and the health conditions being addressed.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ijmedinf.2019.03.018

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SUS was the most frequently used questionnaire for evaluating the usability of DHAs according to a 2019 scoping review.

    “According to a scoping review from 2019 [10], SUS was the most frequently used questionnaire for evaluating the usability of DHAs.”

    From System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis (2022) · cites this paper as [10]

  • The mHealth App Usability Questionnaire (MAUQ) is a validated alternative to SUS for measuring usability that is tailored to mobile health apps.

    “The mHealth App Usability Questionnaire (MAUQ) is a validated alternative to SUS for measuring usability that is tailored to mobile health (mHealth) apps [10].”

    From System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis (2022) · cites this paper as [10]

    The Abstract reports that the System Usability Scale (SUS) was the most frequently used questionnaire, but it does not discuss MAUQ. Searches of our copy for “MAUQ” and for a validated mobile-health alternative to SUS found no passage addressing MAUQ or its validation. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

  • Scores above 68 (SD 12.5) indicate above-average usability.

    “Scores above 68 (SD 12.5) indicate above-average usability [82].”

    From Development and usability testing of a web-based and therapist-assisted coping skills program for managing psychosocial problems in individuals with hand and upper limb injuries: Mixed methods study (2020) · cites this paper as [82]

    The Key findings section says that some studies reported scores using the System Usability Scale, but gives no cutoff for above-average usability. Searches for “68” and “12.5” found no passage establishing the claimed threshold or standard deviation. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 34 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Maramba, I., Chatterjee, A., & Newman, C.
Affiliations
University of Plymouth (GB)
Corresponding author
Maramba, Inocencio
ORCID
Venue
International Journal of Medical Informatics
Publisher
Elsevier BV
Published
2019-06
Volume
126
Pages
95-104
ISSN
1386-5056 (ISSN-L)
DOI
10.1016/j.ijmedinf.2019.03.018
OpenAlex
W2934166592
Cited by
602
References
34
Citations by year
2026: 76 · 2025: 76 · 2024: 121 · 2023: 103 · 2022: 113 · 2021: 77 · 2020: 28 · 2019: 8
FWCI
5.01
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Medical Informatics (publishedVersion)
  • PubMed (publishedVersion)
  • PEARL (University of Plymouth) (submittedVersion, cc-by)
Funders
  • eHealth Productivity and Innovation in Cornwall
  • Isles of Scilly
  • European Regional Development fund
  • EPIC Project Team
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
Keywords
e-health applications, usability testing, usability evaluation methods, mobile health apps, System Usability Scale, think-aloud, task completion, heuristic evaluation, questionnaire-based evaluation, digital health
MeSH terms
Smartphone, Diabetes Mellitus, Humans, Mental Health, Surveys and Questionnaires, Smoking, User-Computer Interface, Telemedicine, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review