CoChat

System Usability Scale Benchmarking for Digital Health Apps: Meta-analysis

Open access

The DOI matches its Crossref record; 32 of the 33 references it cites were found in a registry or an index; 8 of the 17 sources its claims cite were checked for support.

Abstract

Background: The System Usability Scale (SUS) is a widely used scale that has been used to quantify the usability of many software and hardware products. However, the SUS was not specifically designed to evaluate mobile apps, or in particular digital health apps (DHAs). Objective: The aim of this study was to examine whether the widely used SUS distribution for benchmarking (mean 68, SD 12.5) can be used to reliably assess the usability of DHAs. Methods: A search of the literature was performed using the ACM Digital Library, IEEE Xplore, CORE, PubMed, and Google Scholar databases to identify SUS scores related to the usability of DHAs for meta-analysis. This study included papers that published the SUS scores of the evaluated DHAs from 2011 to 2021 to get a 10-year representation. In total, 117 SUS scores for 114 DHAs were identified. R Studio and the R programming language were used to model the DHA SUS distribution, with a 1-sample, 2-tailed t test used to compare this distribution with the standard SUS distribution. Results: The mean SUS score when all the collected apps were included was 76.64 (SD 15.12); however, this distribution exhibited asymmetrical skewness (–0.52) and was not normally distributed according to Shapiro-Wilk test (P=.002). The mean SUS score for “physical activity” apps was 83.28 (SD 12.39) and drove the skewness. Hence, the mean SUS score for all collected apps excluding “physical activity” apps was 68.05 (SD 14.05). A 1-sample, 2-tailed t test indicated that this health app SUS distribution was not statistically significantly different from the standard SUS distribution (P=.98). Conclusions: This study concludes that the SUS and the widely accepted benchmark of a mean SUS score of 68 (SD 12.5) are suitable for evaluating the usability of DHAs. We speculate as to why physical activity apps received higher SUS scores than expected. A template for reporting mean SUS scores to facilitate meta-analysis is proposed, together with future work that could be done to further examine the SUS benchmark scores for DHAs.

Overview

CoChat Summary

TL;DR

This meta-analysis pooled 117 System Usability Scale (SUS) scores from 114 digital health apps (DHAs) to test whether the standard SUS benchmark of mean 68 (SD 12.5) applies to DHAs. It found the benchmark fits DHAs once physical activity apps are excluded, while physical activity apps scored substantially higher and skewed the overall distribution.

Key findings

  • Across all collected apps, the mean SUS score was 76.64 (SD 15.12); the distribution was asymmetrical (skewness –0.52) and not normal by Shapiro-Wilk test (P=.002).
  • Physical activity apps had a mean SUS score of 83.28 (SD 12.39) and accounted for 66 of 117 scores (56.4%); excluding them yielded a mean of 68.05 (SD 14.05), which was not statistically significantly different from the standard SUS distribution (mean 68, SD 12.5) by 1-sample, 2-tailed t test (P=.98).
  • Category means were 71.30 (SD 12.72) for health care, 61.29 (SD 15.08) for first aid/CPR/choking, 71.06 (SD 14.55) for diet/food/nutrition, and 69.45 (SD 5.30) for health information.
  • The probability of a SUS score ≤68 was 28.39% for all categories, 10.88% for physical activity apps only, and 49.85% for all categories excluding physical activity apps, compared with the standard 50%.

Methods

  • The authors searched ACM Digital Library, IEEE Xplore, CORE, PubMed, and Google Scholar for papers published from 2011 to 2021 that reported SUS scores for DHAs evaluated by end users, collecting 117 SUS scores from 114 DHAs across 19 papers.
  • Data were separated into three subsets: all DHAs, physical activity apps only, and all DHAs excluding physical activity apps. Analyses in R 4.0.3 included Shapiro-Wilk normality tests, skewness and kurtosis, density plots, histograms, boxplots, 1-sample 2-tailed t tests, and Wilcoxon signed rank tests against the standard SUS distribution of mean 68 (SD 12.5).
  • Extracted variables included first author, DHA health area, app name, device, platform, sample size used for the mean SUS score, publication year, and SUS score.

Limitations

  • The paper states the dataset is small (n=117 SUS scores), came from 19 papers, and some mean SUS scores were based on as few as 2 or 3 reviewers; 65 physical activity apps came from the same paper, which used 2 reviewers per app, raising the possibility of inflated scores or limited evaluation time.
  • The paper also states possible publication bias/file drawer effect, overrepresentation of particular DHA genres, language and cultural bias when SUS is used with non-English speakers, and the need for genre- or device-specific benchmarks.
  • Evident from the text: participant sample sizes used to collect SUS scores had a distribution of 6 (SD 6.16; range 2-31), and physical activity apps dominated the dataset (66/117), limiting generalizability of the pooled distribution.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/37290

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Analysis of SUS distribution for all categories of digital health apps: A) histogram of SUS scores, B) density plot of SUS scores, C) boxplot of SUS scores, and D) normal curve probabilities of SUS scores for all categories (mean 76.64, SD 15.12; shaded area: 0.2839). Blue line=68 (average SUS score for apps), red line=78.75 (median), orange line=76.64 (mean). SUS: System Usability Score.
Figure 1 Analysis of SUS distribution for all categories of digital health apps: A) histogram of SUS scores, B) density plot of SUS scores, C) boxplot of SUS scores, and D) normal curve probabilities of SUS scores for all categories (mean 76.64, SD 15.12; shaded area: 0.2839). Blue line=68 (average SUS score for apps), red line=78.75 (median), orange line=76.64 (mean). SUS: System Usability Score.CC BY · PubMed Central
Figure 2 Analysis of SUS distribution for physical activity apps only: A) histogram of SUS scores, B) density plot of SUS scores, C) boxplot of SUS scores, and D) normal curve probabilities of SUS scores for all categories (mean 83.28, SD 12.39; shaded area: 0.1088). Blue line=68 (average SUS score for apps), red line=86 (median), orange line=83.28 (mean). SUS: System Usability Score.
Figure 2 Analysis of SUS distribution for physical activity apps only: A) histogram of SUS scores, B) density plot of SUS scores, C) boxplot of SUS scores, and D) normal curve probabilities of SUS scores for all categories (mean 83.28, SD 12.39; shaded area: 0.1088). Blue line=68 (average SUS score for apps), red line=86 (median), orange line=83.28 (mean). SUS: System Usability Score.CC BY · PubMed Central
Figure 3 Analysis of SUS distribution for all categories excluding physical activity apps: A) histogram of SUS scores, B) density plot of SUS scores, C) boxplot of SUS scores, and D) normal curve probabilities of SUS scores for all categories (mean 68.05, SD 14.05; shaded area: 0.4985). Blue line=68 (average SUS score for apps), red line=68.30 (median), orange line=68.05 (mean). SUS: System Usability Score.
Figure 3 Analysis of SUS distribution for all categories excluding physical activity apps: A) histogram of SUS scores, B) density plot of SUS scores, C) boxplot of SUS scores, and D) normal curve probabilities of SUS scores for all categories (mean 68.05, SD 14.05; shaded area: 0.4985). Blue line=68 (average SUS score for apps), red line=68.30 (median), orange line=68.05 (mean). SUS: System Usability Score.CC BY · PubMed Central

Verification

Claims the paper makes

7 Valid · 1 Not valid · 9 Not checked · 44 Own results and interpretations

30 more claims

Create a free account to see all 61 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 17
  • Probably not a substantial claim. 13

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 32 references

Research works printed but not linked: 5

  1. BrookeJSUS: a quick and dirty usability scaleResearchGate1995112021-11-21https://www.researchgate.net/publication/228593520_SUS_A_quick_and_dirty_usability_scale
  2. Sauro, J, 2016, Quantifying the User Experience: Practical Statistics for User Research. 2nd ed
  3. Brooke, J, 2013, J Usability Stud, 8, 29
  4. Kizakevich, Paul N, 2014, Stud Health Technol Inform, 199, 35
  5. Jon, Yablonski, 2020, Jakob's Law

Cited by

Items 301–400 of 516

Metadata

Authors
Hyzy, M., Bond, R., Mulvenna, M., Bai, L., Dix, A., Leigh, S., & Hunt, S.
Affiliations
  • University of Ulster (GB)
  • Swansea University (GB)
  • University of Warwick (GB)
Corresponding author
Hyzy, Maciej
Venue
JMIR mHealth and uHealth
Published
2022-08-18
Volume
10
Issue
8
Pages
e37290
ISSN
2291-5222 (ISSN-L)
OpenAlex
W4287886354
Cited by
510
References
33
Citations by year
2026: 202 · 2025: 161 · 2024: 91 · 2023: 51 · 2022: 5
FWCI
7.96
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR mhealth and uhealth (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Ulster University Research Portal (Ulster University) (submittedVersion, cc-by)
Funders
Department for the Economy
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
Keywords
System Usability Scale, digital health apps, physical activity apps, skewness, usability evaluation, report templates
MeSH terms
Humans, Telemedicine, Benchmarking, Mobile Applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.