CoChat
Sign Up
Cited by 92Digital Mental Health InterventionsUsability and User Interface DesignAnxiety, Depression, Psychometrics, Treatment, Cognitive Processes

Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis

Open access
Mol, Mayke, van Schaik, Anneke, Dozeman, Els, Ruwaard, Jeroen, Vis, Christiaan, Ebert, David D., Etzelmueller, Anne, Mathiasen, Kim +14 more·2020·BMC Psychiatry
DOI 10.1186/s12888-020-02627-8PMC PMC7216472
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: The System Usability Scale (SUS) is used to measure usability of internet-based Cognitive Behavioural Therapy (iCBT). However, whether the SUS is a valid instrument to measure usability in this context is unclear. The aim of this study is to assess the factor structure of the SUS, measuring usability of iCBT for depression in a sample of professionals. In addition, the psychometric properties (reliability, convergent validity) of the SUS were tested. Methods: A sample of 242 professionals using iCBT for depression from 6 European countries completed the SUS. Confirmatory Factor Analysis (CFA) was conducted to test whether a one-factor, two-factor, tone-model or bi-direct model would fit the data best. Reliability was assessed using complementary statistical indices (e.g. omega). To assess convergent validity, the SUS total score was correlated with an adapted Client Satisfaction Questionnaire (CSQ-3). Results: CFA supported the one-factor, two-factor and tone-model, but the bi-factor model fitted the data best (Comparative Fit Index = 0.992, Tucker Lewis Index = 0.985, Root Mean Square Error of Approximation = 0.055, Standardized Root Mean Square Residual = 0.042 (respectively χ 2 diff (9) = 69.82, p < 0.001; χ 2 diff (8) = 33.04, p < 0.001). Reliability of the SUS was good (ω = 0.91). The total SUS score correlated moderately with the CSQ-3 (CSQ1 r s = .49, p < 0.001; CSQ2 r s = .46, p < 0.001; CSQ3 r s = .38, p < 0.001), indicating convergent validity. Conclusions: Although the SUS seems to have a multidimensional structure, the best model showed that the total sumscore of the SUS appears to be a valid and interpretable measure to assess the usability of internet-based interventions when used by professionals in mental healthcare.

AI snapshot

Written from the paper’s full text.

TL;DR

This confirmatory factor analysis tested the dimensionality and psychometric properties of the System Usability Scale (SUS) among 242 professionals using internet-based CBT for depression across six European countries. A bi-factor model fit best, but the SUS total score was reliable and showed convergent validity, supporting use of the total sumscore as a valid usability measure in this context.

Key findings

  • The bi-factor model fit the data best: CFI = 0.992, TLI = 0.985, RMSEA = 0.055, SRMR = 0.042, and it significantly outperformed the one-factor, two-factor, and tone models (e.g., one-factor χ²diff(9) = 69.82, p < 0.001; two-factor χ²diff(8) = 33.04, p < 0.001).
  • Reliability of the 10 SUS items was good: ω = 0.91. Subscale analyses indicated the Usability and Learnability subscales did not add value over the total score (PRMSEtotal = 0.93; PRMSEusability = 0.70; PRMSElearnability = 0.55).
  • Convergent validity was supported by moderate correlations between the SUS total score and the adapted CSQ-3 items: CSQ1 rs = .49, CSQ2 rs = .46, CSQ3 rs = .38, all p < 0.001.
  • The pooled mean SUS total score was 67.9 (SD 16.3; range 20–100), described as just below average.

Methods

  • A sample of 242 professionals providing guided or blended iCBT to patients with depressive symptoms in the Netherlands (n = 51), Germany (n = 16), Denmark (n = 4), Spain (n = 135), Italy (n = 33), and Turkey (n = 3) completed the SUS and an adapted CSQ-3.
  • Confirmatory factor analysis compared four models: one-factor, two-factor (Usability/Learnability), tone model (positive/negative), and bi-factor, using WLSMV estimation for ordinal data.
  • Reliability was assessed with omega and related indices; convergent validity was assessed with Spearman rank-order correlations between the SUS total score and CSQ-3 items.

Limitations

  • The paper states that the sample had an uneven distribution of professionals per country, causing under- and over-representation, limiting country-specific factor structure analysis and possibly biasing standard errors; iCBT technical formats and content also differed between countries.
  • The paper states that most SUS translations were informal, so different interpretations by professionals may have occurred, though back-translation was used to minimize this risk.
  • The paper states that the mixed tone of negatively and positively worded items may distort the SUS factor structure, and notes prior evidence that mixed tone can cause respondent mistakes and miscoding.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12888-020-02627-8

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 45 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The total SUS score correlated moderately with the CSQ-3 (CSQ1 rs = .49, p < 0.001; CSQ2 rs = .46, p < 0.001; CSQ3 rs = .38, p < 0.001).

    “The total SUS score correlated moderately with the CSQ-3 (CSQ1 rs = .49, p < 0.001; CSQ2 rs = .46, p < 0.001; CSQ3 rs = .38, p < 0.001), indicating convergent validity.”
    Quote in the paperSubstantial claim
  • Findingown data

    The SUS total score correlated with CSQ1 at rs = 0.49 (95% CI 0.39 to 0.58).

    “CSQ1. Have the needs been met? | 0.49 (0.39 to 0.58)a”
    Quote in the paperSubstantial claim
  • Findingown data

    A sample of 242 professionals using iCBT for depression from 6 European countries completed the SUS.

    “A sample of 242 professionals using iCBT for depression from 6 European countries completed the SUS.”
    • Also stated in: § Recruitment · plain
      “For the purpose of this study, the data provided by the professionals was used (n = 242).”
    Quote in the paperSubstantial claim
  • Findingown data

    The Usability and Learnability factors in the bi-factor model correlated positively (rs = 0.70).

    “Fig. 2 shows a visualization of the bi-factor model, with a positive correlation (rs = 0.70) between the factors Usability and Learnability.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Reliability of the SUS was good (ω = 0.91).

    “Reliability of the SUS was good (ω = 0.91).”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 44
  • May combine two claims. 22

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    About half of the respondents (54.5%) had more than 5 years of professional experience in mental healthcare.

    “About half of the respondents had more than 5 years of professional experience in mental healthcare (54.5%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The PUC was 0.36, the ECV was 0.75, and omega hierarchical was 0.78.

    “The PUC was 0.36, the ECV was 0.75 and omega hierarchical was 0.78, meaning that the SUS is not entirely convincingly unidimensional and at the same time the presence of the two subscales (Usability/Learnability) not serious enough is to disqualify undimensionality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The respondents (n = 242) completed the questionnaire between August and December 2016.

    “The respondents (n = 242) completed the questionnaire between August and December 2016.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Among the respondents, 39.3% were GPs, 34.7% were psychologists or psychiatrists and 24.8% had a different professional background.

    “39.3% of the respondents were GP’s, 34.7% were psychologists or psychiatrists and 24.8% had a different professional background (e.g. specialized health nurses, health workers).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Confirmatory factor analysis supported the one-factor, two-factor and tone-model.

    “CFA supported the one-factor, two-factor and tone-model, but the bi-factor model fitted the data best”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The majority of respondents (57%) had provided iCBT at least five times to treat depressed patients.

    “The majority of respondents (57%) had provided iCBT at least five times to treat depressed patients.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The PRMSE values were: total = 0.93, usability = 0.70, learnability = 0.55.

    “This was made clear by the PRMSE results of the total (PRMSEtotal = 0.93 and subscale scores (PRMSEusability = 0.70, PRMSE learnability = 0.55): both PRMSE ratio values exceed 1 confirming that the subscales do not have added value over the total score.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A relatively large group (36.4%) had little experience with iCBT, defined as providing iCBT less than 5 times to patients.

    “Still, a relatively large group (36.4%) had little experience with iCBT (i.e. providing iCBT less than 5 times to patients).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The professionals provided guided and blended iCBT to patients with depressive symptoms in the Netherlands (n = 51), Germany (n = 16), Denmark (n = 4), Spain (n = 135), Italy (n = 33) and Turkey (n = 3).

    “They provided guided and blended iCBT to patients with depressive symptoms in the Netherlands (n = 51), Germany (n = 16), Denmark (n = 4), Spain (n = 135), Italy (n = 33) and Turkey (n = 3).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The one-factor, two-factor, and tone models had acceptable fit according to CFI ≥ 0.95, TLI ≥ 0.95, and SRMR ≤ 0.08.

    “The overall goodness-of-fit indices suggested that the one-factor, the two-factor and tone models have an acceptable fit (i.e. CFI ≥ 0.95, TLI ≥ 0.95, SRMR ≤0.08).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The amount of iCBT experience among professionals in the sample varied considerably.

    “In our sample, the amount of iCBT experience among the professionals varied considerably.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Satisfaction is related to system usability, either contributing to the level of usability or being a consequence of usability.

    “Satisfaction is related to system usability in the sense that satisfaction can contribute to the level of usability or where satisfaction is a consequence of usability [12].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Tractinsky N. The usability construct: a dead end? Hum Comput Interact. 2018;33:131–77. https://doi.org/10.1080/07370024.2017.1298038. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The SUS provides a single score for usability and is designed as a unidimensional (one factor) measurement.

    “The SUS provides a single score for usability and is designed as a unidimensional (one factor) measurement [13].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] Brooke J. SUS-A quick and dirty usability scale. Usability Eval Ind. 1996;189:4–7.

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The System Usability Scale (SUS) has increasingly been applied to measure usability of internet-based cognitive behavioural therapy (iCBT) in feasibility and evaluation studies in mental healthcare.

    “In feasibility and evaluation studies on iCBT in mental healthcare, the System Usability Scale (SUS) has increasingly been applied to measure usability [7–9]”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [7] Kooistra LC, Ruwaard J, Wiersma JE, van Oppen P, van der Vaart R, van Gemert-Pijnen JEWC, et al. Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interv. 2016;4:61–71. https://doi.org/10.1016/j.invent.2016.01.003. source

      Not checked in the free run.

    • cites [8] Wozney L, Baxter P, Newton AS. Usability evaluation with mental health professionals and young people to develop an internet-based cognitive-behaviour therapy program for adolescents with anxiety disorders. BMC Pediatr. 2015;15:213. https://doi.org/10.1186/s12887-015-0534-1. source

      Not checked in the free run.

    • cites [9] Kleiboer A, Smit J, Bosmans J, Ruwaard J, Andersson G, Topooco N, et al. European COMPARative effectiveness research on blended depression treatment versus treatment-as-usual (E-COMPARED): study protocol for a randomized controlled, non-inferiority trial in eight European countries. Trials. 2016;17. https://doi.org/10.1186/s13063-016-1511-1. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several studies encountered an unintentional SUS factor structure caused by the mixed tone.

    “However, several studies encountered an unintentional SUS factor structure caused by the mixed tone (e.g. ) [24].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Kortum P, Sorber M. Measuring the usability of Mobile applications for phones and tablets. Int J Hum Comput Interact. 2015;31:518–29. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    When PUC is greater than .70 and ECV greater than .70, relative bias will be slight and the common variance can be regarded as essentially unidimensional.

    “When PUC is greater than .70 and ECV greater than .70 relative bias will be slight and the common variance can be regarded as essentially unidimensional [39].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] Rodriguez A, Reise SP, Haviland MG. Applying Bifactor statistical indices in the evaluation of psychological measures. J Pers Assess. 2016;98:223–37. https://doi.org/10.1080/00223891.2015.1089249. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    The analysis of the factor structure was limited in accounting for different countries and possible biased standard error.

    “This also limited the analysis of the factor structure in taking into account the different countries and the possible biased standard error.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The CSQ shows good reliability and validity and is used across a range of services, from inpatient to forensic services, without a specific setting of care, and in internet-based interventions.

    “The CSQ shows good reliability and validity and is used across a range of services, from inpatient to forensic services, without a specific setting of care [31] and internet-based interventions [32].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [31] Miglietta E, Belessiotis-Richards C, Ruggeri M, Priebe S. Scales for assessing patient satisfaction with mental health care: a systematic review. J Psychiatr Res. 2018;100:33–46. https://doi.org/10.1016/j.jpsychires.2018.02.014. source

      Not checked in the free run.

    • cites [32] Boβ L, Lehr D, Reis D, Vis C, Riper H, Berking M, et al. Reliability and validity of assessing user satisfaction with web-based health interventions. J Med Internet Res. 2016;18. https://doi.org/10.2196/jmir.5952. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Lewis and Sauro concluded to treat the SUS as a single factor structure.

    “As the tone structure is of little practical and theoretical interest, their conclusion was to treat SUS as a single factor structure.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] Lewis JR, Sauro J. Revisiting the factor structure of the system usability scale. J Usability Stud. 2017;12:183–92.

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    There is no cut-off point for omega to evaluate acceptable reliability; a minimum of .50 and values closer to .75 are recommended for satisfactory and good reliability.

    “There is no cut-off point for omega to evaluate acceptable reliability, a minimum of .50 and values closer to .75 are recommended for satisfactory and good reliability [40].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [40] Reise SP, Scheines R, Widaman KF, Haviland MG. Multidimensionality and structural coefficient Bias in structural equation modeling: a Bifactor perspective. Educ Psychol Meas. 2013;73:5–26. https://doi.org/10.1177/0013164412449831. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    Researchers interested in the usability of internet-based interventions in mental healthcare can use the proposed scoring of the SUS and in particular the calculated sumscore.

    “However the bi-factor model showed the best results in this sample indicating that researchers interested in the usability of internet-based interventions in mental healthcare can use the proposed scoring of the SUS and in particular the calculated sumscore.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Brown suggested a set of goodness-of-fit indices and criterion values that provide an overall satisfactory performance in evaluating models.

    “Overall model fit was assessed using a set of goodness-of-fit indices and criterion values, as suggested by Brown [36] as these indices provide an overall satisfactory performance in evaluating models”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] Brown T. Confirmatory factor analysis for applied research. New York, NY: Guilford Press; 2006.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The mixed tone was originally used to control for acquiescence bias.

    “The mixed tone was originally used to control for acquiescence bias; the hypothesized tendency of respondents to agree with statements with a mix of positive and negative tone [19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] Lewis JR, Sauro J. Item benchmarks for the system usability scale. J Usability Stud. 2018;13:158–67.

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The fit indices should be considered in combination, as a good fit meets all the chosen criteria.

    “These fit indices were considered in combination, as a good fit meets all the chosen criteria [37].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [37] Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model A Multidiscip J. 1999;6:1–55. https://doi.org/10.1080/10705519909540118. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The SUS is without costs, technology agnostic, brief, reliable and valid.

    “In addition, it is without costs, technology agnostic, brief, reliable and valid [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] Lewis JR. The system usability scale: past, present, and future. Int J Hum Comput Interact. 2018;34:577–90. https://doi.org/10.1080/10447318.2018.1455307. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In countries where no translation to the local language was available, the forward and backward method was followed to translate the SUS items.

    “In the countries where no translation to the local language was available, the forward and backward method was followed to translate the SUS items”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Studies have also replicated the one-factor model of the SUS.

    “and the one-factor model as well [24, 25].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [24] Kortum P, Sorber M. Measuring the usability of Mobile applications for phones and tablets. Int J Hum Comput Interact. 2015;31:518–29. source

      Not checked in the free run.

    • cites [25] Lewis JR, Brown J, Mayes DK. Psychometric evaluation of the EMO and the SUS in the context of a large-sample Unmoderated usability study. Int J Hum Comput Interact. 2015;31:545–53. https://doi.org/10.1080/10447318.2015.1064665. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    If the ratio of PRMSE based on subscale scores over total score exceeds one, the subscore does not have added value and it is not recommended to use the subscore in statistical models.

    “If the ratio of these values, subscore over total score, exceed one, the subscore does not have added value and it is not recommended to use the subscore in statistical models.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [41] Haberman SJ. When can subscores have value? J Educ Behav Stat. 2008;33:204–29. https://doi.org/10.3102/1076998607302636. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Most translations of the original SUS items were informal, so possible different interpretations by professionals may have occurred.

    “Secondly, as most translations of the original SUS items were informal, possible different interpretations by the professionals may have occurred.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Studies have replicated inconsistent findings pointing towards the two-factor model of the SUS.

    “Since then, studies replicated inconsistent findings pointing towards this two-factor model (e.g. ) [23]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [23] Borsci S, Federici S, Lauriola M. On the dimensionality of the system usability scale: a test of alternative measurement models. Cogn Process. 2009;10:193–7. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    The Weighted Least Squares Mean and Variance adjusted (WLSMV) estimator is recommended for the analysis of ordinal data.

    “Hence, the Weighted Least Squares Mean and Variance adjusted (WLSMV) estimator was used as a method of parameter estimation as this is recommended for the analysis of ordinal data [36].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [36] Brown T. Confirmatory factor analysis for applied research. New York, NY: Guilford Press; 2006.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Sample sizes between 200 and 500 are recommended for CFA with ordinal data.

    “In which sample sizes between 200 and 500 are recommended.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [44] Bandalos DL. Relative performance of categorical diagonally weighted least squares and robust maximum likelihood estimation. Struct Equ Model Multidiscip J. 2014;21:102–16. https://doi.org/10.1080/10705511.2014.859510. source

      Not checked in the free run.

    • cites [45] Forero CG, Maydeu-Olivares A, Gallardo-Pujol D. Factor analysis with ordinal indicators: a Monte Carlo study comparing DWLS and ULS estimation. Struct Equ Model. 2009;16:625–41. https://doi.org/10.1080/10705510903203573. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The SUS has sensitivity.

    “and sensitivity (e.g. ) [14]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Lewis JR, Sauro J. The factor structure of the system usability scale. Hum Centered Design. 2009:94–103. https://doi.org/10.1007/978-3-642-02806-9_12. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    Data from Scotland, Wales, Estonia and Norway were not suitable for the purpose of this study because these countries evaluated unguided iCBT and the professionals' interaction with the interface of iCBT was very minimal.

    “Data from the other MasterMind countries (Scotland, Wales, Estonia and Norway) were not suitable for the purpose of this study, as these countries evaluated unguided iCBT and the professionals’ interaction with the interface of iCBT was very minimal.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Usability of internet-based interventions is an important factor in successful implementation and patient engagement.

    “Usability of internet-based interventions are an important factor in successful implementation and patient engagement [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Vis C, Mol M, Kleiboer A, Bührmann L, Finch T, Smit J, et al. Improving implementation of emental health for mood disorders in routine practice: systematic review of barriers and facilitating factors. JMIR Mental Health. 2018;20. https://doi.org/10.2196/mental.9769. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The factor structure of the SUS may be distorted by the mixed tone of negatively and positively worded items.

    “It is possible that the factor structure of the SUS is distorted by the mixed tone of negatively and positively worded items.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The technical formats and content of the iCBT interventions differed between countries.

    “Plus, the technical formats of the iCBT applications as well as the content of the iCBT interventions differed between countries.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The risk of different interpretations due to informal translations was minimized by using a back-translation method.

    “On the other hand, this risk was minimized by using a back-translation method [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Brislin RW. Back-translation for cross-cultural research. J Cross-Cult Psychol. 1970;1:185–216. https://doi.org/10.1177/135910457000100301. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    More recent research showed that the two-factor structure of the SUS possibly depends on the amount of experience that users have with a given product.

    “More recent research showed that two-factor structure possibly depends on the amount of experience that users have with a given product [18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Borsci S, Federici S, Bacci S, Gnaldi M, Bartolucci F. Assessing user satisfaction in the era of user experience: comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience. Int J Hum Comput Interact. 2015;31:484–95. https://doi.org/10.1080/10447318.2015.1064648. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Although the SUS seems to have a multidimensional structure, the best model showed that the total sumscore of the SUS appears to be a valid and interpretable measure to assess the usability of internet-based interventions when used by professionals in mental healthcare.

    “Although the SUS seems to have a multidimensional structure, the best model showed that the total sumscore of the SUS appears to be a valid and interpretable measure to assess the usability of internet-based interventions when used by professionals in mental healthcare.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The sample of 242 professionals from six European countries is representative of professionals using iCBT for depression in mental healthcare.
    • Rests on an assumption: The forward and backward translation method ensures equivalence of the SUS items across languages.
    • Rests on an assumption: The factor structure of the SUS is equivalent across the six European countries from which data were pooled.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In 2017, Lewis and Sauro found a different two-factor structure of the SUS produced by the positive and negative tone of the items.

    “In 2017, Lewis and Sauro revisited the factor structure of the SUS and tried to replicate the two-factor structure [26]. However, they found a different two-factor structure produced by the positive and negative tone of the items.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] Lewis JR, Sauro J. Revisiting the factor structure of the system usability scale. J Usability Stud. 2017;12:183–92.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The SUS has acceptable levels of convergent validity with other measures of perceived usability.

    “acceptable levels of convergent validity with other measures of perceived usability (e.g. ) [18]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Borsci S, Federici S, Bacci S, Gnaldi M, Bartolucci F. Assessing user satisfaction in the era of user experience: comparison of the SUS, UMUX, and UMUX-LITE as a function of product experience. Int J Hum Comput Interact. 2015;31:484–95. https://doi.org/10.1080/10447318.2015.1064648. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The SUS is not entirely convincingly unidimensional, but the presence of the two subscales (Usability/Learnability) is not serious enough to disqualify unidimensionality.

    “The PUC was 0.36, the ECV was 0.75 and omega hierarchical was 0.78, meaning that the SUS is not entirely convincingly unidimensional and at the same time the presence of the two subscales (Usability/Learnability) not serious enough is to disqualify undimensionality.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Lewis and Sauro proposed that there might be two factors in the SUS: Usability and Learnability.

    “As for the factor structure, Lewis and Sauro [14] proposed that there might be two factors in the SUS: Usability and Learnability.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Lewis JR, Sauro J. The factor structure of the system usability scale. Hum Centered Design. 2009:94–103. https://doi.org/10.1007/978-3-642-02806-9_12. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    A common criterion for the absolute magnitude of correlations that supports the hypothesis of convergent validity is a minimum of 0.30.

    “A common criterion for the absolute magnitude of correlations that supports the hypothesis of convergent validity is a minimum of 0.30 [42].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] Swank JM, Mullen PR. Evaluating evidence for conceptually related constructs using bivariate correlations. Meas Eval Couns Dev. 2017;50:270–4. https://doi.org/10.1080/07481756.2017.1339562. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The psychometric properties of the SUS are sufficiently studied, with reported reliabilities between 0.79 and 0.97.

    “The psychometric properties of the SUS are sufficiently studied with reported reliabilities between 0.79 and 0.97 (e.g. )[16, 17]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [16] Dianat I, Ghanbari Z, AsghariJafarabadi M. Psychometric properties of the persian language version of the system usability scale. Health Promot Perspect. 2014;4:82–9. https://doi.org/10.5681/hpp.2014.011. source

      Not checked in the free run.

    • cites [17] Finstad K. The usability metric for user experience. Interact Comput. 2010;22:323–7. https://doi.org/10.1016/j.intcom.2010.04.004. source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    When PUC is lower than .80, general ECV is greater than .60 and omega hierarchical is greater than .70, multidimensionality is not severe enough to disqualify the interpretation of the instrument as primarily unidimensional.

    “When a PUC value is lower than .80, the general ECV value greater than .60 and omega hierarchical (of the general factor) is greater than .70 it is suggested that the presence of multidimensionality is not severe enough to disqualify the interpretation of the instrument as primarily unidimensional [40].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [40] Reise SP, Scheines R, Widaman KF, Haviland MG. Multidimensionality and structural coefficient Bias in structural equation modeling: a Bifactor perspective. Educ Psychol Meas. 2013;73:5–26. https://doi.org/10.1177/0013164412449831. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Omega is a reliability estimate that does not depend on the assumption of tau equivalence unlike Cronbach's alpha.

    “Omega is a reliability estimate that does not depend on the assumption of tau equivalence unlike its classic counterpart Cronbach’s alpha [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Rodriguez A, Reise SP, Haviland MG. Applying Bifactor statistical indices in the evaluation of psychological measures. J Pers Assess. 2016;98:223–37. https://doi.org/10.1080/00223891.2015.1089249. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A large variety of product experience was also reported in studies that found a one-factor model.

    “However, a large variety of product experience was also reported in studies that found a one-factor model (e.g. ) [24, 25].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [24] Kortum P, Sorber M. Measuring the usability of Mobile applications for phones and tablets. Int J Hum Comput Interact. 2015;31:518–29. source

      Not checked in the free run.

    • cites [25] Lewis JR, Brown J, Mayes DK. Psychometric evaluation of the EMO and the SUS in the context of a large-sample Unmoderated usability study. Int J Hum Comput Interact. 2015;31:545–53. https://doi.org/10.1080/10447318.2015.1064665. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Sauro and Lewis found no evidence for strong acquiescence bias or extreme response bias.

    “In addition, they found no evidence for strong acquiescence bias or extreme response bias.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] Sauro J, Lewis JR. When designing usability questionnaires, does it hurt to be positive? In: Conference on Human Factors in Computing Systems - Proceedings; 2011. p. 2215–23. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    There is evidence that the mixed tone caused respondents to make mistakes and researchers to miscode the questionnaire.

    “Moreover, there is evidence that the mixed tone caused respondents to make mistakes and researchers to miscode the questionnaire [48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] Sauro J, Lewis JR. When designing usability questionnaires, does it hurt to be positive? In: Conference on Human Factors in Computing Systems - Proceedings; 2011. p. 2215–23. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Sauro and Lewis (2011) found no significant difference between the mean overall SUS scores of the positive and mixed versions.

    “In 2011, Sauro and Lewis tested a positive version of the SUS and found no significant difference between the mean overall SUS scores of the positive and mixed versions [48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] Sauro J, Lewis JR. When designing usability questionnaires, does it hurt to be positive? In: Conference on Human Factors in Computing Systems - Proceedings; 2011. p. 2215–23. source

      Not checked in the free run.

52 of 52

  • The paper relies on this without support.

    The SUS has not yet been validated in the emerging field of iCBT.

    “however this instrument has not yet been validated in this emerging field.”

  • The paper relies on this without support.

    The sample of 242 professionals from six European countries is representative of professionals using iCBT for depression in mental healthcare.

    “A sample of 242 professionals using iCBT for depression from 6 European countries completed the SUS.”

  • The paper relies on this without support.

    The forward and backward translation method ensures equivalence of the SUS items across languages.

    “In the countries where no translation to the local language was available, the forward and backward method was followed to translate the SUS items”

  • The paper relies on this without support.

    The factor structure of the SUS is equivalent across the six European countries from which data were pooled.

    “Data of six European countries were pooled for analytic purposes.”

  • The paper relies on this without support.

    The CSQ-3 is a valid measure of satisfaction for assessing convergent validity with the SUS.

    “The total score of the SUS correlated moderately with the three items on the CSQ-3, indicating convergent validity between the two measures (See Table 7).”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • PRMSE ratios are below 1, not above it

    Major

    The stated ratio is subscale score divided by total score. The reported values give 0.70/0.93 ≈ 0.75 and 0.55/0.93 ≈ 0.59, so neither ratio exceeds 1 as claimed in support of the subscale conclusion.

  • Three models called acceptable despite failing stated fit criteria

    Minor

    The methods specify an RMSEA of about 0.06 or below and say fit indices are considered in combination. The one-factor, two-factor, and tone models have RMSEAs of 0.103, 0.077, and 0.096, respectively; the one-factor model also has a TLI of 0.949, below the stated 0.95 criterion.

2 of 2

Cites

Flagged first, then by citations

1 of the 47 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. ISO I. 9241--11: 1998, Ergonomic requirements for work with visual display terminals (VDTs)-Part 11: Guidance on usability. Brussels: CEN; 1998.
  2. Sauro J. A practical guide to the system usability scale: background, benchmarks & best practices. Denver: Measuring Usability LLC; 2011.
  3. Attkisson CC, Greenfield TK. The client satisfaction questionnaire (CSQ) scales. Outcome assessment in clinical practice. Baltimore: Williams & Wilkins; 1995.
  4. Dai S, Wang X, Svetina D. Subscore: computing subscores in classical test theory and item response theory. R package. Bloomington, Indiana: Indiana University; 2019.

Cited by

Sorted by citations

Related

Authors
Mol, M., van Schaik, A., Dozeman, E., Ruwaard, J., Vis, C., Ebert, D. D., Etzelmueller, A., Mathiasen, K., Moles, B., Mora, T., Pedersen, C. D., Skjøth, M. M., Pensado, L. P., Piera-Jimenez, J., Gokcay, D., Ince, B. Ü., Russi, A., Sacco, Y., Zanalda, E., Zabala, A. F., Riper, H., & Smit, J. H.
Affiliations
  • Amsterdam Public Health (NL)
  • Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
  • University of Southern Denmark (DK)
  • Odense University Hospital (DK)
  • AULSS 2 Marca Trevigiana (IT)
Corresponding author
Mol, Mayke
ORCID
Venue
BMC Psychiatry
Publisher
Springer Science and Business Media LLC
Published
2020-05-12
Volume
20
Issue
1
Pages
218-218
ISSN
1471-244X (ISSN-L)
DOI
10.1186/s12888-020-02627-8
PMC
PMC7216472
OpenAlex
W3025136172
Semantic Scholar
7c9bab4d2245f7c20228e25754458e99ae3fcd69
Cited by
92
References
47
Citations by year
2026: 11 · 2025: 15 · 2024: 16 · 2023: 18 · 2022: 13 · 2021: 17 · 2020: 1 · 2019: 1
FWCI
11.08
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Psychiatry (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (publishedVersion, other-oa)
  • Amsterdam UMC (VU Amsterdam) - Institutional Repository (publishedVersion, other-oa)
  • VU Research Portal (publishedVersion, other-oa)
  • University of Southern Denmark Research Portal (University of Southern Denmark) (publishedVersion, cc-by)
  • OPUS FAU (Kooperativer Bibliotheksverbund Berlin-Brandenburg (KOBV), on behalf of the Universitätsbibliothek Erlangen-Nürnberg) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • OpenMETU (Middle East Technical University) (submittedVersion, cc-by-nc-nd)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Funders
European Commission (621000)
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
Keywords
System Usability Scale, usability measurement, internet-based cognitive behavioral therapy, depression, factor structure, bifactor model, convergent validity, reliability statistics, Client Satisfaction Questionnaire, mental health professionals, psychometric properties
MeSH terms
Internet-Based Intervention, Depression, Europe, Factor Analysis, Statistical, Humans, Psychometrics, Surveys and Questionnaires, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review