CoChat
Sign Up
Cited by 36Digital Mental Health InterventionsSocial Robot Interaction and HRIEmotion and Mood Recognition

Usability and acceptability assessment of an empathic virtual agent to prevent major depression

Open access
Bresó, A., Martínez‐Miranda, J., Botella, C., Baños, R. M., García‐Gómez, J. M.·2016·Expert Systems
DOI 10.1111/exsy.12151
Open this paper in CoChat — free accountOpen PDF

Abstract

In Human–Computer Interaction, the adaptation of the content and the way of how this content is communicated to the users in interactive sessions is a critical issue to promote the acceptability and usability of any computational system. We present a user‐adapted interactive platform to identify and provide an early intervention for symptoms of depression and suicide. In particular, we describe the work performed to assess users' system acceptability and usability. An empathic Virtual Agent is the main interface with the user, and it has been designed to generate the appropriate dialogues and emotions during the interactions according to the detected user's specific needs. This personalization is based on a dynamic user model nurtured with clinical, demographical and behavioural information. The evaluation was performed with 60 participants from the university community. The obtained results were promising, allowing the execution of a further clinical trial. The system's usability score was 75.7%, and the score of the user‐adapted content and the emotional responses of the Virtual Agent was 70.9%.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper presents and evaluates a user-adapted interactive system with an empathic virtual agent for preventing depression and suicide in university students. In a pre-clinical usability/acceptability evaluation with 60 participants, usability was 75.7/100 and overall acceptability of content and virtual-agent emotional responses was 70.9%.

Key findings

  • System Usability Scale score was 75.7/100, above the 68 benchmark cited from Sauro; 93.6% of the 60 participants fully completed the evaluation questionnaire.
  • Session-content acceptability: length q14 weighted average 3.22/4, variability q15 2.66/4, content q16 2.97/4; overall session management score was 2.95/4.
  • Virtual-agent acceptability: appearance/behavior score was 2.75/4; 66% disagreed or strongly disagreed that the agent was “distant and aloof” (q17), 57% perceived it as trustworthy (q18), 82% perceived it as “emotionally stable” (q19), and 48% considered it a decisive factor motivating system use (q20).
  • No significant difference in virtual-agent acceptance was found based on participant gender; qualitative feedback asked for shorter questionnaires, better GUI, more university-context questions, more therapy-based activities, and improved avatar appearance.

Methods

  • Design: pre-clinical usability and acceptability evaluation of the PrevenDep system before a clinical trial; participants watched pre-recorded videos of interactive sessions and then completed an online 22-question questionnaire over 15 days.
  • Sample: 60 participants accessed the evaluation, recruited from approximately 100 professors, researchers, and final-degree students across four locations: Universitat Politècnica de València, Universitat de València, Universitat Jaume I, and CICESE-UT3.
  • Measures and analysis: questionnaire sections covered demographics, SUS usability (q4–q13), session-content acceptability (q14–q16), virtual-agent acceptability (q17–q20), and free-text comments (q21–q22); responses used a 0–4 Likert scale, with SUS scored by the standard method and descriptive weighted averages plus qualitative comments reported.

Limitations

  • Paper states: the evaluation was performed with professors and researchers before use with target users for security and ethical reasons, not as a clinical efficacy trial; full natural-language interaction was not implemented because current technology does not allow it free of errors; and avatar appearance requires deeper programming/customization.
  • Evident from the text: participants evaluated pre-recorded videos rather than using the system themselves, so usability/acceptability may not reflect real interaction; the sample was largely academic staff/researchers rather than the target university-student population; and free-text completion was lower (65.8%) than for the first 20 questions (96.4%).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1111/exsy.12151

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 13 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The system's usability was 75.7%.

    “The system's usability was 75.7%”
    Quote in the paperSubstantial claim
  • Findingown data

    Participants valued the appearance and behavior of the virtual agent positively, with a score of 2.75 out of 4.

    “Regarding the evaluation of the appearance and behavior of the virtual agent -as the main interface between the user and the system-, the participants also valued positively this component (getting a score of 2.75 out of 4).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Evaluation via pre-recorded videos reflects the perceived acceptability and usability of the system.
  • Findingown data

    No significant difference was found in acceptance of the virtual agent based on participant gender.

    “Nevertheless, in our evaluation we did not find any significant difference in the acceptance of virtual agent based on the gender of the participant.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The average SUS score obtained was 75.7.

    “Figure 9 presents the average result obtained from the collected responses corresponding to a value of 75.7 (gray line), which is above the average proposed by Jeff Sauro (orange line).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Sauro, 2011

      Not checked in the free run.

  • Findingown data

    Participants rated the usability of the system as highly positive, with a score of 75.7 out of 100.

    “The participants rated the usability of the system as highly positive (with a result 75.7 out of 100).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Evaluation via pre-recorded videos reflects the perceived acceptability and usability of the system.
  • Classified as probably not a substantial claim. 18
  • May combine two claims. 13

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The gender distribution of participants was 26 (43.3%) women, 33 (55%) men, and one participant (1.7%) omitted this information.

    “the gender distribution included 26 (43.3%) women, 33 (55%) men, and one participant omitted this information (1.7%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The qualitative suggestions indicated that more context-based questions (e.g., university context) and questions about perceived usefulness should be included.

    “From the second group of suggestions we can observe that more context-based questions (i.e. from the university context) needs to be included, as well as some question(s) about the perceived usefulness from the target users regarding the contents offered by the system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The collected comments could be grouped into those focused on how session contents are presented through the GUI (P4, P21, P37, P51) and those focused on the session contents themselves (P36, P44, P49, P50).

    “From the collected comments, we can identify that some of them are more focused on how the contents of the session are presented through the GUI (P4, P21, P37 and P51) and others in the contents of the sessions themselves (P36, P44, P49 and P50).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The qualitative suggestions indicated that more therapy-based activities were suggested.

    “Moreover, we also observed that the inclusion of more therapy-based activities has been suggested.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Many comments were positive, but many also addressed particular issues that can be improved before the pre-clinical trial.

    “Many comments were positive but also many of them addressed particular issues that can be improved before the execution of the pre-clinical trial of the system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    ICT-based tools to prevent depression or other disorders might be a promising way forward.

    “ICT-based tools to prevent depression or other disorders might be one promising way forward (Kazdin & Blase, 2011).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kazdin & Blase, 2011

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The functionalities of each system component ensure a continuous user-adapted intervention, which is key to maximizing benefits for people with mental health problems.

    “The functionalities of each component of the system ensure a continuous user-adapted intervention, a key aspect to maximize the benefits that this type of interventions can produce in people with problems related to mental health.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Participation was 96.4% for the first 20 questions and 65.8% for the last two questions (q21 and q22).

    “Specifically, the participation in the first 20 questions was 96.4% and in the two last questions (q21 and q22) was 65.8%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The FAtiMA architecture offers a generic appraisal framework where a set of different basic emotions and the coping behaviors can be generated according to a set of goals, preferences and action tendencies predefined for a virtual agent.

    “Based on this theory, the FAtiMA architecture offers a generic appraisal framework where a set of different basic emotions and the coping behaviors can be generated according to a set of goals, preferences and action tendencies predefined for a virtual agent.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Dias, et al., 2014 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    All SUS results exceeded the threshold, indicating a good assessment of the perceived usability of the system.

    “All the results exceed the threshold meaning a good assessment in the perceived usability of the system”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Early diagnosis and treatment of depression can relieve depression symptoms, prevent depression from returning, and help students succeed in college and university.

    “Early diagnosis and treatment of depression can relieve depression symptoms, prevent depression from returning, and help students succeed in college and university (Cuijpers, et al., 2012).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Cuijpers, et al., 2012

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The cognitive evaluation (or appraisal) process leads to an emotional response (according to the relevance that the event has for the person), which in turn generates a specific behavior to cope with the appraised events.

    “This cognitive evaluation (or appraisal) process leads to an emotional response (according to the relevance that the event has for the person), which in turn generates a specific behavior to cope with the appraised events.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Scherer, et al., 2001 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Psychotherapy research needs to broaden to adopt large-scale public health strategies and treatments applicable to more patients in a simpler and cost-effective way.

    “psychotherapy research needs to broaden in terms of adoption of large-scale public health strategies and treatments that can be applied to more patients in a simpler and cost-effective way (Emmelkamp, et al., 2014)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Emmelkamp, et al., 2014

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Different settings produce different contents in the sessions and the interaction can be more flexible or more restrictive in terms of variability and adaptability.

    “As evaluated in (Bresó, et al., 2015), different settings produce different contents in the sessions and the interaction can be more flexible or more restrictive in terms of variability and adaptability.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bresó, et al., 2015

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The SUS questionnaire was created by the Digital Equipment Corporation and mainly assesses perceived usability (subjective aspects of usability) on any system.

    “The SUS questionnaire was created by the Digital Equipment Corporation and it is mainly focused on the assessment of perceived usability (subjective aspects of usability) on any system.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Brooke, 1996

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The evaluation questionnaire was online for 15 days and 60 participants accessed it.

    “The questionnaire was online during 15 days and a total of 60 participants accessed to the evaluation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A previous study performed a functional evaluation of the system components through simulations.

    “In a previous study we perform a functional evaluation of the components through a set of simulations (Bresó, et al., 2015).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bresó, et al., 2015

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    The obtained results were promising, allowing the execution of a further clinical trial.

    “The obtained results were promising, allowing the execution of a further clinical trial.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Improving the suggested features of the virtual agent's appearance would positively influence perception of more suitable behavior and could help optimize system effectiveness.

    “The further improvement of the suggested features in the appearance of the virtual agent would even positively influence the perception of a more suitable behavior during the interactive sessions, which in turn could help to optimize the effectiveness of the system.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The acceptability results can be summarized as 70.9%, reflecting that participants rated a high level of acceptability of the framework.

    “We can summarize these acceptability results as a percentage of 70.9% reflecting that the participants have rated a high level of acceptability of the framework.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Evaluation via pre-recorded videos reflects the perceived acceptability and usability of the system.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    People usually feel more comfortable when interviewed by media-mediated electronic doctors and are more likely to release their flinched mind during the consultation process in contrast to human doctors.

    “Regarding the use of virtual agents, there are supporting studies suggesting that people usually feel more comfortable when interviewed by media-mediated electronic doctors and are more likely to release their flinched mind during the consultation process in contrast to human doctors (Yoshida, et al., 2002).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yoshida, et al., 2002

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Many people experience the first symptoms of depression during their college or university years.

    “Unfortunately, many people experience the first symptoms of depression during their college or university years (Pelkonen, et al., 2005).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Pelkonen, et al., 2005

      Not checked in the free run.

22 of 22

  • The paper relies on this without support.

    Depression and suicide are two major problems in the global society and are closely related.

    “Both, depression and suicide are two major problems in the global society and are close related.”

  • The paper relies on this without support.

    If depression symptoms are not opportunely detected and the individual does not receive appropriate support, in some cases the person commits suicide.

    “If these symptoms are not opportunely detected and the individual does not receive an appropriate support, in some cases the person commits suicide.”

  • The paper relies on this without support.

    University students are exposed to a challenging and competitive environment in which they must overcome new situations, stress, academic challenges, or social relationships.

    “University students are exposed to challenging and competitive environment in which they must to overcome new situations, stress, academic challenges, or social relationships.”

  • The paper relies on this without support.

    In order to optimize the effectiveness of these tools, a high level of personalization is required during the course of a session and through every session.

    “Nevertheless, in order to optimize the effectiveness of these tools, a high level of personalization is required during the course of a session and through every session.”

  • The authors state this as an assumption.

    The events produced in a person's environment are constantly evaluated by the individual.

    “The key assumption of the appraisal theory of emotions is that the events produced in a person's environment are constantly evaluated by the individual.”

  • The paper relies on this without support.

    Realism in the appearance and behaviour in a virtual agent are key issues to achieve good levels of acceptability in the target users.

    “Realism in the appearance and behaviour in a virtual agent are key issues to achieve good levels of acceptability in the target users.”

  • The paper relies on this without support.

    Offering adequate support during these periods helps to prevent depression and suicide.

    “Offering adequate support during these periods helps to prevent depression and suicide.”

  • The paper relies on this without support.

    Depression and suicidal ideation have an enormous prevalence, especially in young people.

    “given the enormous prevalence of depression and suicidal ideation, especially in young people.”

  • The paper relies on this without support.

    A good relationship between a virtual agent and the user is essential to promote continuous use of the system and to establish a therapeutic alliance that benefits the user.

    “The building of a good relationship between a virtual agent and the user is essential to promote a continuous use of the system and a key ingredient to establish a therapeutic alliance that benefit the user.”

  • The paper relies on this without support.

    The current state of technology does not allow full natural language interaction free of errors.

    “As the current state of the technology does not allow a full natural language interaction free of errors, we decided to maintain the input from the user through the buttons and other controls such as combo boxes included in the GUI and not to receive the input from the user through natural language processing.”

  • The paper relies on this without support.

    The system should adapt the length of each session according to the user's status and the clinician's requirements.

    “The produced length of a session is highly important taking into account that the system should adapt the length of each session according to the status (e.g.mood) detected in the user and at the same time meeting the clinician's requirements (i.e. the scheduling of certain activities a minimum number of times per week).”

  • The paper relies on this without support.

    Evaluation via pre-recorded videos reflects the perceived acceptability and usability of the system.

    “In this study we present an initial evaluation about the perceived acceptability and usability involving 60 participants who answered a questionnaire after watching a set of pre-recorded videos containing different interactive sessions.”

  • The paper relies on this without support.

    A sample of 60 participants is adequate to evaluate the usability and acceptability of the system.

    “The questionnaire was online during 15 days and a total of 60 participants accessed to the evaluation.”
13 of 13

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The usability level (SUS score of 61.6) is comparable with the 2 psychological therapy chatbots with SUS scores of 63.6 and 57.0 but lower than that of asthma management (SUS score of about 83) and depression prevention chatbots (SUS score of approximately 75.7).

    “This level of usability (SUS scores of 61.6) is comparable with the 2 psychological therapy chatbots with SUS scores of 63.6 and 57.0 [34] but lower than that of asthma management (SUS score of about 83) [35] and depression prevention chatbots (SUS score of approximately 75.7) [36].”

    From Feasibility, Usability, and Effectiveness of a Machine Learning–Based Physical Activity Chatbot: Quasi-Experimental Study (2021) · cites this paper as [36]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Daily-use motivation is assigned to the wrong question

    Minor

    The results text identifies q19 as assessing whether the agent motivates daily use, whereas the questionnaire assigns that statement to q20. Q19 asks whether the agent is emotionally stable.

  • Gender figure cites the age question

    Minor

    The figure caption attributes gender distribution to q1, but the questionnaire defines q1 as age and q2 as gender.

  • Disagreement with the 'distant and aloof' statement is misstated

    Minor

    For q17, 26% strongly disagreed and 36% disagreed, totaling 62%, not 66%. The counts likewise total 36 of 58 respondents, or about 62%.

    Reported 66 · recomputed 62

  • Age groups are repeated instead of matching the plotted categories

    Minor

    The prose assigns the groups of 15, 12, and 9 participants all to ages 31-35. The age-distribution figure instead labels these successive groups 31-35, 36-40, and 41-45.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Bole Williams B., 2013, Suicide Prevention
  2. Bresó A., 2016, Intelligent virtual agents applied in mental health care: a survey [under review], Journal of Medical Systems
  3. LisettiL. C.2008Embodied conversational agents for psychotherapy.In CHI 2008 Workshop on Technology in Mental Health.
  4. Organization W. H., 2014, Preventing Suicide: A Global Imperative
  5. Sauro J., 2011, A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices

Cited by

Sorted by citations

Related

Authors
Bresó, A., Martínez‐Miranda, J., Botella, C., Baños, R. M., & García‐Gómez, J. M.
Affiliations
  • Universitat Politècnica de València (ES)
  • CONACYT Research Fellow – Centro de Investigación Científica y de Educación Superior de Ensenada Unidad de Transferencia Tecnológica (CICESE‐UT3) Tepic México
  • Universitat Jaume I (ES)
  • Universitat de València (ES)
  • Grupo de Informática Biomédica, Instituto ITACA Universitat Politècnica de València Spain
Corresponding author
Bresó, A.
ORCID
Venue
Expert Systems
Publisher
Wiley
Published
2016-05-25
Volume
33
Issue
4
Pages
297-312
ISSN
  • 0266-4720 (ISSN-L)
  • 1468-0394
DOI
10.1111/exsy.12151
OpenAlex
W2398410084
Cited by
36
References
33
Citations by year
2026: 2 · 2025: 4 · 2024: 2 · 2023: 5 · 2022: 2 · 2021: 4 · 2020: 7 · 2019: 4 · 2018: 1 · 2017: 5
FWCI
5.12
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
  • Expert Systems (publishedVersion)
  • RiuNet (Universitat Politècnica de València) (submittedVersion, cc-by-nc-nd)
Funders
Generalitat Valenciana
Topics
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
Keywords
depression prevention, suicide prevention, early intervention, usability evaluation, acceptability, personalized dialogue generation, university students

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review