A chatbot for hypertension self-management support: user-centered design, development, and usability testing

May combine two claims.

Involving patients throughout design processes and ensuring technologies are usable have illustrated positive improvements in patient empowerment and safety.

“Involving patients throughout design processes and ensuring technologies are usable have illustrated positive improvements in patient empowerment and safety.31”

• cites [31] Carayon, 2019, Human factors and usability for health information technology: old and new challenges, Yearb Med Inform sourcePartly supported

  Section 4.1 says patient and caregiver participation in health IT design can produce more usable technologies and lead to empowerment, but frames these as potential benefits rather than demonstrated improvements. Sections 2.1 and 4.2 link usability and human-factors design to patient safety; they do not show that involving patients throughout design improved safety. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Purposive sampling was used to select 10 individuals with at least 50% of specified characteristics.

“Purposive sampling was used to select 10 individuals 43 with at least 50% of the following characteristics: 65þ years old, individuals of diverse race/ethnicity, male, education less than college, and taking at least 3 medications.”

• cites [43] Faulkner, 2003, Beyond the five-user assumption: benefits of increased sample sizes in usability testing, Behav Res Methods Instrum Comput sourcePartly supported

  The Results and Analyses section reports findings for groups of 10 usability-test users, so the paper supports using 10 as a sample size. Its Method section describes 60 participants grouped by experience, not purposive selection of 10 people meeting the stated age, demographic, education, and medication criteria or a 50% threshold. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Unrecognized free-text responses of health information may pose potential patient safety risks if appropriate measures and human handoffs are not in place.

“However, unrecognized free-text responses of health information may pose potential patient safety risks if appropriate measures and human handoffs are not in place.9,60”

• cites [9] Laranjo, 2018, Conversational agents in healthcare: a systematic review, J Am Med Inform Assoc sourcePartly supported

  The Discussion (Implications for healthcare, public health, and future research; page unavailable) says conversational agents sometimes failed to recognize serious health concerns and that unconstrained input increases the risk of errors. It recommends verifying user input, but does not discuss human handoffs as a safety measure. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [60] Bickmore, 2018, Patient and consumer safety risks when using conversational assistants for medical information: an observational study of siri, alexa, and google assistant, J Med Internet Res sourceSupported

May combine two claims.

Handling unrecognized utterances gracefully is particularly important for chatbots because people generally perceive robots as intelligent and may have less tolerance for mistakes.

“Handling unrecognized utterances gracefully is particularly important for chatbots because people generally perceive robots as intelligent and may have less tolerance for mistakes.61,62”

• cites [61] Ragni, 2016 sourceNot checked — no full text of the source
• cites [62] Toader, 2019, The effect of social presence and chatbot errors on trust, Sustainability sourcePartly supported

  Section 3.1.3 (p. 8) says people generally perceive robots as competent and intelligent, may assume they are flawless, and view agents that make mistakes as less reliable. The Conclusions (p. 20) report negative effects of chatbot errors on perceived competence and trust. However, the study's error was a wrong product recommendation (Section 3.2.4, p. 12); our copy does not address unrecognized utterances or how to handle them gracefully. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Adding context to error messages to enable users to better understand the cause of the error may reduce frustration.

“Thus, adding context to error messages to enable users to better understand the cause of the error may reduce frustration.63”

• cites [63] Ceaparu, 2004, Determining causes and severity of end-user frustration, Int J Hum Comput Interact sourceSupported

May combine two claims.

Erroneous agents are perceived as less reliable which negatively affects task performance.

“Erroneous agents are also perceived as less reliable which negatively affects task performance.61,62”

• cites [61] Ragni, 2016 sourceNot checked — no full text of the source
• cites [62] Toader, 2019, The effect of social presence and chatbot errors on trust, Sustainability sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Zand et al used natural language processing to develop a chatbot knowledge base by categorizing electronic messages between patients and providers.

“Zand et al 68 used natural language processing to develop a chatbot knowledge base by categorizing electronic messages between patients and providers.”

• cites [68] Zand, 2020, An exploration into the use of a chatbot for patients with inflammatory bowel diseases: retrospective cohort study, J Med Internet Res sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Similar manual processes have been reported in chatbot development studies to fine-tune the system.

“Similar manual processes have been reported in chatbot development studies to fine-tune the system.67”

• cites [67] Preininger, 2020, Artificial intelligence-based conversational agent to support medication prescribing, JAMIA Open sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Overall, ECAs have been received positively by patients and increased adherence to treatment regimens.

“Overall, these have been received positively by patients and increased adherence to treatment regimens.11,20,65,66”

• cites [11] Bickmore, 2006, Health dialog systems for patients and consumers, J Biomed Inform sourcePartly supported

  Section 8.2 (page unavailable) reports that users of the FitTrack ECA increased physical activity relative to controls, supporting a narrower claim about health behavior change. It does not report increased adherence to treatment regimens. Section 7.2 mentions satisfied users of a telephone-based system, but does not establish that ECAs were generally received positively by patients. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [20] Bickmore, 2010, Usability of conversational agents by patients with inadequate health literacy: evidence from two clinical trials, J Health Commun sourcePartly supported

  The Discussion reports that patients generally found the ECAs acceptable and usable, supporting positive reception. The ECA-Based Health Intervention Clinical Trials section says the authors assessed satisfaction and usability while remaining blinded to health outcomes; it does not establish that ECAs increased adherence to treatment regimens. (LLM, conf=0.90)

  Experimental — accuracy not yet measured
• cites [65] Bickmore, 2005, It's just like you talk to a friend' relational agents for older adults, Interact Comput sourceNot checked — no full text of the source
• cites [66] Bickmore, 2005, Establishing the computer-patient working alliance in automated health behavior change interventions, Patient Educ Couns sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The majority had scores above the U.S. population average for medication self-efficacy and felt that the greatest barriers to adherence were treatment beliefs.

“The majority had scores above the U.S. population average for medication self-efficacy and felt that the greatest barriers to adherence were treatment beliefs (ie, “I feel confident that each one of my medicines will help me”).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Among the 10 participants, a total of 252 button clicks and 128 utterances were made throughout the testing sessions.

“Among the 10 participants, a total of 252 button clicks and 128 utterances were made throughout the testing sessions.”

Classified as probably not a substantial claim.

Ten participants completed usability testing sessions.

“Ten participants completed usability testing sessions.”

May combine two claims.

Many participants felt recovering from errors was challenging and suggested having a back button in the chat or more instructive error messages.

“Additionally, many felt recovering from errors was challenging and suggested having a back button in the chat or more instructive error messages.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A few participants felt these attributes would help to establish credibility of the system.

“A few felt these attributes would help to establish credibility of the system.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants described a hamburger-like icon that included dropdown options of common tasks.

“When probed about the type of desired menu, some participants described a hamburger-like icon that included dropdown options of common tasks.”

May combine two claims.

As users progressed through the tasks, data entry task duration decreased, whereas data retrieval duration somewhat increased.

“As users progressed through the tasks, data entry task duration decreased, whereas, data retrieval duration somewhat increased (Table 2).”

Probably not a substantial claim: may not pass S3 (load-bearing).

For those who preferred navigating from the menu, a menu icon at the top was desired to select from a dropdown list.

“For those who preferred navigating from the menu, a menu icon at the top was desired to select from a dropdown list.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Half had used a chatbot before and reported using Apple Siri, Amazon Alexa, Google Assistant, or customer service chatbots from websites.

“Half had used a chatbot before and reported using Apple Siri, Amazon Alexa, Google Assistant, or customer service chatbots from websites.”

May combine two claims.

Adding a new medication took the most time on average (132 s), whereas scheduling an appointment took the least time on average (44 s).

“Adding a new medication took the most time on average (132 s), whereas scheduling an appointment took the least time on average (44 s).”

May combine two claims.

Participants who mainly used buttons began tasks by clicking on a menu button, while those who used utterances started tasks using the text input box.

“Participants who mainly used buttons began tasks by clicking on a menu button, while those who used utterances started tasks using the text input box.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A few participants wanted to interact with Medicagent through voice or a combination of voice and text.

“A few also wanted to interact with Medicagent through voice or a combination of voice and text.”

May combine two claims.

Participants inputted a variety of phrases, and some described how the default fallback messages were too generic.

“Participants inputted a variety of phrases, and some described how the default fallback messages were too generic.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Several participants thought the chatbot could be improved by adding a persona of a health professional, such as a medical avatar that could interact with them.

“Several thought the chatbot could be improved by adding a persona of a health professional, such as a medical avatar that could interact with them.”

May combine two claims.

In these cases, Medicagent prompted the user to re-enter the information with a suggested format, and all users were then able to complete the task.

“In these cases, Medicagent prompted the user to re-enter the information with a suggested format, and all users were then able to complete the task.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The majority of participants had been diagnosed with hypertension at least 5 years ago (80%) and took an average of 4 medications.

“The majority of participants had been diagnosed with hypertension at least 5 years ago (80%) and took an average of 4 medications.”

May combine two claims.

The average age was 60 years, and half were female, Black, and had at least a college education.

“The average age was 60 years, and half were female, Black, and had at least a college education (Table 1).”

May combine two claims.

Two participants made errors that prevented task completion, which included inputting the incorrect medication and not confirming a new medication was added to the medication list.

“Two participants made errors that prevented task completion, which included inputting the incorrect medication and not confirming a new medication was added to the medication list.”

May combine two claims.

The chatbot was found to have high user acceptance and good usability.

“In the emergent field of health chatbots, we describe the design, development, and usability evaluation of one of the first known chatbots focused on hypertension and medication selfmanagement, which was found to have high user acceptance and good usability.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Participants had adequate health literacy.

“The sample was limited to individuals with a computer from a single geographic region in the Southeast, and participants had adequate health literacy, high levels of medication self-efficacy, and high educational attainment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants had high levels of medication self-efficacy.

“The sample was limited to individuals with a computer from a single geographic region in the Southeast, and participants had adequate health literacy, high levels of medication self-efficacy, and high educational attainment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Task time varied based on the amount of feedback provided by participants.

“although task time varied based on the amount of feedback provided by participants.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The prototype underwent extensive pilot testing from health professionals and informaticians during development.

“The prototype underwent extensive pilot testing from health professionals (ie, physician, nurse, pharmacist) and informaticians during development.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study was reviewed and considered exempt by the University of North Carolina at Chapel Hill Office of Human Research Ethics Institutional Review Board.

“This study was reviewed and considered exempt by the University of North Carolina at Chapel Hill Office of Human Research Ethics Institutional Review Board.”

May combine two claims.

These systems demonstrated improvements in self-confidence in controlling blood pressure, knowledge and skills of home blood pressure monitoring, and diet quality and energy expenditure.

“These systems demonstrated improvements in self-confidence in controlling blood pressure, knowledge and skills of home blood pressure monitoring, and diet quality and energy expenditure.[15][16][17]”

• cites [15][16][17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Older adults and those with low health and computer literacy have expressed positive attitudes and found chatbots easy to use.

“Older adults and those with low health and computer literacy have expressed positive attitudes and found them easy to use.[20][21][22]”

• cites [20][21][22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Scores below 50 are generally considered not acceptable, 50 to 70 as marginal, and above 70 acceptable.

“Scores below 50 are generally considered not acceptable, 50 to 70 as marginal, and above 70 acceptable.50,51”

• cites [50] Bangor, 2009, Determining what individual SUS scores mean: adding an adjective rating scale, J Usability StudNot checked yet

  Not checked in the free run.

• cites [51] Brooke, 2013, SUS: a retrospective, J Usability StudNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The increase in data retrieval duration may be due to 2 data retrieval tasks that involved clicking on a hyperlink which opened up a new tab.

“This may be due to 2 data retrieval tasks that involved clicking on a hyperlink which opened up a new tab.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Incorporating features that may ease data entry, such as a calendar of dates to select, could be helpful to minimize unrecognized utterances arising from data entry errors.

“Incorporating features that may ease data entry, such as a calendar of dates to select, could be helpful to minimize unrecognized utterances arising from data entry errors.21”

• cites [21] Bickmore, 2005 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

All tasks were completed in the same order for all participants.

“The interviewer asked participants to click on a single task at a time, and all tasks were completed in the same order for all participants.”

Classified as probably not a substantial claim. May combine two claims.

Users' clicks, utterances, and task durations were extracted from Google Cloud Dialogflow analytics and verified by watching recorded Zoom sessions.

“Users' clicks, utterances, and task durations were extracted from Google Cloud Dialogflow analytics and verified by watching recorded Zoom sessions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some participants wanted the flexibility to communicate through both voice and text, which may be due to half of participants in this study reporting previous use of voice assistants.

“Some wanted the flexibility to communicate through both voice and text, which may be due to half of participants in this study reporting previous use of voice assistants.”

May combine two claims.

Participants spent an average of 18 min completing 10 self-management tasks, which is similar to other health chatbot usability interactions ranging between 10 and 25 min.

“Participants spent an average of 18 min (SD ¼ 10 min) completing 10 self-management tasks, which is similar to other health chatbot usability interactions ranging between 10 and 25 min.10”

• cites [10] Griffin, 2020, Conversational agents for chronic disease self-management: a systematic review, AMIA Annu Symp ProcNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Most of these studies evaluated usability of mental health chatbots, and the participant characteristics varied from our study (ie, the mean age of participants was younger than in our study).

“Most of these studies evaluated usability of mental health chatbots, and the participant characteristics varied from our study (ie, the mean age of participants was younger than in our study).”

• cites [52][53][54][55][56][57]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Hypothetical data within Medicagent were the same across all participants.

“Hypothetical data within Medicagent were the same across all participants, and participants were informed that the tasks did not contain their actual health data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One researcher trained in usability testing and interviewing conducted the sessions using a testing guide.

“One researcher (A.C.G.), who had been trained in usability testing and interviewing, conducted the sessions using a testing guide (Appen- dix S2).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Currently, most chatbots are still limited by their inability to hold meaningful conversations and personalized interactions.

“Currently, most chatbots are still limited by their inability to hold meaningful conversations and personalized interactions.14”

• cites [14] McTear, 2020 sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Questionnaire scores were calculated following standard scoring methodology from each validated instrument and summarized with descriptive statistics.

“Questionnaire scores were calculated following standard scoring methodology from each validated instrument and summarized with descriptive statistics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The guide and questionnaires were pilot tested with members of the study team.

“The guide and questionnaires were pilot tested with members of the study team.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The SUS questionnaire is a reliable 10-item questionnaire used to measure the usability of a system.

“Following task completion, participants completed the SUS questionnaire which is a reliable 10-item questionnaire used to measure the usability of a system.48”

• cites [48] Brooke, 1996Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Each session was transcribed verbatim using Zoom transcription and analyzed by a trained qualitative researcher using a thematic analysis approach.

“Each session was transcribed verbatim using Zoom transcription and analyzed by a trained qualitative researcher (A.C.G.) using a thematic analysis approach.49”

• cites [49] Guest, 2012 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Themes were discussed and revised with members of the research team.

“Themes were discussed and revised with members of the research team.”

Probably not a substantial claim: may not pass S2 (checkable).

At this early stage with limited health care corpora for training data, it is unlikely a chatbot would be able to recognize the vast number of possible free-text inputs for all situations.

“At this early stage with limited health care corpora for training data, it is unlikely a chatbot would be able to recognize the vast number of possible free-text inputs for all situations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior studies have found that users prefer to have the option to input free-text or use buttons.

“Prior studies have found that users prefer to have the option to input freetext or use buttons.59”

• cites [59] Budiu, 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The sample was limited to individuals with a computer from a single geographic region in the Southeast.

“The sample was limited to individuals with a computer from a single geographic region in the Southeast, and participants had adequate health literacy, high levels of medication self-efficacy, and high educational attainment.”

Classified as probably not a substantial claim. May combine two claims.

This research provides an overview of the design process and highlights several components that may enrich the user experience of health-related chatbots.

“We provide an overview of our design process and highlight several components that may enrich the user experience of health-related chatbots.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Comparable approaches may be useful to inform self-management knowledge bases, which could be used during chatbot development and tailored for cultural and linguistic differences.

“Comparable approaches may be useful to inform self-management knowledge bases, which could be used during chatbot development and tailored for cultural and linguistic differences.”

May combine two claims.

For the prototype, hypothetical data were used and protected health information (PHI) was not included.

“For our prototype, we used hypothetical data and did not include protected health information (PHI)”

Classified as probably not a substantial claim. May combine two claims.

Flexibility of various interaction styles, handling unrecognized utterances gracefully with contextual error messages, and having a credible health professional persona were highlighted as design features that could facilitate usability and navigation within chatbots for self-management.

“Flexibility of various interaction styles, handling unrecognized utterances gracefully with contextual error messages, and having a credible health professional persona were highlighted as design features that could facilitate usability and navigation within chatbots for self-management.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There were variances in Internet speed and potential differences in hardware across participants.

“there were variances in Internet speed and potential differences in hardware.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All tasks were completed in the same order, which may present ordering bias.

“All tasks were completed in the same order, which may present ordering bias.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Conducting remote usability testing allowed observation of interactions within a user's natural environment at home where self-management tasks would typically take place.

“However, conducting remote usability testing allowed us to observe interactions within a user's natural environment at home where self-management tasks would typically take place.”

• AssumptionRests on an assumption: The home environment is where self-management tasks would typically take place.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Including additional visual attributes, such as those in ECAs, may be beneficial in establishing rapport and engagement with the chatbot.

“Consequently, including additional visual attributes, such as those in ECAs, may be beneficial in establishing rapport and engagement with the chatbot.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This may provide some insights for chatbot development in the context of other chronic conditions requiring self-management.

“This may provide some insights for chatbot development in the context of other chronic conditions requiring self-management.”