CoChat
Sign Up
Cited by 68Mobile Health and mHealth ApplicationsDigital Mental Health InterventionsPhysical Activity and Health

Feasibility, Usability, and Effectiveness of a Machine Learning–Based Physical Activity Chatbot: Quasi-Experimental Study

Open access
To, Quyen G, Green, Chelsea, Vandelanotte, Corneel·2021·JMIR mHealth and uHealth
DOI 10.2196/28577PMC PMC8665384

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Behavioral eHealth and mobile health interventions have been moderately successful in increasing physical activity, although opportunities for further improvement remain to be discussed. Chatbots equipped with natural language processing can interact and engage with users and help continuously monitor physical activity by using data from wearable sensors and smartphones. However, a limited number of studies have evaluated the effectiveness of chatbot interventions on physical activity. Objective: This study aims to investigate the feasibility, usability, and effectiveness of a machine learning–based physical activity chatbot. Methods: A quasi-experimental design without a control group was conducted with outcomes evaluated at baseline and 6 weeks. Participants wore a Fitbit Flex 1 (Fitbit LLC) and connected to the chatbot via the Messenger app. The chatbot provided daily updates on the physical activity level for self-monitoring, sent out daily motivational messages in relation to goal achievement, and automatically adjusted the daily goals based on physical activity levels in the last 7 days. When requested by the participants, the chatbot also provided sources of information on the benefits of physical activity, sent general motivational messages, and checked participants’ activity history (ie, the step counts/min that were achieved on any day). Information about usability and acceptability was self-reported. The main outcomes were daily step counts recorded by the Fitbit and self-reported physical activity. Results: Among 116 participants, 95 (81.9%) were female, 85 (73.3%) were in a relationship, 101 (87.1%) were White, and 82 (70.7%) were full-time workers. Their average age was 49.1 (SD 9.3) years with an average BMI of 32.5 (SD 8.0) kg/m2. Most experienced technical issues were due to an unexpected change in Facebook policy (93/113, 82.3%). Most of the participants scored the usability of the chatbot (101/113, 89.4%) and the Fitbit (99/113, 87.6%) as at least “OK.” About one-third (40/113, 35.4%) would continue to use the chatbot in the future, and 53.1% (60/113) agreed that the chatbot helped them become more active. On average, 6.7 (SD 7.0) messages/week were sent to the chatbot and 5.1 (SD 7.4) min/day were spent using the chatbot. At follow-up, participants recorded more steps (increase of 627, 95% CI 219-1035 steps/day) and total physical activity (increase of 154.2 min/week; 3.58 times higher at follow-up; 95% CI 2.28-5.63). Participants were also more likely to meet the physical activity guidelines (odds ratio 6.37, 95% CI 3.31-12.27) at follow-up. Conclusions: The machine learning–based physical activity chatbot was able to significantly increase participants’ physical activity and was moderately accepted by the participants. However, the Facebook policy change undermined the chatbot functionality and indicated the need to use independent platforms for chatbot deployment to ensure successful delivery of this type of intervention.

AI snapshot

Written from the paper’s full text.

TL;DR

A machine learning–based physical activity chatbot delivered via Facebook Messenger, paired with a Fitbit Flex 1, significantly increased objectively measured steps and self-reported physical activity over a 6-week quasi-experimental study, but usability and acceptability were only moderate. A Facebook policy change caused widespread technical failures and forced early study termination.

Key findings

  • Among 116 participants who completed follow-up, adjusted analyses showed an increase of 627 steps/day (95% CI 219-1035) and total physical activity 3.58 times higher at follow-up (95% CI 2.28-5.63), representing an increase of 154.2 min/week; odds of meeting physical activity guidelines were higher at follow-up (OR 6.37, 95% CI 3.31-12.27).
  • Usability was moderate: mean chatbot System Usability Scale score was 61.6 (SD 9.7), with 89.4% (101/113) rating the chatbot at least “OK”; 53.1% (60/113) agreed the chatbot helped them become more active, 35.4% (40/113) would continue using it, and 43.4% (49/113) would recommend it.
  • Engagement averaged 6.7 (SD 7.0) messages/week sent to the chatbot and 5.1 (SD 7.4) min/day spent using it; 93.8% (106/113) read the chatbot messages.
  • Technical problems were common: 82.3% (93/113) experienced technical issues and 84.1% (95/113) had the chatbot stop sending messages at some point, largely due to an unexpected Facebook policy change. BMI did not significantly improve (adjusted difference −0.13, 95% CI −0.37 to 0.11).

Methods

  • Quasi-experimental design without a control group, with outcomes at baseline and 6 weeks; participants were inactive Australian adults recruited from the 10,000 Steps program database, with 120 enrolled at baseline and 116 completing follow-up.
  • Intervention: participants wore a Fitbit Flex 1 and connected to the chatbot “Ida” via Facebook Messenger; the chatbot used Dialogflow natural language processing and COM-B–informed proactive messages (daily activity updates, motivational messages, adaptive goals increasing by 500 steps/day or 5 min MVPA/day when the 7-day average met the current goal) plus reactive information on request.
  • Main outcomes were Fitbit-recorded daily step counts and self-reported physical activity via the Active Australia Survey; usability/acceptability used the System Usability Scale and self-report questions. Generalized linear mixed models produced crude and adjusted estimates, with a full sample and a subsample excluding those with <4 weeks of chatbot use; posthoc power was 81.3%.

Limitations

  • The paper states: no control group meant unknown confounders could not be controlled, and increases may have occurred from Fitbit use alone; the sample was mostly women, White, and had high BMI, limiting generalizability; Facebook policy–related technical issues likely reduced usability, acceptability, and effectiveness; and the 6-week duration was short.
  • The paper also notes that self-reported physical activity may be affected by recall and social desirability bias, and that the Fitbit Flex 1 was an old model with battery and connectivity problems.
  • Evident from the text: rolling recruitment and variable participation durations (17 participants <4 weeks, 51 between 4 and <6 weeks, 48 at least 6 weeks) complicate interpretation of a fixed 6-week intervention effect.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/28577

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 18 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Average messages/week sent to the chatbot was 6.7 (SD 7.0) among participants (n=113).

    “Average messages/week sent to the chatbot (n=113), mean (SD) | Average messages/week sent to the chatbot (n=113), mean (SD) | 6.7 (7.0)”
    Quote in the paperSubstantial claim
  • Findingown data

    About one-third thought the chatbot was quite or very useful in helping them stay motivated to participate in physical activity.

    “About one-third thought the chatbot was quite or very useful in helping them increase confidence for engaging in regular physical activity, and in helping them stay motivated to participate in physical activity.”
    Quote in the paperNot a substantial claim
  • Findingown data

    About one-third thought the chatbot was quite or very useful in helping them increase confidence for engaging in regular physical activity.

    “About one-third thought the chatbot was quite or very useful in helping them increase confidence for engaging in regular physical activity”
    Quote in the paperNot a substantial claim
  • Findingown data

    Most participants experienced technical issues (93/113, 82.3%).

    “Most participants experienced technical issues (93/113, 82.3%)”
    Quote in the paperNot a substantial claim
  • Findingown data

    On average, 6.7 (SD 7.0) messages/week were sent to the chatbot.

    “On average, the participants sent 6.7 messages to the chatbot per week”
    • Also stated in: § Results · plain
      “On average, 6.7 (SD 7.0) messages/week were sent to the chatbot”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 11
  • The quote only partly states this claim. 1
  • May combine two claims. 15

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Most experienced technical issues were due to an unexpected change in Facebook policy (93/113, 82.3%).

    “Most experienced technical issues were due to an unexpected change in Facebook policy (93/113, 82.3%).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The average usability score for Fitbit Flex 1 was 64.0 (SD 11.1).

    “The average usability for Fitbit was 64.0 (SD 11.1)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Significant improvements in both step count and self-reported physical activity were observed.

    “Significant improvements in both the step count and self-reported physical activity were observed.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The participants liked the chatbot, with some even asking to continue using it after they completed the 6-week trial.

    “The findings showed that the participants liked the chatbot with some even asking for continuing to use it after they completed the 6-week trial.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the subsample, total physical activity minutes at follow-up were 4.17 (95% CI 2.55-6.80) times higher than at baseline, representing an increase of 176.6 min/week.

    “the total physical activity minutes at follow-up were 3.58 (95% CI 2.28-5.63) times higher in the full sample and 4.17 (95% CI 2.55-6.80) times higher in the subsample than at baseline, representing an increase 154.2 and 176.6 min/week, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For both samples, BMI improved but was not statistically significant at follow-up compared with baseline.

    “For both samples, BMI was improved but was not statistically significant at follow-up compared with baseline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Among the 116 participants, 17 participated in <4 weeks, 51 between 4 and <6 weeks, and 48 at least 6 weeks.

    “Among the 116 participants, 17 participated in <4 weeks, 51 between 4 and <6 weeks, and 48 at least 6 weeks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The analysis was adjusted for sample characteristics including age, gender, marital status, years of schooling, ethnicity, household income, living area, work status, and daily work time.

    “For each outcome, 2 models were run to generate crude estimates and estimates adjusted for sample characteristics including age, gender, marital status, years of schooling, ethnicity, household income, living area, work status, and daily work time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    120 participants were enrolled at baseline, but only 116 completed the follow-up surveys.

    “As a result, 120 participants were enrolled at baseline. However, only 116 completed the follow-up surveys.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    As a robustness check, the analysis was performed separately for a full sample and a subsample excluding those using the chatbot <4 weeks.

    “As a robustness check, the analysis was performed separately for 2 samples, a full sample and a subsample (excluding those using the chatbot <4 weeks).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    An increase of 1000 steps/day can help reduce all-cause mortality risk between 6% and 36%, although an increase of approximately 600 steps may not have a similar effect.

    “It is also worth noting that an increase of 1000 steps/day can help reduce all-cause mortality risk between 6% and 36%, although an increase of approximately 600 steps may not have a similar effect [32].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A study that used a chatbot to promote stair climbing reported a significant increase in physical activity after 12 weeks of intervention, although it is not clear how physical activity was measured and what the effect size was.

    “Finally, we also found a study that used a chatbot to promote stair climbing, and it also reported a significant increase in physical activity after 12 weeks of intervention, although it is not clear how physical activity was measured and what the effect size was [33].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    The usability level (SUS score of 61.6) is comparable with the 2 psychological therapy chatbots with SUS scores of 63.6 and 57.0 but lower than that of asthma management (SUS score of about 83) and depression prevention chatbots (SUS score of approximately 75.7).

    “This level of usability (SUS scores of 61.6) is comparable with the 2 psychological therapy chatbots with SUS scores of 63.6 and 57.0 [34] but lower than that of asthma management (SUS score of about 83) [35] and depression prevention chatbots (SUS score of approximately 75.7) [36].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [34] source

      Not checked in the free run.

    • cites [35] source

      Not checked in the free run.

    • cites [36] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The effect measured by the Fitbits appeared to be smaller, as the average step count only increased by approximately 600 steps/day or 4200 steps/week, which roughly corresponds to an increase of 42 minutes of physical activity per week.

    “However, the effect measured by the Fitbits appeared to be smaller, as the average step count only increased by approximately 600 steps/day or 4200 steps/week, which roughly corresponds to an increase of 42 minutes of physical activity per week (assuming it takes approximately 10 minutes to take 1000 steps) [20].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

    • The authors state this as an assumption: It takes approximately 10 minutes to take 1000 steps.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    A possible explanation for the lower Fitbit usability is that the research team was forced to use Fitbit Flex 1, an old model, because of budgetary reasons.

    “A possible explanation might be that because of budgetary reasons, the research team was forced to use Fitbit Flex 1, which is an old model in this study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The technical issues experienced in this study might have reduced the effectiveness of the intervention and explained the differences between the 2 studies.

    “In addition, the technical issues experienced in this study might have reduced the effectiveness of the intervention and explained the differences between the 2 studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The technical issues caused by Facebook's policy changes, which were beyond the research team's control, were likely responsible for a reduction in chatbots' usability, acceptability, and effectiveness.

    “the technical issues caused by Facebook’s policy changes, which were beyond the research team’s control, were likely responsible for a reduction in chatbots’ usability, acceptability, and effectiveness.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The machine learning-based physical activity chatbot was able to significantly increase participants' physical activity and was moderately accepted by the participants.

    “The machine learning–based physical activity chatbot was able to significantly increase participants’ physical activity and was moderately accepted by the participants.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    That study also included a large dietary component and allowed more time (12 weeks) for weight loss to occur.

    “However, that study also included a large dietary component and allowed more time (12 weeks) for weight loss to occur [20].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The sample (majority were women and White with high BMI) was not representative of the broader Australian population, so generalizability of the findings may be limited.

    “the sample (majority were women and White with high BMI) was not representative of the broader Australian population, so generalizability of the findings may be limited”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    This finding is not surprising, as the study did not target weight loss and therefore no direct activity related to weight loss or weight maintenance was delivered.

    “This finding is not surprising, as our study did not target weight loss and therefore, no direct activity related to weight loss or weight maintenance was delivered.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The short duration of the intervention (6 weeks) is a limitation.

    “the short duration of the intervention (6 weeks) is a limitation”
    Quote in the paperNot a substantial claim
22 of 22

  • The paper relies on this without support.

    The observed increases in physical activity are attributable to the chatbot intervention rather than to secular trends, regression to the mean, or other uncontrolled factors.

    “A quasi-experimental design without a control group was conducted with outcomes evaluated at 2 time points—baseline and 6 weeks after participants started to use the chatbot.”

  • The paper relies on this without support.

    Fitbit Flex 1 step counts accurately measure participants' physical activity.

    “Physical activity was objectively measured using the Fitbit Flex 1.”
    “The Fitbit Flex 1 device was used to measure the participants’ daily physical activity.”

  • The authors state this as an assumption.

    It takes approximately 10 minutes to take 1000 steps.

    “assuming it takes approximately 10 minutes to take 1000 steps”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • To et al reported that the decrease in BMI was not significant (95% CI −0.37 to 0.11).

    “Fourth, only To et al [32] reported that the decrease in BMI was not significant (95% CI −0.37 to 0.11).”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [32]

    The Results (Effectiveness of the Intervention; page unavailable) say BMI improved but the change was not statistically significant at follow-up. The passages retrieved from our copy do not give the claimed 95% CI of −0.37 to 0.11, so that numerical detail is unverified. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Many studies (5/15, 33%) reported total number of messages exchanged with the chatbot; only 7% (1/15) reported exact proportion of user-initiated conversations (approximately 30%).

    “Many studies reported the total number of messages exchanged with the chatbot (5/15, 33%) [6,26,29,31,32]; however, only 7% (1/15) of studies reported the exact proportion of user-initiated conversations (approximately 30%) [6], which depicted the participants’ level of interest in having health-related conversations with the chatbot.”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [32]

    The Results > Usability and Acceptability section reports an average of 6.7 messages sent by participants to the chatbot per week. It does not report a total number of messages exchanged or the claimed approximately 30% proportion of user-initiated conversations; the Methods > Intervention section describes participant-initiated requests without quantifying their proportion. Page numbers were not available in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Of the 15 studies, 7 (47%) targeted healthy lifestyles, and 5 (33%) assessed efficacy of AI chatbots in promoting healthy lifestyles.

    “Out of the 15 studies, 7 (47%) studies [5,21,22,28,29,32,33] targeted healthy lifestyles, and 5 (33%) studies [21,22,28,32,33] assessed the efficacy of AI chatbots in promoting healthy lifestyles through (1) physical activity levels, (2) healthy diet, (3) blood pressure, and (4) BMI.”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [32]

  • To et al reported that participants recorded more steps (P<.01) and more total physical activity (3.58 times higher; P<.001), and were more likely to meet physical activity guidelines (95% CI 3.31-12.27) at follow-up.

    “To et al [32] reported that the participants recorded more steps (P<.01) and more total physical activity (3.58 times higher; P<.001). Moreover, the participants were also more likely to meet the physical activity guidelines (95% CI 3.31-12.27) at follow-up.”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [32]

  • In 7% (1/15) of studies, most participants reported technical issues in using the chatbot (82.3%), one reason being that they stopped receiving chatbot messages during the study period (84.1%).

    “In 7% (1/15) of studies [32], most participants reported technical issues in using the chatbot (82.3%), one of the reasons being that they stopped receiving the chatbot messages during the study period (84.1%).”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [32]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

  • Once-a-day messaging percentage uses the wrong denominator

    Minor

    The table gives n=77 for this question, but 28 of 77 is 36.4%, not 24.8%.

    Reported 24.8 · recomputed 36.363636

  • Several-times-a-day messaging percentage uses the wrong denominator

    Minor

    The table gives n=77 for this question, but 30 of 77 is 39.0%, not 26.6%.

    Reported 26.6 · recomputed 38.961039

  • Less-than-once-a-day messaging percentage is incorrect

    Minor

    The table gives n=77 for this question, but 19 of 77 is 24.7%, not 48.6%.

    Reported 48.6 · recomputed 24.675325

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Short, CE, 2015, Eur Health Psychol, 17, 32
  2. Vandelanotte, C, 2020, Information Technology Based Methods for Health Behaviours (Part of "Global Telehealth 2019: Studies in Health Technology and Informatics" series), 15
  3. 2003, The Active Australia Survey: A Guide and Manual for Implementation, Analysis and Reporting
  4. 10.2196/mhealth.9912 · DOI

Cited by

Sorted by citations

Related

Authors
To, Q. G., Green, C., & Vandelanotte, C.
Affiliations
Central Queensland University (AU)
Corresponding author
To, Quyen G
ORCID
Venue
JMIR mHealth and uHealth
Publisher
JMIR Publications Inc.
Published
2021-11-26
Volume
9
Issue
11
Pages
e28577
ISSN
2291-5222 (ISSN-L)
DOI
10.2196/28577
PMC
PMC8665384
OpenAlex
W3197562942
Semantic Scholar
e3aeace6e77396dcedcc5a40ad2771739079904c
Cited by
68
References
40
Citations by year
2026: 18 · 2025: 20 · 2024: 7 · 2023: 18 · 2022: 5
FWCI
8.43
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR mhealth and uhealth (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
Funders
  • National Heart Foundation of Australia (ID100427, 100427)
  • Central Queensland University
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
  • Health Sciences › Medicine › Physiology › Physical Activity and Health
Keywords
machine learning, mobile health, wearable sensors, Fitbit, step count monitoring, self-monitoring, motivational messages, Facebook Messenger, usability, acceptability, physical activity promotion, feasibility, user engagement, underachievement
MeSH terms
Machine Learning, Fitness Trackers, Feasibility Studies, Female, Humans, Middle Aged, Motivation, Exercise

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review