CoChat
Sign Up
Cited by 46Medication Adherence and ComplianceMobile Health and mHealth ApplicationsHealth, Technology, Consumer Behavior

Impact of Social Determinants of Health and Demographics on Refill Requests by Medicare Patients Using a Conversational Artificial Intelligence Text Messaging Solution: Cross-Sectional Study

Open access
Brar Prayaga, Rena, Agrawal, Ridhika, Nguyen, Benjamin, Jeong, Erwin W, Noble, Harmony K, Paster, Andrew, Prayaga, Ram S·2019·JMIR mHealth and uHealth
DOI 10.2196/15771PMC PMC6887813
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Nonadherence among patients with chronic disease continues to be a significant concern, and the use of text message refill reminders has been effective in improving adherence. However, questions remain about how differences in patient characteristics and demographics might influence the likelihood of refill using this channel. Objective: The aim of this study was to evaluate the efficacy of an SMS-based refill reminder solution using conversational artificial intelligence (AI; an automated system that mimics human conversations) with a large Medicare patient population and to explore the association and impact of patient demographics (age, gender, race/ethnicity, language) and social determinants of health on successful engagement with the solution to improve refill adherence. Methods: The study targeted 99,217 patients with chronic disease, median age of 71 years, for medication refill using the mPulse Mobile interactive SMS text messaging solution from December 2016 to February 2019. All patients were partially adherent or nonadherent Medicare Part D members of Kaiser Permanente, Southern California, a large integrated health plan. Patients received SMS reminders in English or Spanish and used simple numeric or text responses to validate their identity, view their medication, and complete a refill request. The refill requests were processed by Kaiser Permanente pharmacists and support staff, and refills were picked up at the pharmacy or mailed to patients. Descriptive statistics and predictive analytics were used to examine the patient population and their refill behavior. Qualitative text analysis was used to evaluate quality of conversational AI. Results: Over the course of the study, 273,356 refill reminders requests were sent to 99,217 patients, resulting in 47,552 refill requests (17.40%). This was consistent with earlier pilot study findings. Of those who requested a refill, 54.81% (26,062/47,552) did so within 2 hours of the reminder. There was a strong inverse relationship (r10=−0.93) between social determinants of health and refill requests. Spanish speakers (5149/48,156, 10.69%) had significantly lower refill request rates compared with English speakers (42,389/225,060, 18.83%; X21 [n=273,216]=1829.2; P<.001). There were also significantly different rates of refill requests by age band (X26 [n=268,793]=1460.3; P<.001), with younger patients requesting refills at a higher rate. Finally, the vast majority (284,598/307,484, 92.23%) of patient responses were handled using conversational AI. Conclusions: Multiple factors impacted refill request rates, including a strong association between social determinants of health and refill rates. The findings suggest that higher refill requests are linked to language, race/ethnicity, age, and social determinants of health, and that English speakers, whites, those younger than 75 years, and those with lower social determinants of health barriers are significantly more likely to request a refill via SMS. A neural network–based predictive model with an accuracy level of 78% was used to identify patients who might benefit from additional outreach to narrow identified gaps based on demographic and socioeconomic factors.

AI snapshot

Written from the paper’s full text.

TL;DR

This cross-sectional study evaluated an SMS refill reminder program using conversational AI among 99,217 Medicare Part D patients with chronic disease, finding a 17.40% refill request rate over 273,356 reminders. Refill requests varied significantly by social determinants of health, language, age, and race/ethnicity, and a neural network model predicted non-engagement with 78% accuracy.

Key findings

  • Of 273,356 SMS refill reminders sent to 99,217 patients, 47,552 refill requests were made (17.40%); 54.81% (26,062/47,552) occurred within 2 hours of the reminder.
  • Refill requests were strongly inversely correlated with social determinants of health bands (r=−0.93); Spanish speakers had lower refill request rates (5149/48,156, 10.69%) than English speakers (42,389/225,060, 18.83%; P<.001).
  • Younger patients requested refills at higher rates, with significantly different rates by age band (P<.001); white patients had the highest response rate (34,134/81,544, 41.86%) and Hispanic/Latino patients the lowest (16,700/67,266, 24.83%).
  • Conversational AI handled 92.23% (284,598/307,484) of patient responses; a neural network model identified over 66% of patients requiring outreach at 78% accuracy using a 37% predicted-reply threshold.

Methods

  • Cross-sectional analysis of a 2-year SMS refill reminder program from December 2016 to February 2019 targeting 99,217 English- and Spanish-speaking Medicare Part D patients (median age 71 years) at Kaiser Permanente, Southern California, with diabetes, hypertension, high cholesterol, and/or anticoagulation.
  • Patients received mPulse Mobile interactive SMS reminders in English or Spanish, validated date of birth, viewed medications, and requested refills; pharmacy staff processed requests for pickup or mail.
  • Descriptive statistics, subgroup analyses, an internally developed SDOH index, a neural network multilayer perceptron model (24 input units, 10 hidden units, 1 output unit; 70% training/30% validation), and qualitative coding of 307,484 responses into structured/unstructured categories were used.

Limitations

  • The paper states it was not a randomized controlled trial, with possible selection bias; only patients who opted into digital engagement were targeted, and patients with only landline numbers were excluded.
  • The paper states bivariate analyses can inflate type 1 error, and it did not address whether patients who refilled after a reminder continue the behavior with future reminders.
  • The paper states the commercial system’s financial benefits and costs are proprietary and cannot be disclosed; evident from the text, the observational design cannot establish causality, and the SDOH index was internally developed with unspecified/unknown race/ethnicity for 28.71% of patients.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/15771

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 30 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Patient race/ethnicity had a significant effect on initial response rates (X24 [n=204,857]=5282.40; P<.001).

    “Patient race/ethnicity had a significant effect on initial response rates (X24 [n=204,857]=5282.40; P<.001).”
    Quote in the paperSubstantial claim
  • Findingown data

    Higher SDOH values are correlated with Spanish language preference (point biserial rpb=0.27; P<.001; N=252,696).

    “We used a pointwise biserial correlation (point biserial correlation rpb=0.27; P<.001; N=252,696) to find that higher SDOH values are correlated with Spanish language preference.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The internally developed SDOH index validly captures social determinants of health barriers.
  • Findingown data

    The neural network-based predictive model had an accuracy level of 78%.

    “A neural network–based predictive model with an accuracy level of 78% was used to identify patients who might benefit from additional outreach to narrow identified gaps based on demographic and socioeconomic factors.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Spanish speakers had significantly lower refill request rates (5149/48,156, 10.69%) compared with English speakers (42,389/225,060, 18.83%; X21 [n=273,216]=1829.2; P<.001).

    “Spanish speakers (5149/48,156, 10.69%) had significantly lower refill request rates compared with English speakers (42,389/225,060, 18.83%; X21 [n=273,216]=1829.2; P<.001).”
    • Also stated in: § Part 2: Exploratory Subgroup Analysis · plain
      “Spanish-speaking patients had significantly lower refill request rates (5149/48,156, 10.69%) compared with English-speaking patients (42,389/225,060, 18.83%; X21, [n=273,216]=1829.2; P<.001).”
    Quote in the paperSubstantial claim
  • Findingown data

    The difference in average SDOH between those who reply and do not reply was statistically significant (t 252,834=−55.07; P<.001).

    “The difference in average SDOH between those who reply and do not reply was statistically significant (t 252,834=−55.07; P<.001), but there was no impact of SDOH for refill requests after the patient engages with the initial text message (t 87,234=1.71; P=.09).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The internally developed SDOH index validly captures social determinants of health barriers.
  • May combine two claims. 19
  • Classified as probably not a substantial claim. 28

  • May combine two claims.

    Findingown data

    The older age group (75 years and older) responded at a rate of 29.84%, whereas patients younger than 45 years responded at a rate of 47.81%.

    “The older age group (75 years and older) responded at a rate of 29.84%, whereas patients younger than 45 years responded at a rate of 47.81%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Patients who identified as white responded at the highest rate (34,134/81,544, 41.86%), whereas Hispanic/Latino patients had the lowest response rate (16,700/67,266, 24.83%).

    “Patients who identified as white responded at the highest rate (34,134/81,544, 41.86%), whereas Hispanic/Latino patients had the lowest response rate (16,700/67,266, 24.83%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Refill activity spikes immediately after the initial message, after the 2-hour reminder, and after the 24-hour reminder.

    “As displayed in Figure 3, there are spikes in refill activity immediately after the initial message (“0”), after the 2-hour reminder (“2”), and the 24-hour reminder (“24”).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    After replying to the reminder, about half of the patients continue the conversation to validate their DOB and then request a refill, while the other half drop off and do not engage further.

    “After replying to the reminder, about half of the patients continue the conversation to validate their DOB and then request a refill of their medication(s), while the other half drop off and do not engage further.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Gender did not have a significant moderating effect on refill rates.

    “We present the following subgroup analysis using 4 variables: SDOH, language, race/ethnicity, and age (gender did not have a significant moderating effect on refill rates).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the 2-year study, 273,356 SMS refill reminders were sent to 99,217 Medicare Part D patients, yielding 47,552 refill requests (17.40%).

    “Over the course of the study, 273,356 refill reminders requests were sent to 99,217 patients, resulting in 47,552 refill requests (17.40%).”
    • Also stated in: § Part 1: Analysis of Scaled Program · plain
      “A total of 273,356 SMS reminders were sent over a 2-year period to 99,217 Medicare Part D patients who had opted in to texting. In response, 17.40% (47,552/273,356) refills were requested.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All unstructured responses shown in Table 6 were understood by the CAI engine, which triggered appropriate replies.

    “All unstructured responses shown in Table 6 were understood by the CAI engine, which triggered the appropriate replies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The overall accuracy of handling patient responses by the CAI was over 92%, and almost 8% were not handled at all.

    “In summary, our results indicate that despite the overall high accuracy (over 92%) of handling patient responses by the CAI, there are a number of instances (almost 8%) that were not handled at all”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Pilot refill request rates of 18.1% were closely mirrored (17.40%), or even higher when limited to requests by English speakers (18.83%).

    “Pilot refill request rates of 18.1% were closely mirrored (17.40%), or even higher when limited to requests by English speakers (18.83%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Barriers to refilling include cost, side effects, already refilled, still have sufficient medication, do not want to refill, and taking differently than prescribed.

    “These barriers include a wide range of topics such as cost, side effects, already refilled, still have sufficient medication, do not want to refill, and taking differently than prescribed.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The CAI's categorization of unstructured responses as barriers (e.g., side effect, cost) accurately reflects the patients' actual reasons for not refilling.

  • May combine two claims.

    Findingown data

    Patients who identified as Hispanic/Latino had the lowest refill request rates of 12.95%, followed by Asian (13.81%), black (16.50%), and white (20.30%).

    “Patients who identified as Hispanic/Latino had the lowest refill request rates of 12.95%, followed by Asian (13.81%), black (16.50%), and white (20.30%).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Adherence has a significant effect on treatment outcomes and a major impact in managing chronic conditions such as hypertension, cardiovascular disease, and diabetes.

    “Adherence has been shown to have a significant effect on treatment outcomes [3,4] and has a major impact in managing chronic conditions such as hypertension, cardiovascular disease, and diabetes.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [3] source

      Not checked in the free run.

    • cites [4] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Dispensing or refill data is commonly used to compute adherence levels because of its validity, relative accessibility, and inexpensiveness.

    “Dispensing or refill data is commonly used to compute adherence levels because of the validity, relative accessibility, and inexpensiveness of such data [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The spike in refill requests in the 85+ years group suggests that caregivers or family members might be more actively assisting patients in this age band.

    “We do see a spike in refill requests in the 85+ years group, and this suggests that caregivers or family members might be more actively assisting patients in this age band.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Multiple factors impacted refill request rates, including a strong association between social determinants of health and refill rates.

    “Multiple factors impacted refill request rates, including a strong association between social determinants of health and refill rates.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The internally developed SDOH index validly captures social determinants of health barriers.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The observed 17.40% refill request rate was consistent with earlier pilot study findings.

    “This was consistent with earlier pilot study findings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Medication nonadherence is associated with a range of patient-related, physician-related, and health system barriers.

    “There are a range of factors, including patient-related, physician-related, and health system barriers, that contribute to nonadherence [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] de Geest, S, 2003, Adherence to Long-term Therapies: Evidence for Action

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The predictive model was trained on 70% of the available data and validated on the remaining 30%.

    “The training set consisted of 70% of the available data, and model validation was done with the remaining 30%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In a study of patients with chronic disease, two-thirds of those who reported not taking medications as prescribed due to cost never shared this with their physician.

    “In a study of patients with chronic disease, two-thirds of those who reported not taking medications as prescribed due to cost never shared this with their physician [30,31].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [30] source

      Not checked in the free run.

    • cites [31] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisecitation

    Text messaging presents an effective, affordable, and scalable tool that can use conversational AI to greatly impact health outcomes.

    “Text messaging presents an effective, affordable, and scalable tool [21] that can use conversational AI to greatly impact health outcomes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several meta-analyses corroborate the effectiveness of SMS for medication adherence.

    “Several meta-analyses corroborate the effectiveness of SMS for medication adherence [18-20].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [18] source

      Not checked in the free run.

    • cites [19] source

      Not checked in the free run.

    • cites [20] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Conversational AI can encourage health care consumers to engage with systems that imitate human conversations using text.

    “More specifically, conversational AI (or conversational agents) can encourage health care consumers to engage with systems that imitate human conversations using text [23,24].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [23] source

      Not checked in the free run.

    • cites [24] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The World Health Organization estimates a medication nonadherence rate of 50% for patients with 1 or more chronic conditions.

    “The World Health Organization estimates a medication nonadherence rate of 50% for patients with 1 or more chronic conditions [2,7].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [2] de Geest, S, 2003, Adherence to Long-term Therapies: Evidence for Action

      Not checked in the free run.

    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Two coders classified each conversational AI rule, and any ambiguity about rule type was resolved by discussion and agreement between the two coders.

    “Two coders classified each conversational AI rule using the description within the solution (eg, DOB validation and change order). There were a total of 75 conversational AI rules. These individual rules were then combined into 13 broader “rule type” categories: initial reply, refill process, change pharmacy, DOB validation, barriers, payment, subscription, language change, member information, ack”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In a recent National Health Interview Survey study, half of adults with diabetes perceived financial stress, while one-fifth reported financial insecurity and food insecurity.

    “Social determinants have also been linked to nonadherence, and a recent study using data from the National Health Interview Survey found that half of the adults with diabetes perceived financial stress, while one-fifth reported financial insecurity and food insecurity [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Subsidy recipients and non-English speakers were significantly less likely to be counseled about drug side effects.

    “Finally, subsidy recipients and non-English speakers were significantly less likely to be counseled about drug side effects [12].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    An earlier study by the authors found that text reminders increased refill rates by 14 percentage points compared with those who did not receive these reminders.

    “An earlier study by the authors [21] measured the impact of SMS text reminders on refill rates of nonadherent and partially adherent Medicare patients with chronic disease. They found that text reminders increased refill rates by 14 percentage points compared with those who did not receive these reminders.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A fully automated conversational AI system has been used to promote weight loss among overweight and obese diabetic patients.

    “For example, a fully automated conversational AI system has been used to promote weight loss among overweight and obese diabetic patients [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In a 2018 multinational survey, 77% of respondents said the ability to request prescription refills via text message would increase their likelihood of choosing a health care provider.

    “Furthermore, in a multinational survey conducted in 2018, 77% of respondents said the ability to request prescription refills via text message would increase their likelihood of choosing a health care provider.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] SafaviKWebbKKalisBAccenture20192019-06-13Accenture 2019 Digital Health Consumer Survey. https://www.accenture.com/_acnmedia/PDF-94/Accenture-2019-Digital-Health-Consumer-Survey.pdf

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The VEText system requires users to only use prompted and structured alphanumeric responses, such as R2 and K3.

    “The system requires users to only use prompted and structured alphanumeric responses, such as R2 and K3 [17].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] Veterans Affairs20192019-10-07VEText FAQs. https://www.va.gov/HEALTH/VEText_FAQs.asp

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Complex conversational AI was not essential for driving outcomes (validating DOB, completing refill), but it played an important role in keeping the conversation going when patients engaged using unprompted or unstructured messages.

    “Although complex conversational AI was not essential for driving outcomes (validating DOB, completing refill, etc), it played an important role in keeping the conversation going when patients engaged using unprompted or unstructured messages”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study was not a randomized controlled trial.

    “The study was not a randomized controlled trial”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Patients with only landline numbers were excluded.

    “patients with only landline numbers were excluded.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    There is the possibility of selection bias.

    “there is the possibility of selection bias”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Using bivariate analyses can inflate the type 1 error.

    “Using bivariate analyses can inflate the type 1 error, and this is a limitation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study did not address the impact of multiple reminders over time.

    “we did not address the impact of multiple reminders over time”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A review of the literature using conversational AI reveals that the focus area is limited in scale and tends to be restricted to apps and not SMS.

    “A review of the literature using conversational AI reveals that the focus area is limited in scale and tends to be restricted to apps [24] and not SMS.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The study attempted to address the type 1 error issue by using a neural network predictive model, which is a form of multivariate regression and can reduce the impact of multicollinearity.

    “we attempted to address this issue by using a neural network predictive model, which is a form of multivariate regression and can reduce the impact of multicollinearity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation

    The model of primarily relying on prompted responses and also understanding unstructured responses allows the user to control the flow of the conversation in keeping with Grice's Maxim of Manner.

    “This model of primarily relying on prompted responses and also understanding those cases where users want to go outside a closed response set allows the user to control the flow of the conversation in keeping with Grice’s Maxim of Manner [34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34] Grice, HP, 1975, Speech Acts source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The value and incremental benefit of SMS text reminders as compared with phone reminders has already been demonstrated.

    “we have already demonstrated the value and incremental benefit of SMS text reminders as compared with phone reminders [21]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    These differences may contribute to health disparities and impact health outcomes in Medicare patients.

    “these differences may contribute to health disparities and impact health outcomes in Medicare patients.”
    Quote in the paperNot a substantial claim
41 of 41

  • The paper relies on this without support.

    There is limited research currently available on the use of conversational AI within SMS (and not app-based) messaging for refill adherence.

    “However, there is limited research currently available on the use of conversational AI within SMS (and not app-based) messaging for refill adherence.”

  • The paper relies on this without support.

    There is still limited research studying the specific ways in which SDOH pathways interact to impact adherence, particularly in older populations.

    “but there is still limited research studying the specific ways in which SDOH pathways interact to impact adherence, particularly in older populations.”

  • The paper relies on this without support.

    SDOH are not always recognized and might be overlooked by a clinician in a medical setting.

    “Since SDOH are not always recognized, they might be overlooked by a clinician in a medical setting.”

  • The paper relies on this without support.

    The internally developed SDOH index validly captures social determinants of health barriers.

    “An internally developed SDOH index was used to understand how unmet needs might impact patient refill behavior. When patient address was available, an SDOH index was computed. Multimedia Appendix 1 outlines the factors that were used to compute the SDOH index and to create low, medium, and high SDOH clusters.”

  • The paper relies on this without support.

    Refill requests via SMS are a valid proxy for medication refill adherence.

    “We found a strong association between refill request rates and patient language, age, race/ethnicity, and SDOH levels, and these differences may contribute to health disparities and impact health outcomes in Medicare patients.”

  • The paper relies on this without support.

    Patients who do not reply to the SMS reminder would not request a refill without additional outreach.

    “we can identify over 66% of those requiring outreach (because they will not reply)”

  • The paper relies on this without support.

    The CAI's categorization of unstructured responses as barriers (e.g., side effect, cost) accurately reflects the patients' actual reasons for not refilling.

    “These barriers include a wide range of topics such as cost, side effects, already refilled, still have sufficient medication, do not want to refill, and taking differently than prescribed.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Among non-RCT studies, Maher et al and Brar Prayaga et al reported highest compliance (5/9, 56%), followed by Masaki et al, Prochaska et al, and To et al (4/9, 44%).

    “Among the non-RCT studies, the studies by Maher et al [22] and Brar Prayaga et al [30] reported the highest compliance (5/9, 56%), followed closely by the studies by Masaki et al [25], Prochaska et al [31], and To et al [32] (4/9, 44%).”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [30]

    The Discussion’s Limitations section (page unavailable) explicitly says this study was not a randomized controlled trial, supporting its classification as a non-RCT. The paper does not report a 5/9 (56%) compliance score or compare its compliance with the other named studies; that appears to be the citing authors’ assessment. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Of the 15 studies, 3 (20%) targeted medication or treatment adherence, but only 2 (67%) reported efficacy of AI chatbots in increasing treatment or medication adherence through timely and personalized reminders.

    “Out of the 15 studies, 3 (20%) studies [6,26,30] targeted medication or treatment adherence, but only 2 (67%) of these studies [26,30] reported the efficacy of AI chatbots in increasing treatment or medication adherence through timely and personalized reminders.”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [30]

    The Methods (Procedure) describe conversational-AI text messages with refill reminders, and the Results (Part 1: Analysis of Scaled Program) report that 17.40% of reminders led to refill requests. But the study measures refill requests, not an increase in medication-taking or treatment adherence attributable to the chatbot; its Limitations state that it was not a randomized controlled trial. The 15-study and 2-of-3 counts are the citing review’s own summary, not findings attributed to this individual study. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Brar Prayaga et al reported that out of total refill reminders (n=273,356), 17.4% (n=47,552) resulted in actual refill requests, and 54.81% (26,062/47,552) of those requests resulted in medications being actually refilled within 2 hours.

    “Brar Prayaga et al [30] reported that out of the total refill reminders (n=273,356), 17.4% (n=47,552) resulted in actual refill requests. Furthermore, 54.81% (26,062/47,552) of those requests resulted in medications being actually refilled within 2 hours.”

    From Artificial Intelligence-Based Chatbots for Promoting Health Behavioral Changes: Systematic Review. (2023) · cites this paper as [30]

    Results, Part 1: Analysis of Scaled Program reports 273,356 reminders and 47,552 refill requests (17.40%). It says 26,062 of those requests were made within two hours of the reminder, not that the medications were actually refilled within two hours. No page number is provided in our copy. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Incorrect percentage for patients with unspecified race/ethnicity

    Minor

    The listed 27,492 patients are 27.71% of the stated total of 99,217, not 28.71%.

    Reported 28.71 · recomputed 27.708961

  • Age-table response counts exceed their reported total

    Minor

    The eight age-band response counts sum to 210,121, not 95,121. The 75-80-year row alone lists 127,732 responses for 42,221 dialogues.

    Reported 95121 · recomputed 210121

  • Survey percentage uses a different denominator

    Minor

    The text says 575 people completed the survey. Of those respondents, 406 are 70.61%, not 69.2%.

    Reported 69.2 · recomputed 70.608696

  • Same age-band test has different reported degrees of freedom

    Minor

    The abstract and Results give the same age-band test statistic and sample size but report different degrees of freedom: 6 versus 7.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. 2016, A Framework to Address the Social Determinants of Health

Cited by

Sorted by citations

Related

Authors
Brar Prayaga, R., Agrawal, R., Nguyen, B., Jeong, E. W., Noble, H. K., Paster, A., & Prayaga, R. S.
Affiliations
  • Grinnell College (US)
  • Kaiser Permanente (US)
  • EnZinc (United States) (US)
ORCID
Venue
JMIR mHealth and uHealth
Publisher
JMIR Publications Inc.
Published
2019-11-18
Volume
7
Issue
11
Pages
e15771
ISSN
2291-5222 (ISSN-L)
DOI
10.2196/15771
PMC
PMC6887813
OpenAlex
W2987778734
Cited by
45
References
24
Citations by year
2026: 6 · 2025: 9 · 2024: 5 · 2023: 14 · 2022: 4 · 2021: 4 · 2020: 3
FWCI
2.83
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR mhealth and uhealth (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Family Practice › Medication Adherence and Compliance
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Health Sciences › Health Professions › Health Information Management › Health, Technology, Consumer Behavior
Keywords
medication adherence, conversational AI, safety management systems, social determinants of health, Medicare Part D, chronic disease, language barriers, Spanish speakers, race and ethnicity, gender differences, patient demographics, Kaiser Permanente, patient engagement, integrated health plans
MeSH terms
Aged, Artificial Intelligence, California, Cross-Sectional Studies, Demography, Female, Medicare, Humans, Male, Middle Aged, Pilot Projects, United States, Drug Monitoring, Qualitative Research, Medication Adherence, Text Messaging, Social Determinants of Health

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review