The peril of self-reported adherence in digital interventions: A brief example
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Adherence is an important predictor of intervention outcomes, but not all measures of adherence are created equally. Here, we analyzed whether there was a discrepancy between self-report adherence and objective adherence in a digital mindfulness meditation randomised, controlled trial. A sample of 174 young adult undergraduate university students trialled either an app-based or email-based mindfulness meditation program (or an app-based attention control). Participants' adherence (number of sessions completed) and mental health was self-reported. Objective adherence data were provided by the owners of the digital mindfulness programs. We found evidence of inflated self-reported adherence to the app-based intervention and argue that the inflation was not explained by social desirability biases because participants were aware we would have access to object data and no remuneration was tied to adherence. We also comment on the different conclusions we would have drawn about the effectiveness of the digital interventions on mental health, had we used the self-reported adherence data rather than the objective adherence data. We use this example to suggest that it may be perilous to rely on self-reported measures of adherence when assessing the effectiveness of digital interventions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Effect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860
Self-reports: How the questions shape the answers.
Schwarz, N. (1999) · American Psychologist · doi:10.1037/0003-066x.54.2.93 · Cited by 2,846 · cited 1×
“Although self-reported adherence data are easily they may be subject to biases that affect all self-report data (e.g., recall bias and response bias; Kimberlin and Winterstein, 2008; Schwarz, 1999), which may lead to inaccurate conclusions about the effectiveness of the intervention.”
BackgroundEysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 1×
“One explanation for these modest effects might be that adherence to digital interventions is low (Cuijpers et al., 2011; Eysenbach, 2005).”
BackgroundPatient Adherence and Medical Treatment Outcomes: A Meta-Analysis
Robin DiMatteo, M., Giordani, P. J., Lepper, H. S., & Croghan, T. W. (2002) · Medical Care · doi:10.1097/00005650-200209000-00009 · Cited by 1,490
Internet-Based and Other Computerized Psychological Treatments for Adult Depression: A Meta-Analysis
Andersson, G., & Cuijpers, P. (2009) · Cognitive Behaviour Therapy · doi:10.1080/16506070903318960 · Cited by 1,368 · cited 2×
“…2009 and 2015, yearly publications on e-mental health interventions trebled (Firth et al., 2016), but meta-analytic reviews reveal that self-guided digital interventions often have only modest effects on mental health (Andersson and Cuijpers, 2009; Cuijpers et al., 2011; Spijkerman et al., 2016).”
BackgroundPersuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 2×
“…were skills acquired), nor do they distinguish between observed adherence (how much the individual experienced the content of the intervention) and prescribed adherence (how much the individual experienced the intervention as recommended or intended; Kelders et al., 2012; Sieverink et al., 2017).”
BackgroundAdherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · cited 3×
“To date, the majority of research on adherence in digital interventions has focused on operationalizing adherence and identifying predictors of adherence (see: Christensen et al., 2009) but the complexity of adherence is often neglected (see: Sieverink et al., 2017 for a systematic review).”
BackgroundSpijkerman, M. P. J., Pots, W. T. M., & Bohlmeijer, E. T. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.009 · Cited by 975
Self-report measures of medication adherence behavior: recommendations on optimal use
Stirratt, M. J., Dunbar-Jacob, J., Crane, H. M., Simoni, J. M., Czajkowski, S., Hilliard, M. E., Aikens, J. E., Hunter, C. M., Velligan, D. I., Huntley, K., et al. (2015) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0315-2 · Cited by 824
Karyotaki, E., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., Mira, A., Mackinnon, A., Meyer, B., Botella, C., Littlewood, E., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0044 · Cited by 691
Parsons, C. E., Crane, C., Parsons, L. J., Fjorback, L. O., & Kuyken, W. (2017) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.05.004 · Cited by 550 · cited 2×
“To be optimally effective, regular practice is considered a key component of mindfulness-based interventions (Segal et al., 2013) and adherence to practice guidelines is correlated with intervention outcomes (meta-analysis: k = 28, r = 0.264, p < .001; Parsons et al., 2017).”
BackgroundResultInfluentialDisclosure of sensitive behaviors across self-administered survey modes: a meta-analysis
Gnambs, T., & Kaspar, K. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0533-4 · Cited by 422 · cited 1×
“…data by using research designs like experience sampling or daily diaries to reduce recall time (Schwarz, 2012) or by nonjudgmentally acknowledging normality of non-adherence2 and anonymizing online reports of sensitive topics to reduce socially desirable responding (Gnambs and Kaspar, 2015).”
MethodSmartphone usage in the 21st century: who is active on WhatsApp?
Montag, C., Błaszkiewicz, K., Sariyska, R., Lachmann, B., Andone, I., Trendafilov, B., Eibes, M., & Markowetz, A. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1280-z · Cited by 408 · cited 1×
“Given that young adults spend on average between 2 and 4 h per day on their mobile phones (Montag et al., 2015, sample primarily from Germany; Liao et al., In Review, sample from New Zealand) and more than two-thirds of global internet use in 2017 was completed on mobile devices rather than laptops…”
BackgroundHowells, A., Ivtzan, I., & Eiroa-Orosa, F. J. (2016) · Journal of Happiness Studies · doi:10.1007/s10902-014-9589-1 · Cited by 407
Cavanagh, K., Strauss, C., Forder, L., & Jones, F. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.01.001 · Cited by 340 · cited 1×
“Consistent with previous mindfulness-based selfhelp interventions (Cavanagh et al., 2014), attrition rates were low (3.”
ResultFlett, J. A. M., Hayne, H., Riordan, B. C., Thompson, L. M., & Conner, T. S. (2019) · Mindfulness · doi:10.1007/s12671-018-1050-9 · Cited by 338
Self-citationSieverink, F., Kelders, S. M., & van Gemert-Pijnen, J. E. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8578 · Cited by 331 · cited 5×
“Access to the interventions followed a hybrid structure whereby the interventions involved fixed core content with additional optional components (Sieverink et al., 2017); app-users had to complete the first 10 sessions consecutively in order to ‘unlock’ other intervention content, whereas…”
BackgroundMethodInfluentialBrief mindfulness training for negative affectivity: A systematic review and meta-analysis.
Schumer, M. C., Lindsay, E. K., & Creswell, J. D. (2018) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000324 · Cited by 316
Self-guided psychological treatment for depressive symptoms: a meta-analysis
Cuijpers, P., Donker, T., Johansson, R., Mohr, D. C., van Straten, A., & Andersson, G. (2011) · PLoS ONE · doi:10.1371/journal.pone.0021274 · Cited by 250 · cited 2×
“One explanation for these modest effects might be that adherence to digital interventions is low (Cuijpers et al., 2011; Eysenbach, 2005).”
BackgroundBlanck, P., Perleth, S., Heidenreich, T., Kröger, P., Ditzen, B., Bents, H., & Mander, J. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2017.12.002 · Cited by 248 · cited 1×
“But, a linear dose-response relationship between adherence and outcomes is not always the case (e.g., Donkin et al., 2013; Blanck et al., 2018).”
BackgroundDonkin, L., Hickie, I. B., Christensen, H., Naismith, S. L., Neal, B., Cockayne, N. L., & Glozier, N. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2771 · Cited by 193
Gamification and Adherence to Web-Based Mental Health Interventions: A Systematic Review
Brown, M., O'Neill, N., van Woerden, H., Eslambolchilar, P., Jones, M., & John, A. (2016) · JMIR Mental Health · doi:10.2196/mental.5710 · Cited by 178 · cited 1×
“But, few digital interventions report adherence rates, and even fewer report how adherence relates to intervention outcomes (Brown et al., 2016; Donkin et al., 2011).”
BackgroundComparing usage of a web and app stress management intervention: An observational study
Morrison, L. G., Geraghty, A. W. A., Lloyd, S., Goodman, N., Michaelides, D. T., Hargood, C., Weal, M., & Yardley, L. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.03.006 · Cited by 57
Research works printed but not linked: 4
- Donkin, 2011, A systematic review of the impact of adherence on the effectiveness of e-therapies, J. Med. Internet Res., 13, 10.2196/jmir.1772 · DOI · Paper page
- Elefant, 2017, Microinterventions produce immediate but not lasting benefits in mood and distress, Internet Interv., 10, 17, 10.1016/j.invent.2017.08.004 · DOI
- Firth, 2016, Ecological momentary assessment and beyond: the rising interest in e-mental health research, J. Psychiatr. Res., 80, 3, 10.1016/j.jpsychires.2016.05.002 · DOI · Paper page
- Kimberlin, 2008, Validity and reliability of measurement instruments used in research, Am. J. Health Syst. Pharm., 65, 2276, 10.2146/ajhp070364 · DOI · Paper page
Cited by
Taylor, H., Strauss, C., & Cavanagh, K. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102078 · Cited by 102
Levin, M. E., Hicks, E. T., & Krafft, J. (2022) · Journal of American College Health · doi:10.1080/07448481.2020.1728281 · Cited by 65
Johnson, E., Corrick, S., Isley, S., Vandermeer, B., Dolgoy, N., Bates, J., Godfrey, E., Soltys, C., Muir, C., Vohra, S., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000435 · Cited by 24
“Survey completion and self-reports were the most common markers, despite their limitations like recall bias, overestimation, and the possibility that participants may feel clinical improvement and stop using the intervention [140].”
BackgroundEconomides, M., Bolton, H., Male, R., & Cavanagh, K. (2022) · JMIR Formative Research · doi:10.2196/34032 · Cited by 19
“This is important, given the recent evidence of substantial discrepancies between self-reported and objective adherence in digital interventions [68].”
Taylor, R. W., Male, R., Economides, M., Bolton, H., & Cavanagh, K. (2023) · JMIR Formative Research · doi:10.2196/41590 · Cited by 15
“Intervention engagement was measured objectively, which is important given the evidence of inflated rates of self-reported digital intervention use in a study setting [72].”
Metadata
- Authors
- Flett, J. A. M., Fletcher, B. D., Riordan, B. C., Patterson, T., Hayne, H., & Conner, T. S.
- Venue
- Internet Interventions
- Publisher
- Elsevier BV
- Published
- 2019-12
- Volume
- 18
- Pages
- 100267
- ISSN
- 2214-7829 (ISSN-L)
- OpenAlex
- W2969414414
- Cited by
- 37
- References
- 23
- FWCI
- 3.22
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
- Funders
- University of Otago
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.