Data Missing Not at Random in Mobile Health Research: Assessment of the Problem and a Case for Sensitivity Analyses
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Background: Missing data are common in mobile health (mHealth) research. There has been little systematic investigation of how missingness is handled statistically in mHealth randomized controlled trials (RCTs). Although some missing data patterns (ie, missing at random [MAR]) may be adequately addressed using modern missing data methods such as multiple imputation and maximum likelihood techniques, these methods do not address bias when data are missing not at random (MNAR). It is typically not possible to determine whether the missing data are MAR. However, higher attrition in active (ie, intervention) versus passive (ie, waitlist or no treatment) conditions in mHealth RCTs raise a strong likelihood of MNAR, such as if active participants who benefit less from the intervention are more likely to drop out. Objective: This study aims to systematically evaluate differential attrition and methods used for handling missingness in a sample of mHealth RCTs comparing active and passive control conditions. We also aim to illustrate a modern model-based sensitivity analysis and a simpler fixed-value replacement approach that can be used to evaluate the influence of MNAR. Methods: We reanalyzed attrition rates and predictors of differential attrition in a sample of 36 mHealth RCTs drawn from a recent meta-analysis of smartphone-based mental health interventions. We systematically evaluated the design features related to missingness and its handling. Data from a recent mHealth RCT were used to illustrate 2 sensitivity analysis approaches (pattern-mixture model and fixed-value replacement approach). Results: Attrition in active conditions was, on average, roughly twice that of passive controls. Differential attrition was higher in larger studies and was associated with the use of MAR-based multiple imputation or maximum likelihood methods. Half of the studies (18/36, 50%) used these modern missing data techniques. None of the 36 mHealth RCTs reviewed conducted a sensitivity analysis to evaluate the possible consequences of data MNAR. A pattern-mixture model and fixed-value replacement sensitivity analysis approaches were introduced. Results from a recent mHealth RCT were shown to be robust to missing data, reflecting worse outcomes in missing versus nonmissing scores in some but not all scenarios. A review of such scenarios helps to qualify the observations of significant treatment effects. Conclusions: MNAR data because of differential attrition are likely in mHealth RCTs using passive controls. Sensitivity analyses are recommended to allow researchers to assess the potential impact of MNAR on trial results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
mice: Multivariate Imputation by Chained Equations in R
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870
A test of missing completely at random for multivariate data with missing values
Little, R. J. A. (1988) · Journal of the American Statistical Association · doi:10.1080/01621459.1988.10478722 · Cited by 8,963 · cited 1×
“Some readers may be familiar with Little [24] MCAR test, which is designed to evaluate the likelihood of MCAR across a data set.”
BackgroundMissing data analysis: making it work in the real world
Graham, J. W. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085530 · Cited by 6,163
Beta blockade during and after myocardial infarction: An overview of the randomized trials
Yusuf, S., Peto, R., Lewis, J., Collins, R., & Sleight, P. (1985) · Progress in Cardiovascular Diseases · doi:10.1016/s0033-0620(85)80003-7 · Cited by 3,205 · cited 1×
“As ORs and the variance of ORs cannot be computed for cells with zeros, we conducted analyses using the Peto method [28], as recommended in the Cochrane handbook [29].”
MethodEysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 4×
“This law of attrition [1] applies both to the use of mHealth interventions in naturalistic settings (eg, internet-based interventions for anxiety and depression [2]) as well as to studies designed to test the efficacy of mHealth interventions (eg, randomized controlled trials [RCTs] of smartphone-based interventions f…”
BackgroundInfluentialThe prevention and treatment of missing data in clinical trials
Little, R. J., D'Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., Frangakis, C., Hogan, J. W., Molenberghs, G., Murphy, S. A., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmsr1203730 · Cited by 1,884 · cited 1×
“Another future direction is intentionally adopting methods that decrease attrition generally [85], given that differential attrition and associated MNAR data are less concerning when the amount of missing data is small.”
Doing Meta-Analysis with R: A Hands-On Guide
Harrer, M., Cuijpers, P., Furukawa, T. A., & Ebert, D. D. (2021) · doi:10.1201/9781003107347 · Cited by 1,850
What to add to nothing? Use and avoidance of continuity corrections in meta‐analysis of sparse data
J. Sweeting, M., J. Sutton, A., & C. Lambert, P. (2004) · Statistics in Medicine · doi:10.1002/sim.1761 · Cited by 1,734 · cited 1×
“5 added to all cells in a study with an empty cell [30]).”
BackgroundHow Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294 · cited 1×
“This lack of association could be because of the limited statistical power for the moderator test itself [77], as only 11 studies tested for differential attrition.”
BackgroundMoltu, C., Stefansen, J., Svisdahl, M., & Veseth, M. (2012) · Disability and Rehabilitation · doi:10.3109/09638288.2012.656792 · Cited by 1,247
Adherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · cited 4×
“This law of attrition [1] applies both to the use of mHealth interventions in naturalistic settings (eg, internet-based interventions for anxiety and depression [2]) as well as to studies designed to test the efficacy of mHealth interventions (eg, randomized controlled trials [RCTs] of smartphone-based interventions f…”
BackgroundInfluentialFirth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097 · cited 1×
“Although several meta-analyses suggest that smartphone-based mental health interventions produce benefits relative to waitlist control conditions [78-80], the lack of sensitivity analyses coupled with literature-wide differential attrition makes the apparent efficacy more tenuous.”
BackgroundLinardon, J., Cuijpers, P., Carlbring, P., Messer, M., & Fuller‐Tyszkiewicz, M. (2019) · World Psychiatry · doi:10.1002/wps.20673 · Cited by 860
Firth, J., Torous, J., Nicholas, J., Carney, R., Rosenbaum, S., & Sarris, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.04.046 · Cited by 675 · cited 1×
“Although several meta-analyses suggest that smartphone-based mental health interventions produce benefits relative to waitlist control conditions [78-80], the lack of sensitivity analyses coupled with literature-wide differential attrition makes the apparent efficacy more tenuous.”
BackgroundLinardon, J., & Fuller-Tyszkiewicz, M. (2020) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000459 · Cited by 573
Mindfulness on-the-go: Effects of a mindfulness meditation app on work stress and well-being
Bostock, S., Crosswell, A. D., Prather, A. A., & Steptoe, A. (2019) · Journal of Occupational Health Psychology · doi:10.1037/ocp0000118 · Cited by 465
Differential dropout and bias in randomised controlled trials: when it matters and when it may not
Bell, M. L., Kenward, M. G., Fairclough, D. L., & Horton, N. J. (2013) · BMJ · doi:10.1136/bmj.e8668 · Cited by 460
Pratap, A., Neto, E. C., Snyder, P., Stepnowsky, C., Elhadad, N., Grant, D., Mohebbi, M. H., Mooney, S., Suver, C., Wilbanks, J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0224-8 · Cited by 410
Heber, E., Ebert, D. D., Lehr, D., Cuijpers, P., Berking, M., Nobis, S., & Riper, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.5774 · Cited by 408
The Use and Effectiveness of Mobile Apps for Depression: Results From a Fully Remote Clinical Trial
Arean, P. A., Hallgren, K. A., Jordan, J. T., Gazzaley, A., Atkins, D. C., Heagerty, P. J., & Anguera, J. A. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.6482 · Cited by 376
Ly, K. H., Ly, A.-M., & Andersson, G. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.10.002 · Cited by 368
Ly, K. H., Trüschel, A., Jarl, L., Magnusson, S., Windahl, T., Johansson, R., Carlbring, P., & Andersson, G. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-003440 · Cited by 269
Roepke, A. M., Jaffee, S. R., Riffle, O. M., McGonigal, J., Broome, R., & Maxwell, B. (2015) · Games for Health Journal · doi:10.1089/g4h.2014.0046 · Cited by 265 · cited 2×
“Yes No MLM Yes, higher in active 57 93 152 190 Roepke et al [65]”
BackgroundTighe, J., Shand, F., Ridani, R., Mackinnon, A., De La Mata, N., & Christensen, H. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-013518 · Cited by 227 · cited 2×
“No No ANOVA N/A 0 30 2 31 Tighe et al [70]”
BackgroundChampion, L., Economides, M., & Chandler, C. (2018) · PLOS ONE · doi:10.1371/journal.pone.0209482 · Cited by 225 · cited 4×
“Yes Yes MLMh No 3 36 9 38 Champion et al [44]”
BackgroundInfluentialA randomized controlled trial of a smartphone app for posttraumatic stress disorder symptoms.
Kuhn, E., Kanuri, N., Hoffman, J. E., Garvert, D. W., Ruzek, J. I., & Taylor, C. B. (2017) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000163 · Cited by 225
Schlosser, D. A., Campellone, T. R., Truong, B., Etter, K., Vergani, S., Komaiko, K., & Vinogradov, S. (2018) · Schizophrenia Bulletin · doi:10.1093/schbul/sby078 · Cited by 206
Blankers, M., Koeter, M. W. J., & Schippers, G. M. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1448 · Cited by 194 · cited 2×
“types of missing data and methods for handling them, interested readers are directed to Enders [8] and Graham [9]; for a tutorial specifically geared to nonstatistician mHealth researchers, refer to Blankers et al [10]).”
MethodMiner, A., Kuhn, E., Hoffman, J. E., Owen, J. E., Ruzek, J. I., & Taylor, C. B. (2016) · Psychological Trauma: Theory, Research, Practice, and Policy · doi:10.1037/tra0000092 · Cited by 185 · cited 2×
“No Yes ANOVA N/A 3 24 2 25 Miner et al [60]”
BackgroundLitvin, S., Saunders, R., Maier, M. A., & Lüttke, S. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237220 · Cited by 177
A randomized controlled trial of three smartphone apps for enhancing public mental health
Bakker, D., Kazantzis, N., Rickwood, D., & Rickard, N. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.08.003 · Cited by 176 · cited 3×
“MI was used in 31% (11/36) of studies, with multiple imputed data sets then analyzed using analysis of variance or t tests [5,42].”
BackgroundMethodSmartphone data as an electronic biomarker of illness activity in bipolar disorder
Faurholt‐Jepsen, M., Vinberg, M., Frost, M., Christensen, E. M., Bardram, J. E., & Kessing, L. V. (2015) · Bipolar Disorders · doi:10.1111/bdi.12332 · Cited by 169 · cited 2×
“Maximum likelihood Multiple imputation MethodeDiffdWL drop WLc ITT Tx dropTxa ITTb Study NoYesANOVAgN/Af2578146234Bakker et al [42] NoYest testsYes, higher in active88195106192Bidargaddi et al [5] NoNoANOVAN/A41105128Bostock et al [43] N/AN/AANOVAN/A018020Carissoli et al [40] YesYesMLMhNo336938Champion et al [44] YesN…”
BackgroundIvanova, E., Lindner, P., Ly, K. H., Dahlin, M., Vernmark, K., Andersson, G., & Carlbring, P. (2016) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2016.09.012 · Cited by 165 · cited 2×
“Maximum likelihood Multiple imputation MethodeDiffdWL drop WLc ITT Tx dropTxa ITTb Study NoYesANOVAgN/Af2578146234Bakker et al [42] NoYest testsYes, higher in active88195106192Bidargaddi et al [5] NoNoANOVAN/A41105128Bostock et al [43] N/AN/AANOVAN/A018020Carissoli et al [40] YesYesMLMhNo336938Champion et al [44] YesN…”
BackgroundHorsch, C. H., Lancee, J., Griffioen-Both, F., Spruit, S., Fitrianie, S., Neerincx, M. A., Beun, R. J., & Brinkman, W.-P. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6524 · Cited by 164
The Law of Attrition Revisited
Christensen, H., & Mackinnon, A. (2006) · Journal of Medical Internet Research · doi:10.2196/jmir.8.3.e20 · Cited by 161
Evaluation of an mHealth App (DeStressify) on University Students’ Mental Health: Pilot Trial
Lee, R. A., & Jung, M. E. (2018) · JMIR Mental Health · doi:10.2196/mental.8324 · Cited by 138 · cited 2×
“No No ANOVA N/A 18 104 25 102 Lee and Jung [54]”
Differential attrition in health behaviour change trials: a systematic review and meta-analysis
Crutzen, R., Viechtbauer, W., Spigt, M., & Kotz, D. (2015) · Psychology & Health · doi:10.1080/08870446.2014.953526 · Cited by 132 · cited 1×
“Although MNAR cannot be assessed directly, we might speculate that participants in the active condition who did not benefit as much from the smartphone-based intervention may be more likely to drop out of the study (refer to Crutzen et al [13] for similar possibilities offered to explain higher attrition in treatment…”
BackgroundProudfoot, J. (2013) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867412471441 · Cited by 126 · cited 2×
“Yes Yes MLM Yes, higher in active 32 230 116 242 Proudfoot et al [64]”
BackgroundLüdtke, T., Pult, L. K., Schröder, J., Moritz, S., & Bücker, L. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.08.113 · Cited by 122 · cited 3×
“In this context, it is common for posttreatment depression scores to be missing for a subset of participants [11].”
BackgroundRosen, K. D., Paniagua, S. M., Kazanis, W., Jones, S., & Potter, J. S. (2018) · Psycho-Oncology · doi:10.1002/pon.4764 · Cited by 106
Pham, Q., Khatib, Y., Stansfeld, S., Fox, S., & Green, T. (2016) · Games for Health Journal · doi:10.1089/g4h.2015.0033 · Cited by 104 · cited 2×
“No No ANOVA N/A 7 32 14 31 Pham et al [63]”
BackgroundStolz, T., Schulz, A., Krieger, T., Vincent, A., Urech, A., Moser, C., Westermann, S., & Berger, T. (2018) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000301 · Cited by 102
Efficacy of a Mindfulness-Based Mobile Application: a Randomized Waiting-List Controlled Trial
van Emmerik, A. A. P., Berings, F., & Lancee, J. (2018) · Mindfulness · doi:10.1007/s12671-017-0761-7 · Cited by 101 · cited 2×
“Unclear Yes MLM Yes, higher in active 45 186 111 191 van Emmerik et al [71]”
BackgroundGoldberg, S. B., Imhoff-Smith, T., Bolt, D. M., Wilson-Mendenhall, C. D., Dahl, C. J., Davidson, R. J., & Rosenkranz, M. A. (2020) · JMIR Mental Health · doi:10.2196/23825 · Cited by 97 · cited 9×
“It is worth noting that the 2 sensitivity analysis approaches illustrated in this study provided somewhat discrepant conclusions regarding the degree to which data from Goldberg et al [27] were robust to MNAR conditions.”
BackgroundMethodResultInfluentialSelf-citationGoldberg, S. B., Imhoff-Smith, T., Bolt, D. M., Wilson-Mendenhall, C. D., Dahl, C. J., Davidson, R. J., & Rosenkranz, M. A. (2020) · doi:10.2196/preprints.23825 · Cited by 97
Self-citationKristjánsdóttir, Ó. B., Fors, E. A., Eide, E., Finset, A., Stensrud, T. L., van Dulmen, S., Wigers, S. H., & Eide, H. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2442 · Cited by 78 · cited 2×
“Maximum likelihood Multiple imputation MethodeDiffdWL drop WLc ITT Tx dropTxa ITTb Study NoYesANOVAgN/Af2578146234Bakker et al [42] NoYest testsYes, higher in active88195106192Bidargaddi et al [5] NoNoANOVAN/A41105128Bostock et al [43] N/AN/AANOVAN/A018020Carissoli et al [40] YesYesMLMhNo336938Champion et al [44] YesN…”
BackgroundMoëll, B., Kollberg, L., Nasri, B., Lindefors, N., & Kaldo, V. (2015) · Internet Interventions · doi:10.1016/j.invent.2014.11.004 · Cited by 61 · cited 2×
“Maximum likelihood Multiple imputation MethodeDiffdWL drop WLc ITT Tx dropTxa ITTb Study NoYesANOVAgN/Af2578146234Bakker et al [42] NoYest testsYes, higher in active88195106192Bidargaddi et al [5] NoNoANOVAN/A41105128Bostock et al [43] N/AN/AANOVAN/A018020Carissoli et al [40] YesYesMLMhNo336938Champion et al [44] YesN…”
BackgroundLevin, M. E., Haeger, J., Pierce, B., & Cruz, R. A. (2017) · Behavior Modification · doi:10.1177/0145445517719661 · Cited by 58 · cited 2×
“Unclear No MLM N/A 0 11 0 12 Levin et al [55]”
BackgroundKrafft, J., Potts, S., Schoendorff, B., & Levin, M. E. (2019) · Behavior Modification · doi:10.1177/0145445517748561 · Cited by 45 · cited 2×
“Maximum likelihood Multiple imputation MethodeDiffdWL drop WLc ITT Tx dropTxa ITTb Study NoYesANOVAgN/Af2578146234Bakker et al [42] NoYest testsYes, higher in active88195106192Bidargaddi et al [5] NoNoANOVAN/A41105128Bostock et al [43] N/AN/AANOVAN/A018020Carissoli et al [40] YesYesMLMhNo336938Champion et al [44] YesN…”
BackgroundLevin, M. E., Haeger, J., An, W., & Twohig, M. P. (2018) · Cognitive Therapy and Research · doi:10.1007/s10608-018-9944-3 · Cited by 36 · cited 2×
“Maximum likelihood Multiple imputation MethodeDiffdWL drop WLc ITT Tx dropTxa ITTb Study NoYesANOVAgN/Af2578146234Bakker et al [42] NoYest testsYes, higher in active88195106192Bidargaddi et al [5] NoNoANOVAN/A41105128Bostock et al [43] N/AN/AANOVAN/A018020Carissoli et al [40] YesYesMLMhNo336938Champion et al [44] YesN…”
Background
Cited by
Hirshberg, M. J., Frye, C., Dahl, C. J., Riordan, K. M., Vack, N. J., Sachs, J., Goldman, R., Davidson, R. J., & Goldberg, S. B. (2022) · Journal of Educational Psychology · doi:10.1037/edu0000739 · Cited by 78
Self-citationHirshberg, M. J., Colaianne, B. A., Greenberg, M. T., Inkelas, K. K., Davidson, R. J., Germano, D., Dunne, J. D., & Roeser, R. W. (2022) · Mindfulness · doi:10.1007/s12671-022-01952-1 · Cited by 41
Self-citationOverall and differential attrition in mindfulness-based interventions: A meta-analysis
Lam, S. U., Kirvin-Quamme, A., & Goldberg, S. B. (2022) · Mindfulness · doi:10.1007/s12671-022-01970-z · Cited by 39
Self-citation
Metadata
- Authors
- Goldberg, S. B., Bolt, D. M., & Davidson, R. J.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2021-06-15
- Volume
- 23
- Issue
- 6
- Pages
- e26749
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W3167431989
- Cited by
- 74
- References
- 50
- FWCI
- 8.78
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.