Methodological challenges in randomized controlled trials on smartphone-based treatment in psychiatry: systematic review
Open accessAbstract
BACKGROUND: Smartphone-based technology is developing at high speed, and many apps offer potential new ways of monitoring and treating a range of psychiatric disorders and symptoms. However, the effects of most available apps have not been scientifically investigated. Within medicine, randomized controlled trials (RCTs) are the standard method for providing the evidence of effects. However, their rigidity and long time frame may contrast with the field of information technology research. Therefore, a systematic review of methodological challenges in designing and conducting RCTs within mobile health is needed. OBJECTIVE: This systematic review aimed to (1) identify and describe RCTs investigating the effect of smartphone-based treatment in adult patients with a psychiatric diagnosis, (2) discuss methodological challenges in designing and conducting individual trials, and (3) suggest recommendations for future trials. METHODS: A systematic search in English was conducted in PubMed, PsycINFO, and EMBASE up to August 12, 2019. The search terms were (1) psychiatric disorders in broad term and for specific disorders AND (2) smartphone or app AND (3) RCT. The Consolidated Standards of Reporting Trials electronic health guidelines were used as a template for data extraction. The focus was on trial design, method, and reporting. Only trials having sufficient information on diagnosis and acceptable diagnostic procedures, having a smartphone as a central part of treatment, and using an RCT design were included. RESULTS: A total of 27 trials comprising 3312 patients within a range of psychiatric diagnoses were included. Among them, 2 trials were concerning drug or alcohol abuse, 3 psychosis, 10 affective disorders, 9 anxiety and posttraumatic stress disorder, 1 eating disorder, and 1 attention-deficit/hyperactivity disorder. In addition, 1 trial used a cross-diagnostic design, 7 trials included patients with a clinical diagnosis that was subsequently assessed and validated by the researchers, and 11 trials had a sample size above 100. Generally, large between-trial heterogeneity and multiple approaches to patient recruitment, diagnostic procedures, trial design, comparator, outcome measures, and analyses were identified. Only 5 trials published a trial protocol. Furthermore, 1 trial provided information regarding technological updates, and only 18 trials reported on the conflicts of interest. No trial addressed the ethical aspects of using smartphones in treatment. CONCLUSIONS: This first systematic review of the methodological challenges in designing and conducting RCTs investigating smartphone-based treatment in psychiatric patients suggests an increasing number of trials but with a lower quality compared with classic medical RCTs. Heterogeneity and methodological issues in individual trials limit the evidence. Methodological recommendations are presented.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The current trial will evaluate intervention effects using only self-reported outcome measures, which may create a risk of bias due to social desirability and shared method variance and reduce the generalizability of the results to clinical practice.
“However, it should be noted that the current trial will evaluate the intervention effects using only self-reported outcome measures, which may create a risk of bias due to social desirability and shared method variance and reduce the generalizability of the results to clinical practice (Tønning et al., 2019).”
From Efficacy and mechanisms of mobile application-delivered Acceptance and Commitment Therapy for posttraumatic stress disorder in China: Study protocol for a randomized controlled trial (2022) · cites this paper as Tønning et al., 2019
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823 · see citing works
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859 · see citing works
James, S. L., Abate, D., Abate, K. H., Abay, S. M., Abbafati, C., Abbasi, N., Abbastabar, H., Abd-Allah, F., Abdela, J., Abdelalim, A., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32279-7 · Cited by 13,867 · see citing works
Vos, T., Abajobir, A. A., Abate, K. H., Abbafati, C., Abbas, K. M., Abd-Allah, F., Abdulkader, R. S., Abdulle, A. M., Abebo, T. A., Abera, S. F., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(17)32154-2 · Cited by 13,220 · see citing works
SCAN: Schedules fonr Clinical Assessment in Neuropsychiatry
Wing, J. K. (1990) · Archives of General Psychiatry · doi:10.1001/archpsyc.1990.01810180089012 · Cited by 2,517 · see citing works
Eysenbach, G., & CONSORT-EHEALTH Group (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1923 · Cited by 1,863 · cited 7× · see citing works
“Some trials [24,27,29-33,35,40] were leaning toward traditional RCT designs conducted in clinical settings and using the CONSORT checklist for reporting and designing trials [20].”
MethodInfluentialSize and burden of mental disorders in Europe—a critical review and appraisal of 27 studies
Wittchen, H.-U., & Jacobi, F. (2005) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2005.04.012 · Cited by 1,503 · cited 1× · see citing works
“Across European countries, 27% of the adult population suffers from at least one psychiatric disorder [3].”
BackgroundSmartphones for smarter delivery of mental health programs: A systematic review
Donker, T., Petrie, K., Proudfoot, J., Clarke, J., Birch, M.-R., & Christensen, H. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2791 · Cited by 1,136 · cited 2× · see citing works
“A previous review from 2013 [18] identified effect studies of smartphone-based treatments within psychiatry (including stress).”
BackgroundFirth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097 · cited 1× · see citing works
“As an example, Firth et al [19] found studies regarding depressive symptoms and conducted a meta-analysis on the possible effects of smartphone-based treatments on these symptoms.”
ResultMachine Learning for Precision Psychiatry: Opportunities and Challenges
Bzdok, D., & Meyer‐Lindenberg, A. (2017) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2017.11.007 · Cited by 833 · cited 1× · see citing works
“Such learning systems will change based on new information and might be used increasingly in psychiatry as well as in many other fields [62].”
BackgroundTrautmann, S., Rehm, J., & Wittchen, H. (2016) · The EMBO Reports · doi:10.15252/embr.201642951 · Cited by 798 · see citing works
Generation of allocation sequences in randomised trials: chance, not choice
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07683-3 · Cited by 774 · see citing works
A Smartphone Application to Support Recovery From Alcoholism: A Randomized Clinical Trial
Gustafson, D. H., McTavish, F. M., Chih, M.-Y., Atwood, A. K., Johnson, R. A., Boyle, M. G., Levy, M. S., Driscoll, H., Chisholm, S. M., Dillenburg, L., et al. (2014) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2013.4642 · Cited by 692 · cited 8× · see citing works
“10 | e15362 | p. 6http://www.jmir.org/2019/10/e15362/ (page number not for citation purposes) XSL•FO RenderX Posttreatment follow-up (weeks) Intervention length (weeks) Age (years), mean (SD) Analyzed (power calculation) Female/totalProtocolaTrial design CountryAuthor, year of publication No84220 (N/A)1/20N/APilot RCT…”
BackgroundInfluentialMental health: there’s an app for that
Anthes, E. (2016) · Nature · doi:10.1038/532020a · Cited by 333 · cited 1× · see citing works
“Nevertheless, hundreds of apps claiming to help or monitor psychiatric disorders are already available in app stores [11].”
BackgroundCBT for depression: a pilot RCT comparing mobile phone vs. computer
Watts, S., Mackenzie, A., Thomas, C., Griskaitis, A., Mewton, L., Williams, A., & Andrews, G. (2013) · BMC Psychiatry · doi:10.1186/1471-244x-13-49 · Cited by 329 · cited 14× · see citing works
“…BDIIIe, K-10f, and other) Open (3 months); advertising + application on a Web page MINIc phone inter- view + PHQ-9d Major depressive disorder (Rb) Watts et al, 2013 [22] Internet platform Questionnaire (LSAS-SRi)aOpen (2011 and 2012); advertising in media and Facebook.”
BackgroundMethodInfluentialWang, K., Varma, D. S., & Prosperi, M. (2018) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2018.10.006 · Cited by 308 · cited 1× · see citing works
“Thus, most available apps have not been scientifically investigated, and the validity, treatment effect, and safety have been sparingly investigated [9,10].”
BackgroundLy, K. H., Trüschel, A., Jarl, L., Magnusson, S., Windahl, T., Johansson, R., Carlbring, P., & Andersson, G. (2014) · BMJ Open · doi:10.1136/bmjopen-2013-003440 · Cited by 269 · cited 15× · see citing works
“Within these 12 trials, 5 trials used the patient-reported outcome as the primary outcome measure in nonblinded patients [25,28,29,32,33].”
BackgroundMethodInfluentialRapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise
Riley, W. T., Glasgow, R. E., Etheredge, L., & Abernethy, A. P. (2013) · Clinical and Translational Medicine · doi:10.1186/2001-1326-2-10 · Cited by 267 · cited 2× · see citing works
“Other groups have previously suggested changes to the development process, to speed up the process from idea to publication [14,16,17].”
BackgroundRoepke, A. M., Jaffee, S. R., Riffle, O. M., McGonigal, J., Broome, R., & Maxwell, B. (2015) · Games for Health Journal · doi:10.1089/g4h.2014.0046 · Cited by 265 · cited 20× · see citing works
“Overall, 11 trials had a sample size above 100 patients [24,30,32-34,39-41,44,45,48].”
BackgroundMethodInfluentialA randomized controlled trial of a smartphone app for posttraumatic stress disorder symptoms.
Kuhn, E., Kanuri, N., Hoffman, J. E., Garvert, D. W., Ruzek, J. I., & Taylor, C. B. (2017) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000163 · Cited by 225 · cited 8× · see citing works
“10 | e15362 | p. 6http://www.jmir.org/2019/10/e15362/ (page number not for citation purposes) XSL•FO RenderX Posttreatment follow-up (weeks) Intervention length (weeks) Age (years), mean (SD) Analyzed (power calculation) Female/totalProtocolaTrial design CountryAuthor, year of publication No84220 (N/A)1/20N/APilot RCT…”
BackgroundMethodInfluentialBaker, T. B., Gustafson, D. H., & Shah, D. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2925 · Cited by 221 · cited 2× · see citing works
“By allowing a more flexible approach to the RCT design, it would conform more to the so-called adaptive trial paradigm [17,63], and then RCTs might be able to answer questions regarding the effect of mHealth interventions without evaluating outdated technology [16,17].”
BackgroundMethodMohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., Kwasny, M. J., Stiles-Shields, C., & Cheung, K. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4391 · Cited by 216 · cited 1× · see citing works
“Furthermore, the locked nature of treatment interventions tested in RCTs contrasts with the constantly evolving and iterative nature of app solutions [15].”
BackgroundBelisario, J. S. M., Jamšek, J., Huckvale, K., O’Donoghue, J., Morrison, C. P., & Car, J. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000042.pub2 · Cited by 216 · cited 1× · see citing works
“However, a Cochrane review from 2015 concludes that “apps might not affect data equivalence as long as the intended clinical application of the survey questionnaire, its intended frequency of administration and the setting in that it was validated remain unchanged” [50].”
BackgroundDepp, C. A., Ceglowski, J., Wang, V. C., Yaghouti, F., Mausbach, B. T., Thompson, W. K., & Granholm, E. L. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.10.053 · Cited by 215 · cited 21× · see citing works
“Numbers in parentheses represent relevant chapter and coding according to International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10): 2 on drug or alcohol misuse (F10-F19) [24,37], 3 on psychosis (F20-29) [35,38,41], 10 on affective disorders (F30-39) [22,25-28,32,36,40,42,45] (compri…”
BackgroundMethodInfluentialFaurholt-Jepsen, M., Frost, M., Ritz, C., Christensen, E. M., Jacoby, A. S., Mikkelsen, R. L., Knorr, U., Bardram, J. E., Vinberg, M., & Kessing, L. V. (2015) · Psychological Medicine · doi:10.1017/s0033291715000410 · Cited by 213 · cited 13× · see citing works
“…Depression 17 item and YMRS)a Closed (18 months); recruitment from specialized hospital function From outpatient clinic + SCANn Bipolar disorder (R+C) Faurholt-Jepsen et al, 2015 [27] Internet platform Questionnaires (BDI-II)aOpen (N/A); advertising in media (for patients’ self-identifying as…”
BackgroundMethodInfluentialSelf-citationSchlosser, D. A., Campellone, T. R., Truong, B., Etter, K., Vergani, S., Komaiko, K., & Vinogradov, S. (2018) · Schizophrenia Bulletin · doi:10.1093/schbul/sby078 · Cited by 206 · cited 18× · see citing works
“Numbers in parentheses represent relevant chapter and coding according to International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10): 2 on drug or alcohol misuse (F10-F19) [24,37], 3 on psychosis (F20-29) [35,38,41], 10 on affective disorders (F30-39) [22,25-28,32,36,40,42,45] (compri…”
BackgroundMethodInfluentialThe digital placebo effect: mobile mental health meets clinical psychiatry
Torous, J., & Firth, J. (2016) · The Lancet Psychiatry · doi:10.1016/s2215-0366(15)00565-9 · Cited by 200 · cited 2× · see citing works
“The term digital placebo has been suggested [52].”
BackgroundActissist: Proof-of-Concept Trial of a Theory-Driven Digital Intervention for Psychosis
Bucci, S., Barrowclough, C., Ainsworth, J., Machin, M., Morris, R., Berry, K., Emsley, R., Lewis, S., Edge, D., Buchan, I., et al. (2018) · Schizophrenia Bulletin · doi:10.1093/schbul/sby032 · Cited by 196 · cited 13× · see citing works
“A total of 5 trials based their diagnosis solely on clinical-based information [24,33,35,37,41].”
BackgroundMethodInfluentialMantani, A., Kato, T., Furukawa, T. A., Horikoshi, M., Imai, H., Hiroe, T., Chino, B., Funayama, T., Yonemoto, N., Zhou, Q., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8602 · Cited by 192 · cited 13× · see citing works
“17 8 25-59d 164 (164) 87/164 Publishedc RCT Japan Mantani et al, 2017 [32]”
InfluentialEnock, P. M., Hofmann, S. G., & McNally, R. J. (2014) · Cognitive Therapy and Research · doi:10.1007/s10608-014-9606-z · Cited by 187 · cited 17× · see citing works
“Overall, 11 trials had a sample size above 100 patients [24,30,32-34,39-41,44,45,48].”
BackgroundInfluentialMethodologies for assessing telemedicine: A systematic review of reviews
Ekeland, A. G., Bowes, A., & Flottorp, S. (2012) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2011.10.009 · Cited by 187 · cited 1× · see citing works
“It is necessary to accept and incorporate technological developments and changes in the design of a specific RCT in line with suggestions from similar but older scientific fields as telehealth in general [64].”
BackgroundSmartphone data as objective measures of bipolar disorder symptoms
Faurholt-Jepsen, M., Frost, M., Vinberg, M., Christensen, E. M., Bardram, J. E., & Kessing, L. V. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.03.009 · Cited by 185 · cited 1× · see citing works
“Even when patients are “lost to follow-up,” assessments based on smartphone sensors could be collected, with patient permission, and might reflect behavioral changes for the user [59-61].”
BackgroundMiner, A., Kuhn, E., Hoffman, J. E., Owen, J. E., Ruzek, J. I., & Taylor, C. B. (2016) · Psychological Trauma: Theory, Research, Practice, and Policy · doi:10.1037/tra0000092 · Cited by 185 · cited 15× · see citing works
“…platform Questionnaire (PCL-C) + feasibility and acceptability Open (16 months); flyers, websites, Craigslist Questionnaire PCLCt >30 PTSDs (R)Miner et al, 2016 [46] N/AFeasibility metrics + Questionnaires (PCL-Su, PHQ-9) Closed (4 months); from veteran care unit, screened for PTSD. Screened…”
BackgroundMethodInfluentialDogan, E., Sander, C., Wagner, X., Hegerl, U., & Kohls, E. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7006 · Cited by 177 · cited 1× · see citing works
“At the same time, smartphones contain several sensors and technologies enabling patients, researchers, and clinicians to access information about physical and social activities [8].”
BackgroundBen-Zeev, D., Brian, R. M., Jonathan, G., Razzano, L., Pashka, N., Carpenter-Song, E., Drake, R. E., & Scherer, E. A. (2018) · Psychiatric Services · doi:10.1176/appi.ps.201800063 · Cited by 173 · cited 20× · see citing works
“A total of 5 trials based their diagnosis solely on clinical-based information [24,33,35,37,41].”
BackgroundMethodInfluentialLy, K. H., Topooco, N., Cederlund, H., Wallin, A., Bergström, J., Molander, O., Carlbring, P., & Andersson, G. (2015) · PLOS ONE · doi:10.1371/journal.pone.0126559 · Cited by 171 · cited 2× · see citing works
“…N/ANoBTFull behavioral activation (10 sessions) Four therapy sessions and a smartphone app, based on behavioral activation, used between sessions Ly et al, 2015 [28] Multiple alreadyavailable apps N/ANoBTWaitlist controlLiving smart: Guided online course to structure life using multiple…”
BackgroundMethodFaurholt‐Jepsen, M., Munkholm, K., Frost, M. M., Bardram, J. E., & Kessing, L. V. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0713-0 · Cited by 169 · see citing works
Possemato, K., Kuhn, E., Johnson, E., Hoffman, J. E., Owen, J. E., Kanuri, N., De Stefano, L., & Brooks, E. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2015.09.005 · Cited by 167 · cited 11× · see citing works
“Numbers in parentheses represent relevant chapter and coding according to International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10): 2 on drug or alcohol misuse (F10-F19) [24,37], 3 on psychosis (F20-29) [35,38,41], 10 on affective disorders (F30-39) [22,25-28,32,36,40,42,45] (compri…”
BackgroundMethodInfluentialIvanova, E., Lindner, P., Ly, K. H., Dahlin, M., Vernmark, K., Andersson, G., & Carlbring, P. (2016) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2016.09.012 · Cited by 165 · cited 13× · see citing works
“3 152 (150) 98/152 Publishedc RCTf Sweden Ivanova et al, 2016 [30]”
BackgroundMethodInfluentialKerst, A., Zielasek, J., & Gäebel, W. (2019) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-018-0974-3 · Cited by 158 · cited 1× · see citing works
“Clinicians and administrators may play important roles in implementing such treatment systems following trial findings if the results indicate a possible effect of the new treatment [55].”
BackgroundEthical perspectives on recommending digital technology for patients with mental illness
Bauer, M., Glenn, T., Monteith, S., Bauer, R., Whybrow, P. C., & Geddes, J. R. (2017) · International Journal of Bipolar Disorders · doi:10.1186/s40345-017-0073-9 · Cited by 132 · cited 2× · see citing works
“The industry has a significant direct or indirect interest in developing app solutions and gathers information from patients as information in the digital economy involves major economic interests [58].”
BackgroundFaurholt‐Jepsen, M., Frost, M. M., Christensen, E. M., Bardram, J. E., Vinberg, M., & Kessing, L. V. (2019) · Psychological Medicine · doi:10.1017/s0033291719000710 · Cited by 129 · cited 11× · see citing works
“Overall, 11 trials had a sample size above 100 patients [24,30,32-34,39-41,44,45,48].”
BackgroundMethodInfluentialDagöö, J., Asplund, R. P., Bsenko, H. A., Hjerling, S., Holmberg, A., Westh, S., Öberg, L., Ljótsson, B., Carlbring, P., Furmark, T., et al. (2014) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2014.02.003 · Cited by 121 · cited 13× · see citing works
“Numbers in parentheses represent relevant chapter and coding according to International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10): 2 on drug or alcohol misuse (F10-F19) [24,37], 3 on psychosis (F20-29) [35,38,41], 10 on affective disorders (F30-39) [22,25-28,32,36,40,42,45] (compri…”
BackgroundMethodInfluentialStolz, T., Schulz, A., Krieger, T., Vincent, A., Urech, A., Moser, C., Westermann, S., & Berger, T. (2018) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000301 · Cited by 102 · cited 2× · see citing works
“…Schizophrenia spectrum disorders (R) Schlosser et al, 2018 [38] Internet platform Questionnaires (Social Pho- bia Scale, SIASq, LSASSR)a Open (N/A); media and online forums DSM-IV (master students) Social anxiety disorder (R) Stolz et al, 2018 [39] J Med Internet Res 2019 | vol. 21 | iss.”
BackgroundMethodSmartphone-based objective monitoring in bipolar disorder: status and considerations
Faurholt‐Jepsen, M., Bauer, M., & Kessing, L. V. (2018) · International Journal of Bipolar Disorders · doi:10.1186/s40345-017-0110-8 · Cited by 100 · cited 1× · see citing works
“Even when patients are “lost to follow-up,” assessments based on smartphone sensors could be collected, with patient permission, and might reflect behavioral changes for the user [59-61].”
BackgroundCritical concepts in adaptive clinical trials
Park, J., Thorlund, K., & Mills, E. J. (2018) · Clinical Epidemiology · doi:10.2147/clep.s156708 · Cited by 100 · cited 1× · see citing works
“By allowing a more flexible approach to the RCT design, it would conform more to the so-called adaptive trial paradigm [17,63], and then RCTs might be able to answer questions regarding the effect of mHealth interventions without evaluating outdated technology [16,17].”
MethodFaurholt‐Jepsen, M., Vinberg, M., Christensen, E. M., Frost, M. M., Bardram, J. E., & Kessing, L. V. (2013) · BMJ Open · doi:10.1136/bmjopen-2013-003353 · Cited by 91 · cited 1× · see citing works
“Furthermore, a thorough description of the used technology should be included [54,56,57].”
BackgroundFaurholt‐Jepsen, M., Vinberg, M., Frost, M. M., Christensen, E. M., Bardram, J. E., & Kessing, L. V. (2014) · BMC Psychiatry · doi:10.1186/s12888-014-0309-5 · Cited by 87 · cited 1× · see citing works
“Furthermore, a thorough description of the used technology should be included [54,56,57].”
BackgroundHur, J.-W., Kim, B., Park, D., & Choi, S.-W. (2018) · Telemedicine and e-Health · doi:10.1089/tmj.2017.0214 · Cited by 82 · cited 17× · see citing works
“No 3 ≈24 34 (N/A) 26/48 N/A Pilot RCT Republic of Korea Hur et al, 2018 [36]”
BackgroundMethodInfluentialHildebrandt, T., Michaelides, A. P., Mackinnon, D., Greif, R., DeBar, L., & Sysko, R. (2017) · International Journal of Eating Disorders · doi:10.1002/eat.22781 · Cited by 77 · cited 19× · see citing works
“Some trials [24,27,29-33,35,40] were leaning toward traditional RCT designs conducted in clinical settings and using the CONSORT checklist for reporting and designing trials [20].”
BackgroundMethodInfluentialMoëll, B., Kollberg, L., Nasri, B., Lindefors, N., & Kaldo, V. (2015) · Internet Interventions · doi:10.1016/j.invent.2014.11.004 · Cited by 61 · cited 13× · see citing works
“Multiple alreadyavailable apps N/A No BT Waitlist control Living smart: Guided online course to structure life using multiple already available apps Moëll et al, 2015 [29]”
BackgroundMethodInfluentialStiles-Shields, C., Montague, E., Kwasny, M. J., & Mohr, D. C. (2019) · Psychological Services · doi:10.1037/ser0000261 · Cited by 59 · cited 18× · see citing works
“Numbers in parentheses represent relevant chapter and coding according to International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10): 2 on drug or alcohol misuse (F10-F19) [24,37], 3 on psychosis (F20-29) [35,38,41], 10 on affective disorders (F30-39) [22,25-28,32,36,40,42,45] (compri…”
BackgroundMethodInfluentialKemmeren, L., van Schaik, A., Riper, H. M., Kleiboer, A. M., Bosmans, J. E., & Smit, J. H. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0818-5 · Cited by 49 · see citing works
Krzystanek, M., Borkowski, M., Skałacka, K., & Krysta, K. (2018) · Schizophrenia Research · doi:10.1016/j.schres.2018.08.016 · Cited by 49 · cited 16× · see citing works
“A total of 5 trials based their diagnosis solely on clinical-based information [24,33,35,37,41].”
BackgroundInfluentialA pilot study of a smartphone application supporting recovery from drug addiction
Liang, D., Han, H., Du, J., Zhao, M., & Hser, Y. (2018) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2018.02.006 · Cited by 47 · cited 13× · see citing works
“A total of 5 trials based their diagnosis solely on clinical-based information [24,33,35,37,41].”
BackgroundMethodInfluentialBoettcher, J., Magnusson, K., Marklund, A., Berglund, E., Blomdahl, R., Braun, U., Delin, L., Lundén, C., Sjöblom, K., Sommer, D., et al. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.04.052 · Cited by 47 · cited 14× · see citing works
“Overall, 11 trials had a sample size above 100 patients [24,30,32-34,39-41,44,45,48].”
BackgroundInfluentialTeng, M.-H., Hou, Y.-M., Chang, S.-H., & Cheng, H.-J. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.12.118 · Cited by 37 · cited 13× · see citing works
“Numbers in parentheses represent relevant chapter and coding according to International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10): 2 on drug or alcohol misuse (F10-F19) [24,37], 3 on psychosis (F20-29) [35,38,41], 10 on affective disorders (F30-39) [22,25-28,32,36,40,42,45] (compri…”
BackgroundMethodInfluentialScott, C. K., Dennis, M. L., & Gustafson, D. H. (2017) · Trials · doi:10.1186/s13063-017-2096-z · Cited by 33 · cited 1× · see citing works
“Notably, numerous trial protocols within the field were identified during the search [54,65-69].”
BackgroundMühlbauer, E., Bauer, M., Ebner‐Priemer, U., Ritter, P. S., Hill, H., Beier, F., Kleindienst, N., & Severus, E. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1929-y · Cited by 32 · cited 1× · see citing works
“Notably, numerous trial protocols within the field were identified during the search [54,65-69].”
BackgroundFaurholt‐Jepsen, M., Frost, M. M., Martiny, K., Tuxen, N., Rosenberg, N., Busk, J., Winther, O., Bardram, J. E., & Kessing, L. V. (2017) · Trials · doi:10.1186/s13063-017-2015-3 · Cited by 28 · cited 3× · see citing works
“Furthermore, a thorough description of the used technology should be included [54,56,57].”
BackgroundDesign of telehealth trials – Introducing adaptive approaches
Law, L. M., & Wason, J. M. S. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.09.002 · Cited by 26 · cited 2× · see citing works
“By allowing a more flexible approach to the RCT design, it would conform more to the so-called adaptive trial paradigm [17,63], and then RCTs might be able to answer questions regarding the effect of mHealth interventions without evaluating outdated technology [16,17].”
BackgroundMethodKolar, D. R., Hammerle, F., Jenetzky, E., & Huß, M. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-018049 · Cited by 26 · see citing works
Psychiatric diagnosis by telephone: is it an opportunity?
Muskens, E. M. H., Lucassen, P., Groenleer, W., van Weel, C., Oude Voshaar, R., & Speckens, A. (2014) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-014-0861-9 · Cited by 24 · cited 1× · see citing works
“Nevertheless, telephonic interviews (by trained lay interviewers or professionals) have been used in the research of depression and anxiety disorders with reasonable validity of diagnosis [51].”
BackgroundGiosan, C., Cobeanu, O., Mogoaşe, C., Szentagotai, A., Mureşan, V., & Boian, R. (2017) · Trials · doi:10.1186/s13063-017-1960-1 · Cited by 23 · cited 1× · see citing works
“Still, little is known about this issue; however, ongoing RCTs are investigating this subject further using a sham app as a comparator [53].”
BackgroundFaurholt‐Jepsen, M., Brage, S., Vinberg, M., Jensen, H. M., Christensen, E. M., Knorr, U. B., & Kessing, L. V. (2014) · Nordic Journal of Psychiatry · doi:10.3109/08039488.2014.936501 · Cited by 21 · cited 1× · see citing works
“Even when patients are “lost to follow-up,” assessments based on smartphone sensors could be collected, with patient permission, and might reflect behavioral changes for the user [59-61].”
BackgroundAnastasiadou, D., Lupiañez-Villanueva, F., Faulí, C., Arcal Cunillera, J., & Serrano-Troncoso, E. (2018) · BMC Psychiatry · doi:10.1186/s12888-018-1664-4 · Cited by 16 · cited 1× · see citing works
“Notably, numerous trial protocols within the field were identified during the search [54,65-69].”
BackgroundTime to publication for publicly funded clinical trials in Australia: an observational study
Strand, L. B., Clarke, P., Graves, N., & Barnett, A. G. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012212 · Cited by 15 · cited 1× · see citing works
“6 years) from funding to the main article on trial results being published [13].”
Background
Flagged first, then by citations
Cited by
Sorted by citations
Timmers, T., Janssen, L., Kool, R. B., & Kremer, J. A. M. (2020) · Journal of Medical Internet Research · doi:10.2196/17342 · Cited by 127 · see citing works
“This challenge was also reported by two recent studies focusing on eHealth interventions in general [36] and, more specifically, in the field of psychiatry [37].”
Systematic Review of Digital Phenotyping and Machine Learning in Psychosis Spectrum Illnesses
Benoit, J. R. A., Onyeaka, H., Keshavan, M. S., & Torous, J. B. (2020) · Harvard Review of Psychiatry · doi:10.1097/hrp.0000000000000268 · Cited by 114 · see citing works
Connolly, S. L., Hogan, T. P., Shimada, S. L., & Miller, C. J. (2020) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-020-00165-4 · Cited by 78 · see citing works
Psychiatry in the Digital Age: A Blessing or a Curse?
Roth, C. B., Papassotiropoulos, A., Brühl, A. B., Lang, U. E., & Huber, C. G. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18168302 · Cited by 71 · see citing works
“pointed to methodological challenges in randomized controlled trials on smartphone-based treatment in psychiatry, showing that the trial design and reporting were lower in quality compared with classic medical randomized controlled trials [78].”
BackgroundMelbye, S. A., Kessing, L. V., Bardram, J. E., & Faurholt‐Jepsen, M. (2020) · JMIR Mental Health · doi:10.2196/17453 · Cited by 57 · see citing works
“Such RCTs should provide details on the methodology, reporting, and interpretation of findings, as recently described by our group [31], thereby making it easier to compare studies and to facilitate future meta-analyses.”
BackgroundFaurholt‐Jepsen, M., Tønning, M. L., Fros, M., Martiny, K., Tuxen, N., Rosenberg, N., Busk, J., Winther, O., Thaysen‐Petersen, D., Andreasson, K., et al. (2020) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13274 · Cited by 46 · see citing works
Tønning, M. L., Faurholt‐Jepsen, M., Frost, M. M., Martiny, K., Tuxen, N., Rosenberg, N., Busk, J., Winther, O., Melbye, S. A., Thaysen‐Petersen, D., et al. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.12.141 · Cited by 42 · see citing works
“Trials, including PROMs as primary outcome measures, are common within smartphone-based treatments in psychiatry (Tønning et al., 2019).”
BackgroundInfluentialTeng, M.-H., Hou, Y.-M., Chang, S.-H., & Cheng, H.-J. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.12.118 · Cited by 37 · see citing works
Ortiz, A., Maslej, M. M., Husain, M. I., Daskalakis, Z. J., & Mulsant, B. H. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.08.140 · Cited by 41 · see citing works
Coaching to Support Mental Health Apps: Exploratory Narrative Review
Meyer, A., Wisniewski, H., & Torous, J. (2022) · JMIR Human Factors · doi:10.2196/28301 · Cited by 34 · see citing works
“We identified the trials using the 2019 review by Tønning et al [14] on the methodological challenges of randomized controlled trials on smartphone-based treatment in psychiatry and the 2021 scoping review by Tokgöz et al [15] on digital health interventions for depression.”
MethodThe impact of mental health literacy initiatives on youth elite athletes: A systematic review
Diamond, S., Wallace, L., English, M., & Caperchione, C. M. (2022) · Performance Enhancement & Health · doi:10.1016/j.peh.2022.100226 · Cited by 22 · see citing works
Stalujanis, E., Neufeld, J., Stalder, M. G., Belardi, A., Tegethoff, M., & Meinlschmidt, G. (2020) · JMIR mhealth and uhealth · doi:10.2196/20329 · Cited by 19 · see citing works
“Previous studies in the context of digital placebo effects introduced the concept [13], focused on methodological recommendations for RCTs of smartphone-based interventions [21,22], or used a sham version of an active app as control condition [23].”
Usability and feasibility of a cognitive-behavioral mobile app for ADHD in adults
Knouse, L. E., Hu, X., Sachs, G., & Isaacs, S. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000083 · Cited by 18 · see citing works
“Such studies present methodological challenges and require close collaboration between researchers from different disciplinary perspectives [38].”
BackgroundGómez-Pérez, M. C., García-Palacios, A., Castilla, D., Zaragozá, I., & Suso-Ribera, C. (2020) · Pain Research and Management · doi:10.1155/2020/7897268 · Cited by 18 · see citing works
“literature on clinical research in general and chronic pain in particular has been largely dominated by large-scale randomized clinical trials in which the analysis of changes, mostly with group interventions, is explored considering average group scores and a reduced number of assessments [39, 40].”
BackgroundOrtiz, A., & Mulsant, B. H. (2024) · Journal of Medical Internet Research · doi:10.2196/59826 · Cited by 16 · see citing works
“We are also mindful of a recent review [50] that emphasized methodological challenges in RCTs investigating smartphone-based treatment interventions for mental disorders, including lack of trial registrations, inappropriate comparators, lack of blinding, selection bias, and lack of generalizability.”
Paquette, C. E., Rubalcava, D. T., Chen, Y., Anand, D., & Daughters, S. B. (2021) · JMIR Formative Research · doi:10.2196/25749 · Cited by 13 · see citing works
“Indeed, in a recent systematic review examining smartphone-based treatments for psychiatric diagnoses, only 2 of 27 studies identified by the review examined SUD interventions [27].”
BackgroundThree Questions to Consider Before Applying Ecological Momentary Interventions (EMI) in Psychiatry
Gründahl, M., Deckert, J., & Hein, G. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00333 · Cited by 11 · see citing works
Zhao, Z., Zhao, C., Ren, Z., Shi, C., & Lai, L. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100585 · Cited by 10 · see citing works
Anýž, J., Bakštein, E., Dally, A., Kolenič, M., Hlinka, J., Hartmannová, T., Urbanová, K., Correll, C. U., Novák, D., & Španiel, F. (2021) · JMIR Mental Health · doi:10.2196/26348 · Cited by 10 · see citing works
López‐Alcalde, J., Wieland, L. S., Yan, Y., Barth, J., Khami, M. R., Shivalli, S., Lokker, C., Kaur, H., Macharia, P. M., Yun, S., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/53187 · Cited by 8 · see citing works
“These included the need for flexible trial designs that can quickly adapt to the fast-evolving nature of mHealth interventions, privacy and security concerns, the lack of standardized adherence metrics, and the difficulties in choosing primary outcomes and handling missing data [24].”
InfluentialFaurholt‐Jepsen, M., & Kessing, L. V. (2022) · Psychiatry Research · doi:10.1016/j.psychres.2022.114844 · Cited by 9 · see citing works
Aggestrup, A. S., Svendsen, S. D., Præstegaard, A., Løventoft, P., Nørregaard, L., Knorr, U., Dam, H., Frøkjær, E., Danilenko, K., Hageman, I., et al. (2023) · JMIR Mental Health · doi:10.2196/50072 · Cited by 8 · see citing works
“It is debatable whether electronic monitoring per se has a positive or negative impact on patients with affective disorders [18,19].”
Feasibility of mHealth interventions for depressive symptoms in Latin America: a systematic review
Escobar-Viera, C. G., Cernuzzi, L., Miller, R. S., Rodríguez-Marín, H. J., Vieta, E., Toñánez, M. G., Marsch, L. A., & Hidalgo‐Mazzei, D. (2021) · International Review of Psychiatry · doi:10.1080/09540261.2021.1887822 · Cited by 11 · see citing works
Apps for mental health care: The raise of digital psychiatry
Faurholt‐Jepsen, M., & Kessing, L. V. (2021) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2021.04.018 · Cited by 8 · see citing works
Lewczuk, K., Górowska, M. P., Li, Y., & Gola, M. K. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00807 · Cited by 7 · see citing works
“Despite these factors, to the best of our knowledge, and recent-meta-analytic work (55), no randomized controlled trials for interventions delivered via smartphones are available for online problematic behavior.”
BackgroundBłachnio, A., Przepiórka, A., Gorbaniuk, O., McNeill, M., Bendayan, R., Durak, M., Şenol-Durak, E., Ben‐Ezra, M., Benvenuti, M., Angeluci, A. C. B., et al. (2022) · Frontiers research topics · doi:10.3389/978-2-88974-249-3 · Cited by 4 · see citing works
Linghu, T., Liu, C., Wang, Q., Tian, J., & Qin, X. (2021) · Journal of Liquid Chromatography & Related Technologies · doi:10.1080/10826076.2021.1975294 · Cited by 2 · see citing works
Text-based psychotherapy: analyzing opportunities and challenges beyond the session
Flaherty, H. B., & Beckerman, N. L. (2025) · Social Work in Mental Health · doi:10.1080/15332985.2025.2468277 · Cited by 2 · see citing works
“Tønning and Bardram (2019) reviewed 38 articles based on 27 different trials of smartphone-based therapy with patients with a range of psychiatric diagnoses.”
Eylem, Ö., Poulter, S., Ashcroft, K., Robinson, T., Mane, P., Islam, M., Condell, J. V., & Leavey, G. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-083554 · Cited by 2 · see citing works
Eylem-van Bergeijk, O., Robinson, T., Manktelow, M., Olympios, M., Poulter, S., Mane, P., Panagioti, M., Condell, J., & Leavey, G. (2025) · JMIR mHealth and uHealth · doi:10.2196/70691 · Cited by 2 · see citing works
“It is well established that participation in waitlist groups often heightens expectations about the intervention, which may in turn motivate participants to remain in the study to access the intervention [82].”
Bolea-Alamanac, B. M., Hidalgo‐Mazzei, D., Geddes, J. R., Kas, M. J., Kuntsi, J., Matura, S., Ruf, A., Vieta, E., Auriacombe, M., Bauer, M., et al. (2026) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2026.112883 · Cited by 1 · see citing works
Electronic monitoring and mental health
Faurholt‐Jepsen, M. (2020) · Nordic Journal of Psychiatry · doi:10.1080/08039488.2020.1852311
“Despite the hype, the validity, treatment effects (both positive and negative), and safety has been sparingly investigated in scientific studies [4].”
BackgroundRong, P., Wang, L., Yu, L., Wang, J., Ma, Y., Wang, L., Li, S., Tang, C., Lu, L., Lu, L., et al. (2020) · doi:10.2196/preprints.25001
Zhao, Q. S., Chen, P., Zhang, Y., Liu, H., & Li, X. (2020) · doi:10.2196/preprints.22474
The Scoping Review of Smart Healthcare Applications in Medical Health (Preprint)Preprint
Wu, C. H., Chiu, C., Liou, Y. J., Lee, C. Y., & Hu, S. C. (2020) · doi:10.2196/preprints.18678
The Scoping Review of Smart Healthcare Applications in Medical Health (Preprint)Preprint
Wu, C. H., Chiu, C., Liou, Y. J., Lee, C. Y., & Hu, S. C. (2020) · doi:10.2196/preprints.18681
Smartphone-Based Treatment in Psychiatry: A Systematic Review
Faurholt‐Jepsen, M., Tønning, M. L., & Kessing, L. V. (2023) · Springer eBooks · doi:10.1007/978-3-031-10698-9_5
Leijse, M. M. L., van Dam, L., Bouwmeester, S., van der Pol, T. M., Breuk, R., & Popma, A. (2025) · medRxiv · doi:10.1101/2025.09.08.25334620
“Smartphones, in particular, offer accessible functionalities that enable users to easily access therapeutic content and engage in skill practice in real time and naturalistic settings [29,30].”
Leijse, M. M. L., van Dam, L., Bouwmeester, S., van der Pol, T. M., Breuk, R., & Popma, A. (2026) · PLoS ONE · doi:10.1371/journal.pone.0332111
Keys, R., O’Kane, A. A., Marshall, P., & Stuart, G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790692
“We argue it is a particularly useful design tool in areas where technology outpaces evidence-based medicine, which has been a growing issue for HCI researchers [121, 127], and of particular concern for the rapid innovation in AI [27].”
- Authors
- Tønning, M. L., Kessing, L. V., Bardram, J. E., & Faurholt-Jepsen, M.
- Affiliations
- Copenhagen University Hospital (DK)
- Technical University of Denmark (DK)
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-10-27
- Volume
- 21
- Issue
- 10
- Pages
- e15362
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6914239
- OpenAlex
- W2982656086
- Cited by
- 40
- References
- 67
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 3 · 2023: 2 · 2022: 6 · 2021: 7 · 2020: 15 · 2018: 1
- FWCI
- 0.98
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, other-oa)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
- Keywords
- mobile health, mental illness, randomized controlled trials, trial design, methodological quality, trial reporting quality, conflicts of interest, mental health apps, outcome measures, patient recruitment, diagnostic procedures, trial size, trial duration, clinical trial registration
- MeSH terms
- Smartphone, Adult, Data Collection, Humans, Mental Health, Psychiatry, Randomized Controlled Trials as Topic, Outcome Assessment, Health Care, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
