GLOBEM: Cross-Dataset Generalization of Longitudinal Human Behavior Modeling
Open accessThe DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Abstract
There is a growing body of research revealing that longitudinal passive sensing data from smartphones and wearable devices can capture daily behavior signals for human behavior modeling, such as depression detection. Most prior studies build and evaluate machine learning models using data collected from a single population. However, to ensure that a behavior model can work for a larger group of users, its generalizability needs to be verified on multiple datasets from different populations. We present the first work evaluating cross-dataset generalizability of longitudinal behavior models, using depression detection as an application. We collect multiple longitudinal passive mobile sensing datasets with over 500 users from two institutes over a two-year span, leading to four institute-year datasets. Using the datasets, we closely re-implement and evaluated nine prior depression detection algorithms. Our experiment reveals the lack of model generalizability of these methods. We also implement eight recently popular domain generalization algorithms from the machine learning community. Our results indicate that these methods also do not generalize well on our datasets, with barely any advantage over the naive baseline of guessing the majority. We then present two new algorithms with better generalizability. Our new algorithm, Reorder, significantly and consistently outperforms existing methods on most cross-dataset generalization setups. However, the overall advantage is incremental and still has great room for improvement. Our analysis reveals that the individual differences (both within and between populations) may play the most important role in the cross-dataset generalization challenge. Finally, we provide an open-source benchmark platform GLOBEM- short for Generalization of Longitudinal BEhavior Modeling - to consolidate all 19 algorithms. GLOBEM can support researchers in using, developing, and evaluating different longitudinal behavior modeling methods. We call for researchers' attention to model generalizability evaluation for future longitudinal human behavior modeling studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large-scale data collection efforts by the wearables community, e.g., the GLOBEM dataset [206] and the UK Biobank project [46,198], demonstrated the relative ease of collecting wearable movement in the wild, albeit without annotations.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [206]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Diagnostic and Statistical Manual of Mental Disorders
American Psychiatric Association (2013) · doi:10.1176/appi.books.9780890425596 · Cited by 116,369
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“The second method, Reorder, creates a new task of solving a temporal reordering puzzle in addition to the main depression detection task, which was inspired by selfsupervised techniques in CV and NLP approaches [24, 77].”
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · cited 1×
“We used SMOTE to mitigate the data imbalance as datasets have different sizes [17].”
MethodFocal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · cited 1×
“Other loss functions that better deals with class imbalance, such as balanced cross entropy and focal loss [51], can be tested and evaluated on our platform.”
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 2×
“If so, it provides the opportunity to leverage a wide range of transfer learning techniques in the domain adaptation area [74, 95].”
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Beck, A. T., Steer, R. A., & Brown, G. (2011) · PsycTESTS Dataset · doi:10.1037/t00742-000 · Cited by 7,535
Vos, T., Allen, C., Arora, M., Barber, R. M., Bhutta, Z. A., Brown, A., Carter, A., Casey, D. C., Charlson, F. J., Chen, A. Z., et al. (2016) · The Lancet · doi:10.1016/s0140-6736(16)31678-6 · Cited by 7,253
An overview of statistical learning theory
Vapnik, V. N. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.788640 · Cited by 6,266 · cited 2×
“(1) ERM (Empirical Risk Minimization) [89]: the basic model training techniques without particular design for domain generalization.”
BackgroundSGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309 · cited 1×
“For all models, we used Adam as the optimizer and adopted a cosine annealing schedule to repeatedly decrease the learning rate and then restart with a higher learning rate [53], with an initial learning rate of 0.001, an annealing decay of 0.95, and an annealing step size of 100.”
Influential
Research works printed but not linked: 38
- Daniel A Adler, Fei Wang, David C Mohr, and Tanzeem Choudhury. 2022. Machine learning for passive mental health symptom prediction: Generalization across different longitudinal mobile sensing studies. PLOS ONE (2022), 20.
- Dror Ben-Zeev, Rachel Brian, Rui Wang, Weichen Wang, Andrew T Campbell, Min SH Aung, Michael Merrill, Vincent WS Tseng, Tanzeem Choudhury, Marta Hauser, et al. 2017. CrossCheck: Integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatric rehabilitation journal 40, 3 (2017), 266.
- Terry C Camacho, Robert E Roberts, Nancy B Lazarus, George A Kaplan, and Richard D Cohen. 1991. Physical activity and depression: evidence from the Alameda County Study. American journal of epidemiology 134, 2 (1991), 220--231.
- Fabio Maria Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. 2019. Domain Generalization by Solving Jigsaw Puzzles. arXiv:1903.06864 [cs] (April 2019). http://arxiv.org/abs/1903.06864 arXiv: 1903.06864. · arXiv
- 10.5555/1622407.1622416 · doi:10.5555/1622407.1622416
- John R Crawford and Julie D Henry. 2004. The Positive and Negative Affect Schedule (PANAS): Construct validity, measurement properties and normative data in a large non-clinical sample. British journal of clinical psychology 43, 3 (2004), 245--265.
- Hal Daumé III. 2007. Frustratingly Easy Domain Adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics, Prague, Czech Republic, 256--263. https://www.aclweb.org/anthology/P07-1033
- Afsaneh Doryab, Prerna Chikarsel, Xinwen Liu, and Anind K Dey. 2018. Extraction of behavioral features from smartphone and wearable data. arXiv preprint arXiv:1812.10394 (2018). · arXiv
- Qi Dou, Daniel C. Castro, Konstantinos Kamnitsas, and Ben Glocker. 2019. Domain Generalization via Model-Agnostic Learning of Semantic Features. arXiv:1910.13580 [cs] (Oct. 2019). http://arxiv.org/abs/1910.13580 arXiv: 1910.13580. · arXiv
- Eiko I Fried, Jessica K Flake, and Donald J Robinaugh. 2022. Revisiting the theoretical and methodological foundations of depression measurement. Nature Reviews Psychology (2022), 1--11.
- 10.1007/978-3-319-58347-1_10 · DOI · Paper page
- Taesik Gong, Yewon Kim, Adiba Orzikulova, Yunxin Liu, Sung Ju Hwang, Jinwoo Shin, and Sung-Ju Lee. 2021. DAPPER: Performance Estimation of Domain Adaptation in Mobile Sensing. arXiv preprint arXiv:2111.11053 (2021). · arXiv
- Ian H Gotlib and Jutta Joormann. 2010. Cognition and depression: current status and future directions. Annual review of clinical psychology 6 (2010), 285--312.
- Joy He-Yueya, Benjamin Buck, Andrew Campbell, Tanzeem Choudhury, John M Kane, Dror Ben-Zeev, and Tim Althoff. 2020. Assessing the relationship between routine and schizophrenia symptoms with passively sensed measures of behavioral stability. NPJ schizophrenia 6, 1 (2020), 1--8.
- Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. 2021. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning. PMLR, 5637--5664.
- Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. 2017. Learning to Generalize: Meta-Learning for Domain Generalization. arXiv:1710.03463 [cs] (Oct. 2017). http://arxiv.org/abs/1710.03463 arXiv: 1710.03463. · arXiv
- Donatella Marazziti, Giorgio Consoli, Michela Picchetti, Marina Carlini, and Luca Faravelli. 2010. Cognitive impairment in major depression. European journal of pharmacology 626, 1 (2010), 83--86.
- Matthew BA McDermott, Shirly Wang, Nikki Marinsek, Rajesh Ranganath, Luca Foschini, and Marzyeh Ghassemi. 2021. Reproducibility in machine learning for health research: Still a ways to go. Science Translational Medicine 13, 586 (2021), eabb1655.
- Tom M Mitchell and Tom M Mitchell. 1997. Machine learning. Vol. 1. McGraw-hill New York.
- Ebrahim Nemati, Xuhai Xu, Viswam Nathan, Korosh Vatanparvar, Tousif Ahmed, Md Mahbubur Rahman, Dan McCaffrey, Jilong Kuang, and Alex Gao. 2022. Ubilung: Multi-Modal Passive-Based Lung Health Assessment. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 551--555.
- Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. 2017. Exploring generalization in deep learning. Advances in neural information processing systems 30 (2017).
- Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. 2020. Learning explanations that are hard to vary. arXiv preprint arXiv:2009.00329 (2020). · arXiv
- Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. 2020. Efficient Domain Generalization via Common-Specific Low-Rank Decomposition. arXiv:2003.12815 [cs, stat] (April 2020). http://arxiv.org/abs/2003.12815 arXiv: 2003.12815. · arXiv
- Alan Ramponi and Barbara Plank. 2020. Neural Unsupervised Domain Adaptation in NLP---A Survey. arXiv:2006.00632 (2020). · arXiv
- Babak Roshanaei-Moghaddam, Wayne J Katon, and Joan Russo. 2009. The longitudinal effects of depression on physical activity. General hospital psychiatry 31, 4 (2009), 306--315.
- Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2019. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 (2019). · arXiv
- 10.1037/0096-3445.127.2.161 · doi:10.1037/0096-3445.127.2.161 · Paper page
- Michael E. Thase. 1998. Depression, sleep, and antidepressants. The Journal of Clinical Psychiatry (1998).
- Norifumi Tsuno, Alain Besset, and Karen Ritchie. 2005. Sleep and depression. The Journal of Clinical Psychiatry (2005).
- Julio Vega, Meng Li, Kwesi Aguillera, Nikunj Goel, Echhit Joshi, Kirtiraj Khandekar, Krina C Durica, Abhineeth R Kunta, and Carissa A Low. 2021. Reproducible Analysis Pipeline for Data Streams: Open-Source Software to Process Data Collected With Mobile Devices. Frontiers in Digital Health 3 (2021).
- Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T. Campbell. 2014. StudentLife: Assessing mental health, academic performance and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 3--14.
- Xuhai Xu, Han Zhang, Yasaman S Sefidgar, Yiyi Ren, Xin Liu, Woosuk Seo, Jennifer Brown, Kevin Scott Kuehn, Mike A Merrill, Paula S Nurius, Shwetak Patel, Tim Althoff, Margaret E Morris, Eve A. Riskin, Jennifer Mankoff, and Anind Dey. 2022. GLOBEM Dataset: Multi-Year Datasets for Longitudinal Human Behavior Modeling Generalization. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=GKOa7yNH8Uh
- Jenny Yang, Andrew AS Soltan, and David A Clifton. 2022. Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening. npj Digital Medicine 5, 1 (2022), 1--8.
- Yi Yao and Gianfranco Doretto. 2010. Boosting for transfer learning with multiple sources. In 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 1855--1862.
- Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond Empirical Risk Minimization. arXiv:1710.09412 [cs, stat] (April 2018). http://arxiv.org/abs/1710.09412 arXiv: 1710.09412. · arXiv
- Han Zhang, Margaret E. Morris, Paula S. Nurius, Kelly Mack, Jennifer Brown, Kevin S. Kuehn, Yasaman S. Sefidgar, Xuhai Xu, Eve A. Riskin, Anind K. Dey, and Jennifer Mankoff. 2022. Impact of Online Learning in the Context of COVID-19 on Undergraduates with Disabilities and Mental Health Concerns. ACM Transactions on Accessible Computing (July 2022), 3538514.
- Xu Zhang, Junghyun Kim, Qingwei Lin, Keunhak Lim, Shobhit O Kanaujia, Yong Xu, Kyle Jamieson, Aws Albarghouthi, Si Qin, Michael J Freedman, et al. 2019. Cross-dataset time series anomaly detection for cloud systems. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). 1063--1076.
- Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2021. Domain Generalization in Vision: A Survey. arXiv:2103.02503 [cs] (July 2021). http://arxiv.org/abs/2103.02503 arXiv: 2103.02503. · arXiv
Cited by
Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data
Xu, X., Yao, B., Dong, Y., Gabriel, S., Yu, H., Hendler, J., Ghassemi, M., Dey, A. K., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3643540 · Cited by 278
Self-citationYang, Z., Xu, X., Yao, B., Rogers, E., Zhang, S., Intille, S., Shara, N., Gao, G. G., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659625 · Cited by 120
Self-citationZhang, S., Yu, J., Xu, X., Yin, C., Lu, Y., Yao, B., Tory, M., Padilla, L. M., Caterino, J., Zhang, P., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642343 · Cited by 99
Self-citationDigital Phenotyping for Stress, Anxiety, and Mild Depression: Systematic Literature Review
Choi, A., Ooi, A., & Lottridge, D. M. (2023) · JMIR mhealth and uhealth · doi:10.2196/40689 · Cited by 108
Hong, Z., Li, Z., Zhong, S., Lyu, W., Wang, H., Ding, Y., He, T., & Zhang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659597 · Cited by 67
Metadata
- Authors
- Xu, X., Liu, X., Zhang, H., Wang, W., Nepal, S., Sefidgar, Y., Seo, W., Kuehn, K. S., Huckins, J. F., Morris, M. E., Nurius, P. S., Riskin, E. A., Patel, S., Althoff, T., Campbell, A., Dey, A. K., & Mankoff, J.
- Affiliations
- University of Washington (US)
- Dartmouth College (US)
- Stevens Institute of Technology (US)
- ORCID
- Venue
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Published
- 2022-12-21
- Volume
- 6
- Issue
- 4
- Pages
- 1-34
- ISSN
- 2474-9567 (ISSN-L)
- OpenAlex
- W4319837253
- Cited by
- 96
- References
- 77
- Citations by year
- 2026: 26 · 2025: 24 · 2024: 35 · 2023: 10 · 2022: 1
- FWCI
- 16.07
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion, cc-by)
- Funders
- University of Washington
- the National Science Foundation (EDA-2009977, CHS-2016365, CHS1941537, IIS1816687, IIS7974751)
- the National Institute on Disability, Independent Living and Rehabilitation Research (90DPGE0003-01)
- Samsung Research America
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- cross-dataset generalization, smartphone sensing, wearable devices, depression detection, human behavior modeling, domain generalization, individual differences, benchmark platform
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: GLOBEM: Cross-Dataset Generalization of Longitudinal Human Behavior Modeling. https://app.cochat.ai/papers/10.1145/3569485
- The paper
- Xu, X., Liu, X., Zhang, H., Wang, W., Nepal, S., Sefidgar, Y., Seo, W., Kuehn, K. S., Huckins, J. F., Morris, M. E., Nurius, P. S., Riskin, E. A., Patel, S., Althoff, T., Campbell, A., Dey, A. K., & Mankoff, J. (2022). GLOBEM: Cross-Dataset Generalization of Longitudinal Human Behavior Modeling. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. https://doi.org/10.1145/3569485
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.