Let's (not) stick together: pairwise similarity biases cross-validation in activity recognition
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
The ability to generalise towards either new users or unforeseen behaviours is a key requirement for activity recognition systems in ubiquitous computing. Differences in recognition performance for the two application cases can be significant, and user-dependent performance is typically assumed to be an upper bound on performance. We demonstrate that this assumption does not hold for the widely used cross-validation evaluation scheme that is typically employed both during system bootstrapping and for reporting results. We describe how the characteristics of segmented time-series data render random cross-validation a poor fit, as adjacent segments are not statistically independent. We develop an alternative approach -- meta-segmented cross validation -- that explicitly circumvents this issue and evaluate it on two data-sets. Results indicate a significant drop in performance across a variety of feature extraction and classification methods if this bias is removed, and that prolonged, repetitive activities are particularly affected.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“subsequent, often overlapping, windows of sensor readings are not i.i.d. and thus care needs to be taken during model training and evaluation[73]”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [73]
In the citing paper
“Overall, every participant appears exactly once as part of the test split and pairwise correlations of analysis windows are effectively eliminated through the user-based splits [22].”
From Assessing the State of Self-Supervised Human Activity Recognition Using Wearables (2022) · cites this paper as [22]
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“We apply support vector machines with default parameters from libsvm [8] with RBF-kernel, a cost parameter of 1 and the kernel scaling parameter set to 1 / # features.”
MethodBook Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066 · cited 2×
“They require significant computational resources during training and potentially extensive experimentation to find suitable parameter settings [4].”
BackgroundMethodThe WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 1×
“We do so as classifiers are usually selected as an off-the-shelf component made available in common machine learning frameworks such as WEKA [13].”
MethodPattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
(2006) · Springer eBooks · Cited by 8,123
A tutorial on human activity recognition using body-worn inertial sensors
Bulling, A., Blanke, U., & Schiele, B. (2014) · ACM Computing Surveys · doi:10.1145/2499621 · Cited by 1,668 · cited 4×
“The sensor data in ubiquitous computing is usually analysed using a pipeline-based approach [7].”
BackgroundMethodInfluentialIntroducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 2×
“It is the de-facto standard that subsequent frames overlap with the last segment to a certain extent, such as 50% [24] or more, e.g. 80% in [30].”
BackgroundEstimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap
Kim, J.-H. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2009.04.009 · Cited by 792 · cited 1×
“Cross-validation is widely used throughout machine learning and has been demonstrated to provide reliable performance figures [19].”
BackgroundCollecting complex activity datasets in highly rich networked sensor environments
Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. (2010) · 2010 Seventh International Conference on Networked Sensing Systems (INSS) · doi:10.1109/inss.2010.5573462 · Cited by 788
SEGMENTING TIME SERIES: A SURVEY AND NOVEL APPROACH
Keogh, E., Chu, S., Hart, D., & Pazzani, M. (2004) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789812565402_0001 · Cited by 600 · cited 1×
“However, the most popular segmentation approach in activity recognition in ubiquitous computing corresponds to a sliding window segmentation [18]: Segments (or frames ) are extracted by “sliding” a window of fixed length over the sensor stream.”
BackgroundPreprocessing techniques for context recognition from accelerometer data
Figo, D., Diniz, P. C., Ferreira, D. R., & Cardoso, J. M. P. (2010) · Personal and Ubiquitous Computing · doi:10.1007/s00779-010-0293-9 · Cited by 578 · cited 1×
“FFTs are computationally efficient to extract and often used as a component in feature extraction approaches [9].”
MethodFeature Selection and Activity Recognition System Using a Single Triaxial Accelerometer
Gupta, P., & Dallas, T. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2307069 · Cited by 335
Classification Accuracies of Physical Activities Using Smartphone Motion Sensors
Wu, W., Dasgupta, S., Ramirez, E. E., Peterson, C., & Norman, G. J. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2208 · Cited by 316 · cited 1×
“For example, in [36] KNN outperforms neural networks, decision trees and other methods in CV experiments on data collected from physical activities.”
MethodBeating the hold-out: bounds for K-fold and progressive cross-validation
Blum, A., Kalai, A., & Langford, J. (1999) · Proceedings of the twelfth annual conference on Computational learning theory · doi:10.1145/307400.307439 · Cited by 265 · cited 3×
“Hold-out validation, while simple and well established in, e.g. general machine learning, may not give sufficient insight into the performance of the system for robust model selection [5], as it is unsuitable for estimation of the variance in performance.”
BackgroundInfluentialActivity recognition using smartphone sensors
Anjum, A., & Ilyas, M. U. (2013) · 2013 IEEE 10th Consumer Communications and Networking Conference (CCNC) · doi:10.1109/ccnc.2013.6488584 · Cited by 233 · cited 1×
“Decision trees are popular in activity recognition, e.g. for the recognition of physical activities [2].”
BackgroundTowards Physical Activity Recognition Using Smartphone Sensors
Shoaib, M., Scholten, H., & Havinga, P. J. M. (2013) · 2013 IEEE 10th International Conference on Ubiquitous Intelligence and Computing and 2013 IEEE 10th International Conference on Autonomic and Trusted Computing · doi:10.1109/uic-atc.2013.43 · Cited by 228 · cited 1×
“Similarly, [33] reports superior performance fig-ures for KNN compared to many other methods.”
ResultPerformance metrics for activity recognition
Ward, J. A., Lukowicz, P., & Gellersen, H. W. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1889681.1889687 · Cited by 182 · cited 1×
“While performance metrics used in ubiquitous computing have been subject to intensive study (see e.g. [35]), so far no work has investigated these different evaluation schemes for specific characteristics or biases.”
BackgroundPD Disease State Assessment in Naturalistic Environments Using Deep Learning
Hammerla, N., Fisher, J., Andras, P., Rochester, L., Walker, R., & Ploetz, T. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9484 · Cited by 179 · cited 2×
“This could be easily achieved by reducing the overlap between subsequent frames in the sliding window procedure, or to even remove the overlap completely (like in [14]).”
BackgroundSelf-citationHammerla, N. Y., Kirkham, R., Andras, P., & Ploetz, T. (2013) · Proceedings of the 2013 International Symposium on Wearable Computers · doi:10.1145/2493988.2494353 · Cited by 138 · cited 2×
“For all experiments we chose to set the feature parameter (number of interpolation points), to 15 , which has shown good performance across different settings [15].”
BackgroundMethodSelf-citationQualitative activity recognition of weight lifting exercises
Velloso, E., Bulling, A., Gellersen, H., Ugulino, W., & Fuks, H. (2013) · Proceedings of the 4th Augmented Human International Conference · doi:10.1145/2459236.2459256 · Cited by 130 · cited 1×
“For example, [34] report a performance of > 95 % accuracy in CV, which drops to 78 .”
BackgroundAutomatic assessment of problem behavior in individuals with developmental disabilities
Plötz, T., Hammerla, N. Y., Rozga, A., Reavis, A., Call, N., & Abowd, G. D. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370276 · Cited by 128
Self-citationClimbAX: skill assessment for climbing enthusiasts
Ladha, C., Hammerla, N. Y., Olivier, P., & Plötz, T. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493492 · Cited by 120
Self-citationDog's life: wearable activity recognition for dogs
Ladha, C., Hammerla, N., Hughes, E., Olivier, P., & Ploetz, T. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493519 · Cited by 112
Self-citationOnline pose classification and walking speed estimation using handheld devices
Park, J.-G., Patel, A., Curtis, D., Teller, S., & Ledlie, J. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370235 · Cited by 109
Designing a Robust Activity Recognition Framework for Health and Exergaming Using Wearable Sensors
Alshurafa, N., Xu, W., Liu, J. J., Huang, M.-C., Mortazavi, B., Roberts, C. K., & Sarrafzadeh, M. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2287504 · Cited by 105
Intelligent sleep stage mining service with smartphones
Gu, W., Yang, Z., Shangguan, L., Sun, W., Jin, K., & Liu, Y. (2014) · Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2632048.2632084 · Cited by 77
Detecting leisure activities with dense motif discovery
Berlin, E., & Van Laerhoven, K. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370257 · Cited by 67 · cited 1×
“Evaluation schemes described as CV by the respective authors, but which actually correspond to repeated hold-out validation (e.g. leave-one-day -out [3]) are classified as hold-out validation.”
MethodPervasive stress recognition for sustainable living
Bogomolov, A., Lepri, B., Ferron, M., Pianesi, F., & Pentland, A. S. (2014) · 2014 IEEE International Conference on Pervasive Computing and Communication Workshops (PERCOM WORKSHOPS) · doi:10.1109/percomw.2014.6815230 · Cited by 54 · cited 1×
“[6] Stress levels phone 117 x Random Forest”
BackgroundPersonalized mobile physical activity recognition
Reiss, A., & Stricker, D. (2013) · Proceedings of the 2013 International Symposium on Wearable Computers · doi:10.1145/2493988.2494349 · Cited by 53
Classifying social actions with a single accelerometer
Hung, H., Englebienne, G., & Kools, J. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493513 · Cited by 44
Sensor-embedded teeth for oral activity recognition
Li, C., Chen, Y., Chen, W.-J., Polly Huang, B., & Chu, H.-H. (2013) · International Symposium on Wearable Computers · doi:10.1145/2493988.2494352 · Cited by 44 · cited 3×
“It is the de-facto standard that subsequent frames overlap with the last segment to a certain extent, such as 50% [24] or more, e.g. 80% in [30].”
BackgroundActivity Recognition and Healthier Food Preparation
Plötz, T., Moynihan, P., Pham, C., & Olivier, P. (2011) · Atlantis Ambient and Pervasive Intelligence · doi:10.2991/978-94-91216-05-3_14 · Cited by 39 · cited 1×
“We chose to replicate the feature representation from [28] which corresponds to 23 extracted features for each sensor used in the recordings.”
MethodSelf-citationComparison of Classification Algorithms for Physical Activity Recognition
Peterek, T., Penhaker, M., Gajdoš, P., & Dohnálek, P. (2013) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-01781-5_12 · Cited by 35
Confidence-based multiclass AdaBoost for physical activity monitoring
Reiss, A., Hendeby, G., & Stricker, D. (2013) · International Symposium on Wearable Computers · doi:10.1145/2493988.2494325 · Cited by 27 · cited 1×
“[29] Physical acc, cardio 9 x x Adaboost”
BackgroundA hybrid unsupervised/supervised model for group activity recognition
Hirano, T., & Maekawa, T. (2013) · International Symposium on Wearable Computers · doi:10.1145/2493988.2494348 · Cited by 22
Connecting personal-scale sensing and networked community behavior to infer human activities
Lane, N. D., Li, P., Zhou, L., & Zhao, F. (2014) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2632048.2636094 · Cited by 21 · cited 1×
“[23] Community activities various 123 x SVM”
Ensembles of multiple sensors for human energy expenditure estimation
Gjoreski, H., Kaluža, B., Gams, M., Milić, R., & Luštrek, M. (2013) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2493432.2493517 · Cited by 16
Research works printed but not linked: 1
- Kose , M. , Incel , O. D. , and Ersoy , C . Online human activity recognition on smart phones . In Workshop on Mobile Sensing: From Smartphones and Wearables to Big Data ( 2012 ), 11--15. Kose, M., Incel, O. D., and Ersoy, C. Online human activity recognition on smart phones. In Workshop on Mobile Sensing: From Smartphones and Wearables to Big Data (2012), 11--15.
Cited by
Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606
“However, more elaborate behaviours which are, for example, of interest in medical applications, pose a significant challenge to this manually tuned approach [Hammerla et al. , 2015 ] .”
BackgroundSelf-citationEnsembles of Deep LSTM Learners for Activity Recognition using Wearables
Guan, Y., & Plötz, T. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3090076 · Cited by 511
Self-citationREFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 129
A review of state-of-the-art techniques for abnormal human activity recognition
Dhiman, C., & Vishwakarma, D. K. (2019) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2018.08.014 · Cited by 265
Wang, R., Aung, M. S. H., Abdullah, S., Brian, R., Campbell, A. T., Choudhury, T., Hauser, M., Kane, J., Merrill, M., Scherer, E. A., et al. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2971648.2971740 · Cited by 257
“We use a 10-fold blocked cross validation method [12,16,28, 48] to evaluate the prediction performance of the individual models and mixed models.”
Method
Metadata
- Authors
- Hammerla, N. Y., & Plötz, T.
- Affiliations
- Newcastle University (GB)
- Venue
- Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing
- Publisher
- ACM
- Published
- 2015-09-07
- Pages
- 1041-1051
- OpenAlex
- W2155961734
- Cited by
- 91
- References
- 37
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 9 · 2023: 8 · 2022: 7 · 2021: 13 · 2020: 15 · 2019: 5 · 2018: 8 · 2017: 10 · 2016: 5
- FWCI
- 3.14
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) (publishedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/G066019/1)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Social Sciences › Transportation › Human Mobility and Location-Based Analysis
- Keywords
- activity recognition, time series segmentation, ubiquitous computing, feature extraction, classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.