CoChat

Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognition

Open access

The DOI matches its Crossref record; 48 of the 81 references it cites were found in a registry or an index; 11 of the 32 sources its claims cite were checked for support.

Abstract

Human activity recognition (HAR) in wearable and ubiquitous computing typically involves translating sensor readings into feature representations, either derived through dedicated pre-processing procedures or integrated into end-to-end learning approaches. Independent of their origin, for the vast majority of contemporary HAR methods and applications, those feature representations are typically continuous in nature. That has not always been the case. In the early days of HAR, discretization approaches had been explored—primarily motivated by the desire to minimize computational requirements on HAR, but also with a view on applications beyond mere activity classification, such as, for example, activity discovery, fingerprinting, or large-scale search. Those traditional discretization approaches, however, suffer from substantial loss in precision and resolution in the resulting data representations with detrimental effects on downstream analysis tasks. Times have changed, and in this paper, we propose a return to discretized representations. We adopt and apply recent advancements in vector quantization (VQ) to wearables applications, which enables us to directly learn a mapping between short spans of sensor data and a codebook of vectors, where the index comprises the discrete representation, resulting in recognition performance that is at least on par with their contemporary, continuous counterparts—often surpassing them. Therefore, this work presents a proof of concept for demonstrating how effective discrete representations can be derived, enabling applications beyond mere activity classification but also opening up the field to advanced tools for the analysis of symbolic sequences, as they are known, for example, from domains such as natural language processing. Based on an extensive experimental evaluation of a suite of wearable-based benchmark HAR tasks, we demonstrate the potential of our learned discretization scheme and discuss how discretized sensor data analysis can lead to substantial changes in HAR.

Overview

CoChat Summary

TL;DR

The paper proposes learning discrete representations for wearable-based human activity recognition by adding vector quantization to an Enhanced CPC self-supervised framework, then classifying the resulting symbolic sequences. Across six benchmarks, the learned discrete representations perform comparably to or better than continuous self-supervised and end-to-end baselines on several tasks, especially locomotion and daily-living activities.

Key findings

  • VQ-CPC discrete representations performed best or similar to the best on three benchmark datasets: Mobiact reached a mean of 77.8%, and HHAR improved over Enhanced CPC by around 1.5%.
  • Discretization was detrimental for fine-grained gym activities: on Myogym, performance dropped significantly compared with other self-supervised methods, and on MHEALTH it dropped by around 4.8% relative to Enhanced CPC.
  • Learned VQ-CPC outperformed traditional SAX and SAX-REPEAT discretization; SAX showed reductions of over 10% for HHAR, Myogym, and Motionsense.
  • Adding RoBERTa pre-training on the discrete symbols improved all target datasets: small RoBERTa increased HHAR by 2%, Myogym by 2.8%, Mobiact by 1.6%, Motionsense by 2.5%, and leg-based datasets by around 2.5%; medium RoBERTa increased HHAR by 3% and Myogym by 3.5%.

Methods

  • Pre-training used the Capture-24 dataset: around 2500 h from 151 participants, single wrist-worn accelerometer, downsampled to 50 Hz; evaluation used six target datasets across wrist, waist, and leg locations, with two datasets per location.
  • The framework combines Enhanced CPC self-supervision with online K-means vector quantization; base setup used G=2 groups and V=100 codebook variables, with a GRU/LSTM classifier and MLP backend, evaluated via five-fold cross-validation and five randomized runs.
  • Additional experiments varied learned alphabet size, encoder output frequency, base self-supervised method, and RoBERTa small/medium pre-training on discrete sequences.

Limitations

  • The paper states this is a proof of concept and that discretization is detrimental for fine-grained movements, as seen on Myogym; it also notes that the medium RoBERTa model did not improve over the small one, likely because Capture-24 is not large enough to leverage the bigger architecture.
  • The paper states that lower encoder output frequency substantially reduces performance, with drops around 10% for HHAR, over 15% for Myogym/MHEALTH/PAMAP2, and over 20% for waist-based datasets.
  • Evident from the text: pre-training and evaluation use a single accelerometer, and evaluation covers two datasets per sensor location rather than the three used in the referenced assessment framework.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/s24041238

Cites

48 of 81 references verified0 citation errors · 33 not checkable

Statistics

Nothing to recompute.

Figures

Figure A1 Using RoBERTa embeddings for classifying discrete representations: first, we pre-train a RoBERTa model on discrete representations (indexed) from Capture-24. Subsequently, we replace the learnable embeddings with frozen RoBERTa embeddings in order to study the impact of the NLP-based pre-training. A GRU network is used along with an MLP for activity recognition.
Figure A1 Using RoBERTa embeddings for classifying discrete representations: first, we pre-train a RoBERTa model on discrete representations (indexed) from Capture-24. Subsequently, we replace the learnable embeddings with frozen RoBERTa embeddings in order to study the impact of the NLP-based pre-training. A GRU network is used along with an MLP for activity recognition.CC BY · PubMed Central
Figure A2 Visualizing the histograms of discrete representations per class for the first fold of Motionsense. This is the full figure from which Figure 5 is obtained by truncating the x-axis to 50 symbols.
Figure A2 Visualizing the histograms of discrete representations per class for the first fold of Motionsense. This is the full figure from which Figure 5 is obtained by truncating the x-axis to 50 symbols.CC BY · PubMed Central
Figure A3 Visualizing how the weights of the VQ-CPC architecture are updated using different parts of the overall loss. As detailed in Section 3.1.1, the Aggregator network is updated using (∑)k=1^KLk^CPC whereas the codebook vectors utilize ∥sg(z)−z^(∥)². Finally, the encoder is updated using both (∑)k=1^KLk^CPC and γ∥z−sg(z^)(∥)².
Figure A3 Visualizing how the weights of the VQ-CPC architecture are updated using different parts of the overall loss. As detailed in Section 3.1.1, the Aggregator network is updated using (∑)k=1^KLk^CPC whereas the codebook vectors utilize ∥sg(z)−z^(∥)². Finally, the encoder is updated using both (∑)k=1^KLk^CPC and γ∥z−sg(z^)(∥)².CC BY · PubMed Central
Figure 1 Discrete representation learning: given a window of accelerometer data as input, the output is a collection of enumerated symbols. We map short spans of continuous-valued sensor data to a list of ‘symbols’ (i.e., the numbers in the figure). Therefore, we obtain the “strings of motion”, as the symbols are discrete.
Figure 1 Discrete representation learning: given a window of accelerometer data as input, the output is a collection of enumerated symbols. We map short spans of continuous-valued sensor data to a list of ‘symbols’ (i.e., the numbers in the figure). Therefore, we obtain the “strings of motion”, as the symbols are discrete.CC BY · PubMed Central
Figure 2 Overview of discrete representation learning for HAR using wearables: we combine a contrastive future timestep prediction problem with vector quantization to map spans of sensor data to a codebook of vectors. The index of the codebook vector closest to each zt functions as the discrete representation.
Figure 2 Overview of discrete representation learning for HAR using wearables: we combine a contrastive future timestep prediction problem with vector quantization to map spans of sensor data to a codebook of vectors. The index of the codebook vector closest to each zt functions as the discrete representation.CC BY · PubMed Central
Figure 3 Visualizing K-means-based quantization: for each z-vector, the l2 distance is computed to the codebook of vectors ei. The index of the nearest codebook vector comprises the discrete representation, whereas the nearest vector itself is passed as the output (i.e., z^-vector). This figure has been adopted and adapted from [27].
Figure 3 Visualizing K-means-based quantization: for each z-vector, the l2 distance is computed to the codebook of vectors ei. The index of the nearest codebook vector comprises the discrete representation, whereas the nearest vector itself is passed as the output (i.e., z^-vector). This figure has been adopted and adapted from [27].CC BY · PubMed Central
Figure 4 Performing classification using the discrete representations: the sequences of symbolic representations (indexed) are first passed through a learnable embedding layer. Subsequently, an RNN network (GRU or LSTM) is utilized along with an MLP network for classifying the sequences into the activities of interest.
Figure 4 Performing classification using the discrete representations: the sequences of symbolic representations (indexed) are first passed through a learnable embedding layer. Subsequently, an RNN network (GRU or LSTM) is utilized along with an MLP network for classifying the sequences into the activities of interest.CC BY · PubMed Central
Figure 5 Plotting the histograms per class of the derived discrete representations for the train set from the first fold of Motionsense: The y-axis corresponds to the fraction of the number of occurrences for the symbols computed against the available data. The x-axis comprises the discrete representations, which are numbered. We observe that standing and sitting are covered by only a few symbols, which can be reasoned by the lack of movements in these classes. Walking up and down the stairs is also similar, yet hard to distinguish via visual examination from both walking and running. For clarity, we truncate the y-axis to 0.6 and x-axis to 50 symbols in order to show the plot in more detail. The full figure is available in the Appendix (refer to Figure A2).
Figure 5 Plotting the histograms per class of the derived discrete representations for the train set from the first fold of Motionsense: The y-axis corresponds to the fraction of the number of occurrences for the symbols computed against the available data. The x-axis comprises the discrete representations, which are numbered. We observe that standing and sitting are covered by only a few symbols, which can be reasoned by the lack of movements in these classes. Walking up and down the stairs is also similar, yet hard to distinguish via visual examination from both walking and running. For clarity, we truncate the y-axis to 0.6 and x-axis to 50 symbols in order to show the plot in more detail. The full figure is available in the Appendix (refer to Figure A2).CC BY · PubMed Central

Verification

Claims the paper makes

9 Valid · 0 Not valid · 13 Not checked · 97 Own results and interpretations

51 more claims

Create a free account to see all 119 claims and to verify the 21 references still open.

See all claims — free account
  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 45
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 21

  1. Bock M. Kuehne H. Van Laerhoven K. Moeller M. WEAR: An Outdoor Sports for Wearable and Egocentric Activity Recognition arXiv 2023 2304.05088
  2. Baevski A. Schneider S. Auli M. vq-wav2vec: Self-supervised learning of discrete speech representations arXiv 2019 1910.05453
  3. Van Den Oord A. Vinyals O. Neural discrete representation learning Advances in Neural Information Processing Systems MIT Press Cambridge, MA, USA 2017 Volume 30
  4. Chan Chang S. Doherty A. Capture-24: Activity Tracker Dataset for Human Activity Recognition University of Oxford Oxford, UK 2021
  5. Liu Y. Ott M. Goyal N. Du J. Joshi M. Chen D. Levy O. Lewis M. Zettlemoyer L. Stoyanov V. Roberta: A robustly optimized bert pretraining approach arXiv 2019 1907.11692
  6. Devlin J. Chang M.W. Lee K. Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding arXiv 2018 1810.04805
  7. Shieh J. Keogh E. i SAX: Indexing and mining terabyte sized time series Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Las Legas, NV, USA 24–27 August 2008 623 631
  8. Ye L. Keogh E. Time series shapelets: A new primitive for data mining Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Paris, France 28 June–1 July 2009 947 956
  9. Jang E. Gu S. Poole B. Categorical reparameterization with gumbel-softmax arXiv 2016 1611.01144
  10. Cuervo S. ańcucki A. Marxer R. Rychlikowski P. Chorowski J. Variable-rate hierarchical CPC leads to acoustic unit discovery in speech arXiv 2022 2206.02211
  11. Dieleman S. Nash C. Engel J. Simonyan K. Variable-rate discrete representation learning arXiv 2021 2103.06089
  12. Kukačka J. Golkov V. Cremers D. Regularization for deep learning: A taxonomy arXiv 2017 1710.10686
  13. Oord A.v.d. Li Y. Vinyals O. Representation learning with contrastive predictive coding arXiv 2018 1807.03748
  14. Bengio Y. Léonard N. Courville A. Estimating or propagating gradients through stochastic neurons for conditional computation arXiv 2013 1308.3432
  15. Paszke A. Gross S. Massa F. Lerer A. Bradbury J. Chanan G. Killeen T. Lin Z. Gimelshein N. Antiga L. Pytorch: An imperative style, high-performance deep learning library Advances in Neural Information Processing Systems MIT Press Cambridge, MA, USA 2019 Volume 32
  16. Kingma D.P. Ba J. Adam: A method for stochastic optimization arXiv 2014 1412.6980
  17. Lan Z. Chen M. Goodman S. Gimpel K. Sharma P. Soricut R. Albert: A lite bert for self-supervised learning of language representations arXiv 2019 1909.11942
  18. Sanh V. Debut L. Chaumond J. Wolf T. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter arXiv 2019 1910.01108
  19. Vaswani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A.N. Kaiser Ł. Polosukhin I. Attention is all you need Advances in Neural Information Processing Systems MIT Press Cambridge, MA, USA 2017 Volume 30
  20. Radford A. Wu J. Child R. Luan D. Amodei D. Sutskever I. Language models are unsupervised multitask learners Openai Blog 2019 1 9
  21. Gokhan T. Smith P. Lee M. Extractive financial narrative summarisation using sentencebert based clustering Proceedings of the 3rd Financial Narrative Processing Workshop Lancaster, UK 15–16 September 2021 94 98

Cited by

See all 18

Metadata

Authors
Haresamudram, H., Essa, I., & Plötz, T.
Affiliations
Georgia Institute of Technology (US)
Corresponding author
Haresamudram, Harish
Venue
Sensors
Publisher
MDPI AG
Published
2024-02-15
Volume
24
Issue
4
Pages
1238
ISSN
1424-8220 (ISSN-L)
OpenAlex
W4391846608
Cited by
14
References
77
Citations by year
2026: 4 · 2025: 5 · 2024: 4
FWCI
1.82
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • arXiv (Cornell University)
Funders
NSF (IIS-2112633)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
Keywords
human activity recognition, wearable sensors, vector quantization, discrete representations, self-supervised learning, codebook learning, activity classification
MeSH terms
Machine Learning, Wearable Electronic Devices, Humans, Human Activities, Natural Language Processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.