CoChat

UniMTS: Unified Pre-training for Motion Time Series

Open access
Zhang, Xiyuan, Teng, Diyan, Chowdhury, Ranak, Li, Shuheng, Hong, Dezhi, Gupta, Rajesh, Shang, Jingbo · 2024 · Advances in Neural Information Processing Systems 37

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; none of the 21 sources its claims cite is checked for support yet.

Abstract

Motion time series collected from mobile and wearable devices such as smartphones and smartwatches offer significant insights into human behavioral patterns, with wide applications in healthcare, automation, IoT, and AR/XR due to their low-power, always-on nature. However, given security and privacy concerns, building large-scale motion time series datasets remains difficult, preventing the development of pre-trained models for human activity analysis. Typically, existing models are trained and tested on the same dataset, leading to poor generalizability across variations in device location, device mounting orientation and human activity type. In this paper, we introduce UniMTS, the first unified pre-training procedure for motion time series that generalizes across diverse device latent factors and activities. Specifically, we employ a contrastive learning framework that aligns motion time series with text descriptions enriched by large language models. This helps the model learn the semantics of time series to generalize across activities. Given the absence of large-scale motion time series data, we derive and synthesize time series from existing motion skeleton data with all-joint coverage. Spatio-temporal graph networks are utilized to capture the relationships across joints for generalization across different device locations. We further design rotation-invariant augmentation to make the model agnostic to changes in device mounting orientations. Our model shows exceptional generalizability across 18 motion time series classification benchmark datasets, outperforming the best baselines by 340% in the zero-shot setting, 16.3% in the few-shot setting, and 9.2% in the full-shot setting.

Overview

CoChat Summary

TL;DR

UniMTS is the first unified pre-training procedure for motion time series, using contrastive learning to align physics-simulated motion time series with LLM-augmented text descriptions so that a graph encoder generalizes across device location, orientation, and activity. Pre-trained only on simulated data, it achieves state-of-the-art results on 18 real-world motion time series classification benchmarks, improving over the best baselines by 340% zero-shot, 16.3% few-shot, and 9.2% full-shot.

Key findings

  • Zero-shot: UniMTS averages 43.5 (17.9) accuracy, 34.3 (14.2) macro-F1, and 58.9 (19.3) R@2 across 18 datasets, versus the best baseline ImageBind at 19.3 (14.6) accuracy, 11.0 (7.8) F1, and 24.0 (20.0) R@2; the improvement is statistically significant (e.g., p = 8 × 10⁻⁶ for ImageBind).
  • Full-shot: UniMTS averages 90.6 (10.6) accuracy and 87.5 (13.4) macro-F1, outperforming pre-trained, self-supervised, and conventional baselines (e.g., SHARE at 85.8 (13.2) accuracy; p = 0.018 for the best baseline).
  • Ablations in the zero-shot setting reduce average accuracy from 43.5 to 27.1 without rotation-invariant augmentation, 39.8 without text augmentation, and 25.1 when the graph encoder is replaced by a CNN that concatenates joints.
  • Efficiency: the graph encoder has 4.94M parameters versus 18.69M for ImageBind's IMU encoder; full-shot fine-tuning on UCI-HAR converges in ~1.3 minutes versus ~9.8 minutes for ImageBind; estimated power on an eNPU is ~22.64 mW versus ~702 mW.

Methods

  • Design: contrastive learning (CLIP-style) aligning a spatio-temporal graph convolutional encoder over simulated motion time series with a frozen CLIP text encoder; rotation-invariant augmentation samples a random rotation matrix per joint per iteration; random joint masking (1–5 joints selected) emulates partial device coverage.
  • Data: pre-training corpus is HumanML3D (14,616 3D motion skeletons, 28.59 hours, average sequence length 7.1 s, 44,970 text descriptions, SMPL 22-joint structure); accelerations and angular velocities are synthesized from skeleton positions/orientations via motion equations with added Gaussian noise; text is augmented with GPT-3.5 ("gpt-3.5-turbo") generating k = 3 paraphrases. Evaluation uses 18 real-world datasets spanning head, chest, back, arm, wrist, waist, hip, leg, knee, and ankle, grouped into easy (<10 activities), medium (10–20), and hard (>20) levels.
  • Analysis: Adam optimizer, learning rate 0.0001, batch size 64, ~13 GB memory on a single NVIDIA A100; real data resampled to 20 Hz and normalized; metrics are accuracy, macro-F1, and R@2, with Wilcoxon signed-rank tests with Holm's α (5%).

Limitations

  • Stated: simulated time series are only approximations of real signals, which are collected near rather than directly on body joints (e.g., smartwatch near the wrist); the framework is currently limited to classification and not extended to tasks such as inertial navigation; pre-training uses only existing motion datasets, with plans to add video-based pose estimation data; deployment optimization (quantization, pruning, distillation) is left to future work.
  • Evident from the text: pre-training relies on a single skeleton corpus (HumanML3D) after KIT-ML and NTU RGB+D 120 yielded no improvement; the text encoder is frozen CLIP, so text-side adaptation is not learned; and the paper reports no evaluation on inertial navigation or other non-classification tasks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.52202/079017-3414

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 2 Not valid · 13 Not checked · 56 Own results and interpretations

29 more claims

Create a free account to see all 71 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 29
  • The quote only partly states this claim. 3
  • Probably not a substantial claim. 10

Create a free account to see these claims.

Citations

Cites

Research works printed but not linked: 9

  1. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979.
  2. Rong Hu, Ling Chen, Shenghuan Miao, and Xing Tang. Swl-adapt: An unsupervised domain adaptation model with sample weight learning for cross-user wearable human activity recog- nition. In Proceedings of the AAAI Conference on artificial intelligence, volume 37, pages 6012-6020, 2023.
  3. Bing Li, Wei Cui, Wei Wang, Le Zhang, Zhenghua Chen, and Min Wu. Two-stream convolution augmented transformer for human activity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 286-293, 2021.
  4. Haojie Ma, Wenzhong Li, Xiao Zhang, Songcheng Gao, and Sanglu Lu. Attnsense: Multi-level attention mechanism for multimodal human activity recognition. In IJCAI, pages 3109-3115, 2019.
  5. Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. Data augmentation of wearable sensor data for parkinson's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international conference on multimodal interaction, pages 216-220, 2017.
  6. Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The eleventh international conference on learning representations, 2022.
  7. Huatao Xu, Pengfei Zhou, Rui Tan, and Mo Li. Practically adopting human activity recogni- tion. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking, pages 1-15, 2023.
  8. Huatao Xu, Pengfei Zhou, Rui Tan, Mo Li, and Guobin Shen. Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pages 220-233, 2021.
  9. Alexander D Young, Martin J Ling, and Damal K Arvind. Imusim: A simulation environment for inertial sensing algorithm design and evaluation. In Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks, pages 199-210. IEEE, 2011.

Cited by

See all 29

Metadata

Authors
Zhang, X., Teng, D., Chowdhury, R., Li, S., Hong, D., Gupta, R., & Shang, J.
Venue
Advances in Neural Information Processing Systems 37
Published
2024
Pages
107469-107493
ISSN
1049-5258 (ISSN-L)
OpenAlex
W4415795924
Cited by
8
References
9
Citations by year
2026: 7 · 2025: 1
FWCI
3.05
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2333790, CAREER, 2239440)
  • Semiconductor Research Corporation (JUMP 2.0)
  • Defense Advanced Research Projects Agency
Topics
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
Keywords
pre-training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.