TS2ACT: Few-Shot Human Activity Sensing with Cross-Modal Co-Learning

TS2ACT outperforms the other baseline models by a large margin in all few-shot cases.

“According to the results, TS2ACT outperforms the other baseline models by a large margin in all few-shot cases.”

TS2ACT significantly outperforms the supervised algorithms DCNN, TCN, and ConformerHAR under few-shot conditions.

“TS2ACT significantly outperforms the other supervised algorithms DCNN, TCN and ConformerHAR under few-shot conditions.”

TS2ACT outperforms the unsupervised representation learning methods SimCLR and Multi-task deep clustering.

“TS2ACT outperforms other unsupervised representation learning methods, such as SimCLR and Multitask deep cluster”

TS2ACT outperforms the semi-supervised representation learning method FixMatch.

“TS2ACT outperforms other semi-supervised representation learning methods such as FixMatch.”

TS2ACT achieves competitive results when using the entire labeled sample set.

“It also achieves competitive results when utilizing the entire labeled samples, which demonstrates the effectiveness of TS2ACT.”

TS2ACT for 1-shot learning achieves accuracy over 0.7 on all datasets.

“TS2ACT for 1-shot (i.e, using one labeled samples for each class) achieves accuracy over 0.7 on all datasets.”

TS2ACT even outperforms the supervised algorithms when using the full labeled dataset in some cases.

“It even outperforms them when using the full labeled dataset for model training in some cases.”

The performance gain is more than a factor of 2 in scenarios with an extremely limited number of labeled samples.

“The performance is significantly improved, with a performance gain of more than a factor of 2 in scenarios with an extremely limited number of labeled samples.”

For 20-shot learning, TS2ACT's performance is close to or better than fully-supervised methods trained with all labeled samples.

“which is close to or better than the performance of fully-supervised methods when all labeled samples are used to train their models.”

TS2ACT achieves performance close to or better than fully supervised methods even when using as few as 1% labeled data for model training.

“it achieves performance close to or better than the fully supervised methods even using as few as 1% labeled data for model training.”

• Also stated in: § 1 INTRODUCTION · plain

  “it performs closely to the fully supervised methods even using labeled data as few as 1% of the training dataset.”

• Also stated in: § 1 INTRODUCTION · plain

  “it performs close to or better than the fully supervised methods using as few as 1% labeled data for model training.”

For 20-shot learning, TS2ACT achieves accuracy and F1-score ranging from 0.88 to 0.92 on all datasets.

“For 20-shot learning, TS2ACT achieves accuracy and F1-score ranging from 0.88-0.92 on all datasets”

Compared with ConformerHAR on the PAMAP2 dataset, TS2ACT improves performance across almost all activities.

“our model demonstrates improved performance across almost all activities.”

TS2ACT significantly outperforms state-of-the-art human activity recognition methods under few-shot conditions.

“The numerical results show that TS2ACT significantly outperforms the state-of-the-art HAR methods”

• Also stated in: § 1 INTRODUCTION · plain

  “The numerical results show that TS2ACT significantly outperforms the state-of-the-art HAR methods, and it performs closely to the fully supervised methods even using labeled data as few as 1% of the training dataset.”

• Also stated in: § 1 INTRODUCTION · plain

  “It is showed that TS2ACT significantly outperforms the state-of-the-art HAR methods under fewshot conditions.”

As the percentage of labeled samples increases, the models achieve higher accuracy.

“As the percentage of labeled samples increases, the models can achieve higher accuracy.”

TS2ACT consistently outperforms other models in all cases across different percentages of labeled samples.

“The results show that TS2ACT consistently outperforms other models in all cases.”

As the number of images increases, the recognition performance of the model increases.

“It shows that as the number of images increases, the recognition performance of the model also increases.”

TS2ACT produces informative features that cluster human activities well with few labeled data.

“It can be seen that the proposed TS2ACT can generate informative features that cluster human activities effectively using a small amount of labeled data.”

Even with only a few labeled data, TS2ACT demonstrates outstanding recognition capabilities.

“Even with only a few labeled data, TS2ACT demonstrates outstanding recognition capabilities”

Image augmentation performs significantly better than text augmentation on the four datasets.

“According to the results, image augmentation performs significantly better than text augmentation on the four datasets”

Across four public datasets, TS2ACT substantially outperforms state-of-the-art human activity recognition methods.

“Extensive experiments based on four public datasets showed that TS2ACT significantly outperforms the state-of-the-art HAR methods”

Even when labeled data are limited, TS2ACT achieves performance comparable to fully supervised models.

“Even when the labeled data samples are limited, the proposed method can achieve comparable performance with the fully supervised models.”

As the number of Transformer blocks increases, performance initially increases and then degrades after reaching some number.

“It shows that as the number of Transformer blocks increases, the performance of the model increases at the beginning.After reaching some number, the performance degrades when the blocks increases.”

The smaller the percentage of labeled samples, the larger the performance gap between TS2ACT and other models.

“And the smaller percentage of labeled samples, the bigger the performance gap between TS2ACT and other models.”

With only a small number of labeled samples, TS2ACT matches or exceeds fully supervised methods.

“it performed close to or better than the fully supervised methods even using a small number of labeled samples”

When the number of images reaches 40, the performance of the model gradually converges.

“When the number of images reaches 40, the performance of the model gradually converges.”

The cross-modal prompt-learning inference method enables efficient activity recognition without training an additional classifier.

“we propose a cross-modal method using prompt learning to enable efficient inference.This approach effectively utilizes the embeddings encoded by the TS encoder along with a pre-traind CLIP text encoder for activity recognition without additional training of a classifier.”

Four publicly available datasets cover a wide variety of device types and activity recognition tasks in different environments.

“We consider four publicly available datasets that cover a wide variety of device types and activity recognition tasks in different environments.”

The labeled contrastive loss causes the time-series encoder to learn embeddings that are close to those produced by the CLIP image encoder.

“L 𝑙𝑎𝑏𝑒𝑙𝑒𝑑 𝑐𝑜𝑛𝑡𝑟𝑎𝑠𝑡 enables the TS encoder to learn embeddings that are close to those generated by the CLIP image encoder”

Adding prompts to label text before encoding it with the CLIP text encoder improves classification accuracy.

“To enhance the classification accuracy, we augment the original label text (i.e., 𝐿 1 , 𝐿 2 , 𝐿 3 , ..., 𝐿 𝐾 ) with prompts before feeding it to the CLIP text encoder.”

HHAR covers 36 smart devices of 13 models from 4 manufacturers with varying sampling rates.

“The dataset covers a diverse set of devices, including 36 smart devices of 13 models from 4 manufacturers, with varying sampling rates to accommodate heterogeneous data.”

• cites [46] sourceNot checked yet

  Not checked in the free run.

The time series encoder learns to map input time series into a feature space close to that of the CLIP image encoder.

“In this way, the TS encoder learns to map the input time series into the feature space close to that of the CLIP image encoder.”

• Also stated in: § 3.3.4 Cross-Modal Contrastive Learning. · plain

  “the TS encoder learns to encode time series data into a feature space closely aligned with that of the image encoder.”

With the augmented semantic-rich image dataset and pre-trained CLIP encoders, sensor-based human activity recognition can be fulfilled with very few labeled samples.

“With the help of the augmented semanticrich image dataset and the pre-trained CLIP encoders, the sensor-based human activity recognition task can be fulfilled with very few labeled samples.”

• AssumptionRests on an assumption: The labels of human activities contain rich semantic information that can be used as keywords to search for images sharing the same labels.

The unlabeled contrastive loss enables the model to use unlabeled data to improve feature extraction by the time-series encoder.

“L 𝑢𝑛𝑙𝑎𝑏𝑒𝑙𝑒𝑑 𝑐𝑜𝑛𝑡𝑟𝑎𝑠𝑡 enables the model to fully leverage unlabeled data to further enhance feature extraction for the TS encoder.”

With co-learning on time-series and image visual features, the time-series encoder embeddings can be aligned to label text for accurate classification.

“Furthermore, with the proposed co-learning on time series features and image visual features, the embedding generated by the TS encoder can be cued to the label text for accurate classification.”

• AssumptionRests on an assumption: Because of CLIP's text-image matching pre-training, the text encoder generates embeddings near the matched image embeddings.

Images can be used to train a cross-modal classifier to compensate for the shortage of labeled samples.

“The image can be used to train a crossmodal classifier to make up the shortage of labeled samples.”

• AssumptionRests on an assumption: The labels of human activities contain rich semantic information that can be used as keywords to search for images sharing the same labels.

TS2ACT extracts informative features that are more discriminative among different activity classes because of contrastive learning.

“The proposed TS2ACT can extract informative features that are more discriminative among different activity classes, thanks to the power of contrastive learning.”

The proposed co-learning method effectively associates diverse types of multimodal data to enhance the comprehension of human activities.

“The proposed co-learning method effectively associates diverse types of multimodal data to enhance the comprehension of human activities, demonstrating the power of using multimodal data joint training approaches.”

The cause of this limitation is that the pretrained image classifier cannot handle unseen classes.

“The limitation is caused by the pretrained image classifier that cannot deal with unseen classes.”

Augmenting with static images cannot represent complex sequential activities that involve multiple processes.

“the proposed method augments dataset with static image, which fails to describe complex sequential activities involving multiple processes”

When lambda is too small, the model pays more attention to supervised learning and cannot effectively use the large amount of unlabeled data.

“When 𝜆 is too small, the model pays more attention to supervised learning, which is unable to effectively use the large amount of unlabeled data.”

In text augmentation, having only one label text per class limits the number of augmented samples.

“In text augmentation, since there is only one label text for each class, the augmented samples are rather limit even the synonyms of a word are counted.”

TS2ACT may output incorrect results for activities that CLIP has not seen.

“When encountering activities that are unseen by CLIP, the proposed method could produce wrong results.”

Because TS2ACT strengthens semantic features via images, its inability to capture activity direction is a limitation.

“Since our approach enhances semantic features through images, the lack of directional information for activities becomes a limitation.”

• AssumptionRests on an assumption: Many activities look visually similar yet are distinct, and sensor data can measure them.

When lambda is large, the model pays more attention to unsupervised learning and cannot effectively use supervision from the labeled time-series data.

“When 𝜆 is large, the model pays more attention to unsupervised learning, which makes it unable to effectively use supervision from the labeled TS data.”

TS2ACT would likely fail on temporally opposite activities.

“Applying TS2ACT to recognize such temporally opposite activities probably lead to failure.”

• AssumptionRests on an assumption: Many activities look visually similar yet are distinct, and sensor data can measure them.