CoChat
Sign Up
Cited by 182Context-Aware Activity Recognition SystemsHuman Pose and Action RecognitionTime Series Analysis and Forecasting

Performance metrics for activity recognition

Open access
Ward, Jamie A., Lukowicz, Paul, Gellersen, Hans W.·2011·ACM Transactions on Intelligent Systems and Technology
DOI 10.1145/1889681.1889687
Open this paper in CoChat — free accountOpen PDF

Abstract

In this article, we introduce and evaluate a comprehensive set of performance metrics and visualisations for continuous activity recognition (AR). We demonstrate how standard evaluation methods, often borrowed from related pattern recognition problems, fail to capture common artefacts found in continuous AR—specifically event fragmentation, event merging and timing offsets. We support our assertion with an analysis on a set of recently published AR papers. Building on an earlier initial work on the topic, we develop a frame-based visualisation and corresponding set of class-skew invariant metrics for the one class versus all evaluation. These are complemented by a new complete set of event-based metrics that allow a quick graphical representation of system performance—showing events that are correct, inserted, deleted, fragmented, merged and those which are both fragmented and merged. We evaluate the utility of our approach through comparison with standard metrics on data from three different published experiments. This shows that where event- and frame-based precision and recall lead to an ambiguous interpretation of results in some cases, the proposed metrics provide a consistently unambiguous explanation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/1889681.1889687

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 38 · Cited by 182

Research works printed but not linked: 32

  1. Agrawal, R., Lin, K.-I., Sawhney, H. S., and Shim, K. 1995. Fast similarity search in the presence of noise, scaling, and translation in time-series databases. In Proc. 21st Int'l Conf. on Very Large Data Bases. Zurich, CH, 490-501.
  2. Albinali, F., Goodwin, M. S., and Intille, S. S. 2009. Recognizing stereotypical motor move- ments in the laboratory and classroom: a case study with children on the autism spectrum. In Proc. Int'l Conf. on Ubiquitous Computing. ACM, New York, NY, USA, 71-80.
  3. Amft, O. and Troester, G. 2006. Methods for detection and classification of normal swallowing from muscle activation and sound. Pervasive Health Conf. and Workshops, 1-10.
  4. Bao, L. and Intille, S. 2004. Activity recognition from user-annotated acceleration data. In Proc. Int'l Conf. on Pervasive Computing, LNCS 3001. LNCS, 1-17.
  5. Berndt, D. and Clifford, J. 1994. Using dynamic time warping to find patterns in time series. In Workshop on Knowledge Discovery in Databases (KDD), AAAI. Seattle, WA, 359-370.
  6. Buettner, M., Prasad, R., Philipose, M., and Wetherall, D. 2009. Recognizing daily activ- ities with rfid-based sensors. In Proc. 11th Int'l Conf. on Ubiquitous computing. ACM, New York, NY, USA, 51-60.
  7. Bulling, A., Ward, J. A., Gellersen, H.-W., and Tröster, G. 2008. Robust recognition of reading activity in transit using wearable electrooculography. In Proc. Int'l Conf. on Pervasive Computing. 19-37.
  8. Choujaa, D. and Dulay, N. 2008. Tracme: Temporal activity recognition using mobile phone data. In Proc. IEEE/IFIP Int'l Conf. on Embedded and Ubiquitous Computing Vol. 1. 119-126.
  9. Fogarty, J., Au, C., and Hudson, S. E. 2006. Sensing from the basement: a feasibility study of unobtrusive and low-cost home activity recognition. In 19th symp. on User interface software and technology. ACM, New York, NY, USA, 91-100.
  10. Hoover, A., Jean-Baptiste, G., Jiang, X., Flynn, P., Bunke, H., Goldof, D., Bowyer, K., Eggert, D., Fitzgibbon, A., and Fisher, R. 1996. An experimental comparison of range image segmentation algorithms. IEEE Trans. Pattern Analysis and Machine Intelligence 18(7), 673-689.
  11. Huynh, T., Blank, U., and Schiele, B. 2007. Scalable recognition of daily activities with wearable sensors. In Proc. Location-and Context-Awareness. LNCS, 50-67.
  12. Huynh, T., Fritz, M., and Schiele, B. 2008. Discovery of activity patterns using topic models. In Proc. Int'l Conf. on Ubiquitous Computing. ACM, New York, NY, USA, 10-19.
  13. Intille, S. S., Larson, K., Tapia, E. M., Beaudin, J., Kaushik, P., Nawyn, J., and Rockinson, R. 2006. Using a live-in laboratory for ubiquitous computing research. In Proc. Int'l Conf. on Pervasive Computing, LNCS. pp. 349-365.
  14. Kasturi, R., Goldgof, D., Soundararajan, P., Manohar, V., Garofolo, J., Bowers, R., Boonstra, M., Korzhova, V., and Zhang, J. 2009. Framework for performance evaluation of face, text, and vehicle detection and tracking in video: Data, metrics, and protocol. IEEE Trans. Pattern Analysis and Machine Intelligence 31, 2 (Feb.), 319-336.
  15. Lester, J., Choudhury, T., Kern, N., Borriello, G., and Hannaford, B. 2005. A hybrid discriminative/generative approach for modeling human activities. In Proc. Int'l Joint Conf. on Artificial Intelligence IJCAI. 766-772.
  16. Ling, C. X., Huang, J., and Zhang, H. 2003. Auc: a statistically consistent and more discrim- inating measure than accuracy. In Proc. 18th Int'l Conf. on Artificial Intelligence (IJCAI). 329-341.
  17. Logan, B., Healey, J., Philipose, M., Munguia-Tapia, E., and Intille, S. 2007. A longitudinal evaluation of sensing modalities for activity recognition. In Proc. Int'l Conf. on Ubiquitous Computing. LNCS, 483-500.
  18. Lukowicz, P., Ward, J., Junker, H., Tröster, G., A.Atrash, and T.Starner. 2004. Rec- ognizing workshop activity using body worn microphones and accelerometers. In Proc. Int'l Conf. on Pervasive Computing, LNCS. 18-32.
  19. Maekawa, T., Yanagisawa, Y., Kishino, Y., Ishiguro, K., Kamei, K., Sakurai, Y., and Okadome, T. 2010. Object-based activity recognition with heterogeneous sensors on wrist. Proc. Int'l Conf. on Pervasive Computing, 246-264.
  20. Minnen, D., Westeyn, T., Ashbrook, D., Presti, P., and Starner, T. 2007. Recognizing soldier activities in the field. In Body Sensor Networks. Vol. 13. 236-242.
  21. Patterson, D., Fox, D., Kautz, H., and Philipose, M. 2005. Fine-grained activity recognition by aggregating abstract object usage. In Proc. Int'l Symp. on Wearable Computers. IEEE, 44-51.
  22. Perng, C.-S., Wang, H., Zhang, S., and Parker, D. 2000. Landmarks: a new model for similarity-based pattern querying in time series databases. In Proc. Int'l Conf. on Data Engi- neering. IEEE, 33-42.
  23. Provost, F., Fawcett, T., and Kohavi, R. 1998. The case against accuracy estimation for comparing induction algorithms. In Proc. 15th Int'l Conf. on Machine Learning 445-453.
  24. Shafait, F., Keysers, D., and Breuel, T. 2008. Performance evaluation and benchmarking of six-page segmentation algorithms. IEEE Trans. Pattern Analysis and Machine Intelli- gence 30, 6, 941-954.
  25. Stiefmeier, T., Ogris, G., Junker, H., Lukowicz, P., and Troester, G. 2006. Combining mo- tion sensors and ultrasonic hands tracking for continuous activity recognition in a maintenance scenario. In Proc. Int'l Symp. on Wearable Computers. 97-104.
  26. Tapia, E., Intille, S., and Larson, K. 2004. Activity recognition in the home using simple and ubiquitous sensors. In Proc. Int'l Conf. on Pervasive Computing, LNCS 3001. 158-175.
  27. van Kasteren, T., Englebienne, G., and Kröse, B. 2010. Transferring knowledge of activity recognition across sensor networks. Proc. Int'l Conf. on Pervasive Computing , 283-300.
  28. van Rijsbergen, C. 1979. Information Retrieval, 2nd edition. Dept. of Computer Science, University of Glasgow.
  29. Ward, J., Lukowicz, P., Tröster, G., and Starner, T. 2006b. Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans. Pattern Analysis and Machine Intelligence 28, 10 (October), 1553-1567.
  30. Westeyn, T., Vadas, K., Bian, X., Starner, T., and Abowd, G. D. 2005. Recognizing mimicked autistic self-stimulatory behaviors using HMMs. In Proc. Int'l Symp. on Wearable Computers. 164-169.
  31. Zhang, Y. 2001. A review of recent evaluation methods for image segmentation. In Intl. Symp. on Signal Processing and its Applications. 148-151.
  32. Zinnen, A., Wojek, C., and Schiele, B. 2009. Multi activity recognition based on bodymodel- derived primitives. Proc. Location-and Context-Awareness, 1-18.

Authors
Ward, J. A., Lukowicz, P., & Gellersen, H. W.
Affiliations
  • Lancaster University (GB)
  • University of Passau (DE)
ORCID
Venue
ACM Transactions on Intelligent Systems and Technology
Publisher
Association for Computing Machinery (ACM)
Published
2011-01
Volume
2
Issue
1
Pages
1-23
ISSN
  • 2157-6904 (ISSN-L)
  • 2157-6912
DOI
10.1145/1889681.1889687
OpenAlex
W2060233269
Cited by
182
References
38
Citations by year
2026: 4 · 2025: 10 · 2024: 11 · 2023: 6 · 2022: 8 · 2021: 15 · 2020: 16 · 2019: 8 · 2018: 12 · 2017: 7 · 2016: 12 · 2015: 18 · 2014: 17 · 2013: 18 · 2012: 10
FWCI
18.61
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Transactions on Intelligent Systems and Technology (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • Lancaster EPrints (Lancaster University) (submittedVersion)
Funders
Engineering and Physical Sciences Research Council (EP/H005064/1)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
Keywords
performance indicators, timing offset, precision and recall

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review