ExtraSensory App: Data Collection In-the-Wild with Rich User Interface to Self-Report Behavior
The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
We introduce a mobile app for collecting in-the-wild data, including sensor measurements and self-reported labels describing people's behavioral context (e.g., driving, eating, in class, shower). Labeled data is necessary for developing context-recognition systems that serve health monitoring, aging care, and more. Acquiring labels without observers is challenging and previous solutions compromised ecological validity, range of behaviors, or amount of data. Our user interface combines past and near-future self-reporting of combinations of relevant context-labels. We deployed the app on the personal smartphones of 60 users and analyzed quantitative data collected in-the-wild and qualitative user-experience reports. The interface's flexibility was important to gain frequent, detailed labels, support diverse behavioral situations, and engage different users: most preferred reporting their past behavior through a daily journal, but some preferred reporting what they're about to do. We integrated insights from this work back into the app, which we make available to researchers for conducting in-the-wild studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although many commercial fitness trackers or smartwatches adopt wrist-mountable form due to the ease of accessibility, no consensus has been made regarding the positioning of sensors and their data acquisition details.”
From Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning (2019) · cites this paper as [16]
In the citing paper
“However, as the dataset adopts heterogeneous sensors on limited position of the human body, it is not sufficient to evaluate the recognition performance according to the sensor position and sampling rate.”
From Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning (2019) · cites this paper as [16]
Citations
Cites
A Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method
Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N., & Stone, A. A. (2004) · Science · doi:10.1126/science.1103572 · Cited by 3,422 · cited 4×
“DRM was used in [26] for eating detection, but the participants struggled remembering when they were eating.”
BackgroundMethodInfluentialFlow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi
Csikszentmihalyi, M. (2014) · doi:10.1007/978-94-017-9088-8 · Cited by 1,653
Activity sensing in the wild: a field trial of ubifit garden
Consolvo, S., McDonald, D. W., Toscos, T., Chen, M. Y., Froehlich, J., Harrison, B., Klasnja, P., LaMarca, A., LeGrand, L., Libby, R., et al. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357335 · Cited by 1,131
Mel Frequency Cepstral Coefficients For Music Modeling.
Logan, B. (2000) · doi:10.5281/zenodo.1416443 · Cited by 913
Experience Sampling: Promises and Pitfalls, Strengths and Weaknesses
Scollon, C. N., Kim-Prieto, C., & Diener, E. (2003) · Journal of Happiness Studies · doi:10.1023/a:1023605205115 · Cited by 825 · cited 6×
“Much like survey-based studies with ESM or DRM [23, 11], we address the multi-aspect nature of behavioral context and allow users to report combinations of activities, as well as location, company, body posture and more.”
BackgroundMethodInfluentialStisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjærgaard, M. B., Dey, A., Sonne, T., & Jensen, M. M. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809718 · Cited by 714 · cited 1×
“Data collection in-the-wild raises technical difficulties related to interruptions in sensor recording and diversity in phonedevices [24] and device placement [15].”
BackgroundDetecting activities of daily living in first-person camera views
Pirsiavash, H., & Ramanan, D. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248010 · Cited by 649 · cited 2×
“In some studies research assistants annotated the images [20, 7], compromising the privacy of the participants and their surrounding.”
BackgroundThe Mobile Sensing Platform: An Embedded Activity Recognition System
Choudhury, T., Borriello, G., Consolvo, S., Haehnel, D., Harrison, B., Hemingway, B., Hightower, J., Klasnja, P. "., Koscher, K., LaMarca, A., et al. (2008) · IEEE Pervasive Computing · doi:10.1109/mprv.2008.39 · Cited by 598
Validity and Reliability of the Experience-Sampling Method
Csikszentmihalyi, M., & Larson, R. (2014) · Flow and the Foundations of Positive Psychology · doi:10.1007/978-94-017-9088-8_3 · Cited by 578
King, A. C., Hekler, E. B., Grieco, L. A., Winter, S. J., Sheats, J. L., Buman, M. P., Banerjee, B., Robinson, T. N., & Cirimele, J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0062613 · Cited by 332 · cited 1×
“) is desirable for many domains, such as health management [10, 12], aging care [14, 16] and office assistant systems [30].”
BackgroundRecognizing Detailed Human Context in the Wild from Smartphones and Smartwatches
Vaizman, Y., Ellis, K., & Lanckriet, G. (2017) · IEEE Pervasive Computing · doi:10.1109/mprv.2017.3971131 · Cited by 324
Self-citationA practical approach for recognizing eating moments with wrist-mounted inertial sensing
Thomaz, E., Essa, I., & Abowd, G. D. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2807545 · Cited by 299 · cited 2×
“Previously suggested solutions involved unnatural equipment [20, 7, 25, 2] or simple self-reporting interfaces [8, 13] and resulted in data that had limited ecological validity and labels that describe behavior in a single-dimensional manner and cover a small portion of everyday life.”
BackgroundMy Phone and Me: Understanding People's Receptivity to Mobile Notifications
Mehrotra, A., Pejovic, V., Vermeulen, J., Hendley, R., & Musolesi, M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858566 · Cited by 261 · cited 1×
“[19] explored people’s receptivity to phone notifications.”
Feature Selection and Activity Recognition from Wearable Sensors
Pirttikangas, S., Fujinami, K., & Nakajima, T. (2006) · Lecture Notes in Computer Science · doi:10.1007/11890348_39 · Cited by 229 · cited 1×
“3174128 their systems with data collected in a lab [21, 3, 9] .”
BackgroundDetecting Periods of Eating During Free-Living by Tracking Wrist Motion
Dong, Y., Scisco, J., Wilson, M., Muth, E., & Hoover, A. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2282471 · Cited by 227 · cited 1×
“eating detection [6]) used a simple interface with a single button for the user to mark the start and stop times of eating.”
MethodActivity Recognition on Smartphones via Sensor-Fusion and KDA-Based SVMs
Khan, A. M., Tufail, A., Khattak, A. M., & Laine, T. H. (2014) · International Journal of Distributed Sensor Networks · doi:10.1155/2014/503291 · Cited by 150 · cited 3×
“Similar to [8, 13], our users can actively report that they are starting an activity.”
BackgroundVaizman, Y., Weibel, N., & Lanckriet, G. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161192 · Cited by 122
Self-citationSensor-based observations of daily living for aging in place
Lee, M. L., & Dey, A. K. (2014) · Personal and Ubiquitous Computing · doi:10.1007/s00779-014-0810-3 · Cited by 111 · cited 1×
“) is desirable for many domains, such as health management [10, 12], aging care [14, 16] and office assistant systems [30].”
BackgroundSensor Placement Variations in Wearable Activity Recognition
Kunze, K., & Lukowicz, P. (2014) · IEEE Pervasive Computing · doi:10.1109/mprv.2014.73 · Cited by 111 · cited 1×
“Data collection in-the-wild raises technical difficulties related to interruptions in sensor recording and diversity in phonedevices [24] and device placement [15].”
BackgroundThe power of mobile notifications to increase wellbeing logging behavior
Bentley, F., & Tollmar, K. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466140 · Cited by 99 · cited 1×
““how much did you eat today?”) every day [1], but for more detail, it can be hard for people to remember their daily events.”
BackgroundMulti-Sensor Fusion for Enhanced Contextual Awareness of Everyday Activities with Ubiquitous Devices
Guiry, J., Van de Ven, P., & Nelson, J. (2014) · Sensors · doi:10.3390/s140305687 · Cited by 91
Predicting daily activities from egocentric images using deep learning
Castro, D., Hickson, S., Bettadapura, V., Thomaz, E., Abowd, G., Christensen, H., & Essa, I. (2015) · Proceedings of the 2015 ACM International Symposium on Wearable Computers - ISWC '15 · doi:10.1145/2802083.2808398 · Cited by 86 · cited 1×
“Previously suggested solutions involved unnatural equipment [20, 7, 25, 2] or simple self-reporting interfaces [8, 13] and resulted in data that had limited ecological validity and labels that describe behavior in a single-dimensional manner and cover a small portion of everyday life.”
BackgroundObjective Assessment of Physical Activity: Classifiers for Public Health
KERR, J., PATTERSON, R. E., ELLIS, K., GODBOLE, S., JOHNSON, E., LANCKRIET, G., & STAUDENMAYER, J. (2016) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000000841 · Cited by 79 · cited 1×
“) is desirable for many domains, such as health management [10, 12], aging care [14, 16] and office assistant systems [30].”
BackgroundSelf-citationContext-aware office assistant
Yan, H., & Selker, T. (2000) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/325737.325872 · Cited by 77 · cited 1×
“) is desirable for many domains, such as health management [10, 12], aging care [14, 16] and office assistant systems [30].”
BackgroundInferring Meal Eating Activities in Real World Settings from Ambient Sounds: A Feasibility Study
Thomaz, E., Zhang, C., Essa, I. A., & Abowd, G. D. (2015) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2678025.2701405 · Cited by 65 · cited 1×
“DRM was used in [26] for eating detection, but the participants struggled remembering when they were eating.”
MethodMultisensor Fusion in Smartphones for Lifestyle Monitoring
Ganti, R. K., Srinivasan, S., & Gacic, A. (2010) · 2010 International Conference on Body Sensor Networks · doi:10.1109/bsn.2010.10 · Cited by 63 · cited 3×
“Similar to [8, 13], our users can actively report that they are starting an activity.”
BackgroundNeurotics Can't Focus: An in situ Study of Online Multitasking in the Workplace
Mark, G., Iqbal, S. T., Czerwinski, M., Johns, P., & Sano, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858202 · Cited by 63 · cited 1×
“Mark et al. [18] explored multitasking at work.”
BackgroundRealizing Effective Behavioral Management of Health: The Metamorphosis of Behavioral Science Methods
Hekler, E. B., Klasnja, P., Traver, V., & Hendriks, M. (2013) · IEEE Pulse · doi:10.1109/mpul.2013.2271681 · Cited by 44 · cited 1×
“) is desirable for many domains, such as health management [10, 12], aging care [14, 16] and office assistant systems [30].”
BackgroundMulti-sensor physical activity recognition in free-living
Ellis, K., Kerr, J., Godbole, S., & Lanckriet, G. (2014) · Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication · doi:10.1145/2638728.2641673 · Cited by 44 · cited 2×
“In some studies research assistants annotated the images [20, 7], compromising the privacy of the participants and their surrounding.”
BackgroundSelf-citationRahman, T., Zhang, M., Voida, S., & Choudhury, T. (2014) · International Conference on Pervasive Computing Technologies for Healthcare · doi:10.4108/icst.pervasivehealth.2014.254957 · Cited by 15 · cited 1×
“[22] dealt with self-assessment of stress-level and discussed the trade-off between in-situ reporting (more ecologically valid but disruptive and may cause stress) and recall-based reporting (non disruptive but introduces memory bias).”
BackgroundCrowdSignals: a call to crowdfund the community's largest mobile dataset
Welbourne, E., & Tapia, E. M. (2014) · Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication · doi:10.1145/2638728.2641309 · Cited by 15 · cited 1×
“The CrowdSignals project [29] aims to collect phone-sensor data from large crowds of users, with additional sparse probing for labels by using quick multiple-choice questions whenever the user unlocks their phone, combined with more in-depth (and less frequent) ESM questionnaires.”
BackgroundQuantitative Study of Music Listening Behavior in a Smartphone Context
Yang, Y., & Teng, Y.-C. (2015) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2738220 · Cited by 14 · cited 1×
“In [31], whenever the user selected a music playlist, she was prompted to report one out of 13 activities and one out of 10 moods.”
Background
Research works printed but not linked: 1
- 10.5555/2354409.2355089 · doi:10.5555/2354409.2355089
Cited by
Asynchronous Online Federated Learning for Edge Devices with Non-IID Data
Chen, Y., Ning, Y., Slawski, M., & Rangwala, H. (2020) · 2020 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata50022.2020.9378161 · Cited by 431
“• ExtraSensory Dataset 5 : Mobile device sensor data (e.g., high-frequency motion-reactive sensors, location services, audio, watch compass) and watch sensor data (accelerator) collected from 60 users; performing any of 51 activities [41].”
Chung, S., Lim, J., Noh, K. J., Kim, G., & Jeong, H. (2019) · Sensors · doi:10.3390/s19071716 · Cited by 215
Removal as a Method: A Fourth Wave HCI Approach to Understanding the Experience of Self-Tracking
Homewood, S., Karlsson, A., & Vallgårda, A. (2020) · Proceedings of the 2020 ACM Designing Interactive Systems Conference · doi:10.1145/3357236.3395425 · Cited by 87
Context-Aware Human Activity Recognition (CAHAR) in-the-Wild Using Smartphone Accelerometer
Asim, Y., Azam, M. A., Ehatisham-ul-Haq, M., Naeem, U., & Khalid, A. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.2964278 · Cited by 75
Advanced IoT-Based Human Activity Recognition and Localization Using Deep Polynomial Neural Network
Khan, D., Alshahrani, A. S., Almjally, A., Mudawi, N. A., Algarni, A. M., Alnowaiser, K., & Jalal, A. (2024) · IEEE Access · doi:10.1109/access.2024.3420752 · Cited by 74
Metadata
- Authors
- Vaizman, Y., Ellis, K., Lanckriet, G., & Weibel, N.
- Affiliations
- University of California San Diego (US)
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-12
- OpenAlex
- W2795423566
- Cited by
- 115
- References
- 31
- Citations by year
- 2026: 7 · 2025: 11 · 2024: 12 · 2023: 9 · 2022: 9 · 2021: 25 · 2020: 18 · 2019: 17 · 2018: 7
- FWCI
- 5.05
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- mobile application, self-report, context recognition, smartphone sensors, user interface design, user experience, health monitoring, elderly care, ecological validity, daily journal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.