The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning
Open accessAbstract
Microphones are remarkably powerful sensors of human behavior and context. However, audio sensing is highly susceptible to wild fluctuations in accuracy when used in diverse acoustic environments (such as, bedrooms, vehicles, or cafes), that users encounter on a daily basis. Towards addressing this challenge, we turn to the field of deep learning; an area of machine learning that has radically changed related audio modeling domains like speech recognition. In this paper, we present DeepEar -- the first mobile audio sensing framework built from coupled Deep Neural Networks (DNNs) that simultaneously perform common audio sensing tasks. We train DeepEar with a large-scale dataset including unlabeled data from 168 place visits. The resulting learned model, involving 2.3M parameters, enables DeepEar to significantly increase inference robustness to background noise beyond conventional approaches present in mobile devices. Finally, we show DeepEar is feasible for smartphones by building a cloud-free DSP-based prototype that runs continuously, using only 6% of the smartphone's battery daily.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
(2006) · Springer eBooks · Cited by 8,123
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“We train these parameters using a Gibbs sampler applying the contrastive divergence approximation [69] of the gradient during activation function parameter search.”
MethodDeep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480
Cited by
Deep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014
“Although some existing work adopted deep HAR on smartphone (Lane et al., 2015) and watch (Bhattacharya and Lane, 2016), they are still far from online and mobile deployment.”
BackgroundMethodInfluentialLearning IoT in Edge: Deep Learning for the Internet of Things with Edge Computing
Li, H., Ota, K., & Dong, M. (2018) · IEEE Network · doi:10.1109/mnet.2018.1700202 · Cited by 1,566
“[10] proposed new acceleration engines, such as DeepEar and DeepX, to support different deep learning applications in latest mobile system-on-chips (SoC).”
MethodEdge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing
Li, E., Zeng, L., Zhou, Z., & Chen, X. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2946140 · Cited by 929
DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Yao, S., Hu, S., Zhao, Y., Zhang, A., & Abdelzaher, T. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052577 · Cited by 667
“DeepEar [22] uses Deep Boltzmann Machines to improve the performance of audio sensing tasks in an environment with background noise.”
MethodDeep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606
Learning-Based Computation Offloading for IoT Devices With Energy Harvesting
Min, M., Xiao, L., Chen, Y., Cheng, P., Wu, D., & Zhuang, W. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2018.2890685 · Cited by 560
Ensembles of Deep LSTM Learners for Activity Recognition using Wearables
Guan, Y., & Plötz, T. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3090076 · Cited by 511
DeepX: A Software Accelerator for Low-Power Deep Learning Inference on Mobile Devices
Lane, N. D., Bhattacharya, S., Georgiev, P., Forlivesi, C., Jiao, L., Qendro, L., & Kawsar, F. (2016) · 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn.2016.7460664 · Cited by 450
Self-citationEfficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review
Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3226481 · Cited by 436
“DeepEar [397] is DL-based audio sensing using a smartphone that used stacked RBM.”
MethodHan, S., Shen, H., Philipose, M., Agarwal, S., Wolman, A., & Krishnamurthy, A. (2016) · Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2906388.2906396 · Cited by 367
DeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications
Huynh, L. N., Lee, Y., & Balan, R. K. (2017) · Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3081333.3081360 · Cited by 329
“took important first steps towards the real-time execution of DNN and CNN on mobile devices [35, 36].”
BackgroundMethodMultimodal Deep Learning for Activity and Context Recognition
Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161174 · Cited by 281
“But with the diversity of exploration rapidly expanding [3, 7, 16, 17, 25, 31, 40, 49, 71].”
BackgroundSelf-citationA Survey on Deep Learning for Human Activity Recognition
Gu, F., Chung, M.-H., Chignell, M., Valaee, S., Zhou, B., & Liu, X. (2022) · ACM Computing Surveys · doi:10.1145/3472290 · Cited by 272
Edge Intelligence: Empowering Intelligence to the Edge of Network
Xu, D., Li, T., Li, Y., Su, X., Tarkoma, S., Jiang, T., Crowcroft, J., & Hui, P. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3119950 · Cited by 272
A review of state-of-the-art techniques for abnormal human activity recognition
Dhiman, C., & Vishwakarma, D. K. (2019) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2018.08.014 · Cited by 265
Bhattacharya, S., & Lane, N. D. (2016) · Proceedings of the 14th ACM Conference on Embedded Networked Sensor Systems CD-ROM · doi:10.1145/2994551.2994564 · Cited by 236
Self-citationOn-Demand Deep Model Compression for Mobile Devices: A Usage-Driven Model Selection Framework
Liu, S., Lin, Y., Zhou, Z., Nan, K., Liu, H., & Du, J. (2018) · Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3210240.3210337 · Cited by 217
Yao, S., Zhao, Y., Zhang, A., Su, L., & Abdelzaher, T. (2017) · Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems · doi:10.1145/3131672.3131675 · Cited by 188
JALAD: Joint Accuracy-And Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution
Li, H., Hu, C., Jiang, J., Wang, Z., Wen, Y., & Zhu, W. (2018) · 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/padsw.2018.8645013 · Cited by 160
Deep Learning for RFID-Based Activity Recognition
Li, X., Zhang, Y., Marsic, I., Sarcevic, A., & Burd, R. S. (2016) · Proceedings of the 14th ACM Conference on Embedded Networked Sensor Systems CD-ROM · doi:10.1145/2994551.2994569 · Cited by 150
DeepWear: Adaptive Local Offloading for On-Wearable Deep Learning
Xu, M., Qian, F., Zhu, M., Huang, F., Pushp, S., & Liu, X. (2020) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2019.2893250 · Cited by 138
“Making deep learning (DL for short in the rest of this paper) tasks run on mobile devices has raised huge interests in both the academia [1], [2], [3], [4], [5], [6], [7], [8] and the industry [9], [10], [11].”
BackgroundDeep Learning for Mobile Multimedia: A Survey
Ota, K., Dao, M. S., Mezaris, V., & Natale, F. G. B. D. (2017) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3092831 · Cited by 135
Meta-HAR: Federated Representation Learning for Human Activity Recognition
Li, C., Niu, D., Jiang, B., Zuo, X., & Yang, J. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3450006 · Cited by 133
“Deep Boltzmann Machine is adopted in DeepEar [14] to improve the performance of audio sensing tasks in an environment with background noise.”
MethodMachine Learning in IoT Security: Current Solutions and Future Challenges
Hussain, F., Hussain, R., Hassan, S. A., & Hossain, E. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986444 · Cited by 129
FedHAR: Semi-Supervised Online Learning for Personalized Federated Human Activity Recognition
Yu, H., Chen, Z., Zhang, X., Chen, X., Zhuang, F., Xiong, H., & Cheng, X. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2021.3136853 · Cited by 113
Suh, S., Rey, V. F., & Lukowicz, P. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110143 · Cited by 107
FoggyCache: Cross-Device Approximate Computation Reuse
Guo, P., Hu, B., Li, R., & Hu, W. (2018) · Proceedings of the 24th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3241539.3241557 · Cited by 91
DeepCham: Collaborative Edge-Mediated Adaptive Deep Learning for Mobile Object Recognition
Li, D., Salonidis, T., Desai, N. V., & Chuah, M. C. (2016) · 2016 IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec.2016.38 · Cited by 84
Guo, H., Chen, L., Peng, L., & Chen, G. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2971648.2971708 · Cited by 83
“Various types of deep networks have been successfully applied in behavior modeling, such as, RBM for skill assessment [16], CNN for activity recognition [17], and DNN for audio sensing [37].”
MethodCloud Versus Edge Deployment Strategies of Real-Time Face Recognition Inference
Koubaa, A., Ammar, A., Kanhouch, A., & AlHabashi, Y. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2021.3055835 · Cited by 69
Mathur, A., Zhang, T., Bhattacharya, S., Velickovic, P., Joffe, L., Lane, N. D., Kawsar, F., & Lio, P. (2018) · 2018 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn.2018.00048 · Cited by 70
Self-citationGeorgiev, P., Bhattacharya, S., Lane, N. D., & Mascolo, C. (2017) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3131895 · Cited by 67
“This topology has proven effective in a variety of audio analysis tasks such as keyword spotting [14], text-dependent speaker verification [53], and emotion recognition [32].”
BackgroundMethodInfluentialiHear Food: Eating Detection Using Commodity Bluetooth Headsets
Gao, Y., Zhang, N., Wang, H., Ding, X., Ye, X., Chen, G., & Cao, Y. (2016) · 2016 IEEE First International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE) · doi:10.1109/chase.2016.14 · Cited by 63
SenseGAN: Enabling Deep Learning for Internet of Things with a Semi-Supervised Framework
Yao, S., Zhao, Y., Shao, H., Zhang, C., Zhang, A., Hu, S., Liu, D., Liu, S., Su, L., & Abdelzaher, T. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264954 · Cited by 54
Yao, S., Zhao, Y., Shao, H., Liu, D., Liu, S., Hao, Y., Piao, A., Hu, S., Lu, S., & Abdelzaher, T. F. (2019) · IEEE INFOCOM 2019 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2019.8737500 · Cited by 52
Toma, C., Popa, M., Iancu, B., Doinea, M., Pascu, A., & Ioan-Dutescu, F. (2022) · Electronics · doi:10.3390/electronics11213507 · Cited by 22
The benefit of the doubt: Uncertainty aware sensing for edge computing platforms
Qendro, L., Chauhan, J., Ramos, A. G. C. P., & Mascolo, C. (2021) · arXiv · doi:10.48550/arxiv.2102.05956 · Cited by 6
Self-citation
Metadata
- Authors
- Lane, N. D., Georgiev, P., & Qendro, L.
- Venue
- Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing
- Publisher
- ACM
- Published
- 2015-09-07
- Pages
- 283-294
- OpenAlex
- W1997430507
- Cited by
- 310
- References
- 65
- FWCI
- 33.41
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.