The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning
Open accessAbstract
Microphones are remarkably powerful sensors of human behavior and context. However, audio sensing is highly susceptible to wild fluctuations in accuracy when used in diverse acoustic environments (such as, bedrooms, vehicles, or cafes), that users encounter on a daily basis. Towards addressing this challenge, we turn to the field of deep learning; an area of machine learning that has radically changed related audio modeling domains like speech recognition. In this paper, we present DeepEar -- the first mobile audio sensing framework built from coupled Deep Neural Networks (DNNs) that simultaneously perform common audio sensing tasks. We train DeepEar with a large-scale dataset including unlabeled data from 168 place visits. The resulting learned model, involving 2.3M parameters, enables DeepEar to significantly increase inference robustness to background noise beyond conventional approaches present in mobile devices. Finally, we show DeepEar is feasible for smartphones by building a cloud-free DSP-based prototype that runs continuously, using only 6% of the smartphone's battery daily.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
(2010) · International Conference on Machine Learning · Cited by 13,995
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
(2006) · Springer eBooks · Cited by 8,123
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“We train these parameters using a Gibbs sampler applying the contrastive divergence approximation [69] of the gradient during activation function parameter search.”
MethodDeep Learning: Methods and Applications
Deng, L., & Yu, D. (2014) · Foundations and Trends® in Signal Processing · doi:10.1561/2000000039 · Cited by 3,480
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · cited 2×
“(Although deep models are also trained at times with just 20 or 30 hours of data, such as in [23]).”
BackgroundRobust text-independent speaker identification using Gaussian mixture speaker models
Reynolds, D. A., & Rose, R. C. (1995) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.365379 · Cited by 2,860
A Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852
Musical genre classification of audio signals
Tzanetakis, G., & Cook, P. (2002) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2002.800560 · Cited by 2,798
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Samy Bengio, A. (2010) · Journal of Machine Learning Research · Cited by 2,631
Perceptual linear predictive (PLP) analysis of speech
Hermansky, H. (1990) · The Journal of the Acoustical Society of America · doi:10.1121/1.399423 · Cited by 2,536
Deep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · IEEE Signal Processing Magazine · Cited by 2,115
Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains
Gauvain, J.-L., & Lee, C.-H. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.279278 · Cited by 2,108
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,861
Deep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,759
Maximum likelihood linear transformations for HMM-based speech recognition
Gales, M. J. F. (1998) · Computer Speech & Language · doi:10.1006/csla.1998.0043 · Cited by 1,482
Improving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310
Nericell: rich monitoring of road and traffic conditions using mobile smartphones
Mohan, P., Padmanabhan, V. N., & Ramjee, R. (2008) · Proceedings of the 6th ACM conference on Embedded network sensor systems · doi:10.1145/1460412.1460444 · Cited by 1,267
The Tradeoffs of Large-Scale Learning
Bottou, L., & Bousquet, O. (2011) · Optimization for Machine Learning · doi:10.7551/mitpress/8996.003.0015 · Cited by 1,128
Deep neural networks for small footprint text-dependent speaker verification
Variani, E., Lei, X., McDermott, E., Moreno, I. L., & Gonzalez-Dominguez, J. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854363 · Cited by 1,035
An empirical evaluation of deep architectures on problems with many factors of variation
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273556 · Cited by 992
(2007) · Cited by 937
Analysis of i-vector length normalization in speaker recognition systems
Garcia-Romero, D., & Espy-Wilson, C. Y. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-53 · Cited by 922
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794
An investigation of deep neural networks for noise robust speech recognition
Seltzer, M. L., Yu, D., & Wang, Y. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639100 · Cited by 640
Comparison of different implementations of MFCC
Zheng, F., Zhang, G., & Song, Z. (2001) · Journal of Computer Science and Technology · doi:10.1007/bf02943243 · Cited by 591
SoundSense: scalable sound sensing for people-centric applications on mobile phones
Lu, H., Pan, W., Lane, N. D., Choudhury, T., & Campbell, A. T. (2009) · Proceedings of the 7th international conference on Mobile systems, applications, and services · doi:10.1145/1555816.1555834 · Cited by 581
Self-citationAn Overview of Noise-Robust Automatic Speech Recognition
Li, J., Deng, L., Gong, Y., & Haeb-Umbach, R. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2304637 · Cited by 568
The Jigsaw continuous sensing engine for mobile phone applications
Lu, H., Yang, J., Liu, Z., Lane, N. D., Choudhury, T., & Campbell, A. T. (2010) · Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/1869983.1869992 · Cited by 560
Self-citationSmall-footprint keyword spotting using deep neural networks
Chen, G., Parada, C., & Heigold, G. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854370 · Cited by 545
StressSense: detecting stress in unconstrained acoustic environments using smartphones
Lu, H., Frauendorfer, D., Rabbi, M., Mast, M. S., Chittaranjan, G. T., Campbell, A. T., Gatica-Perez, D., & Choudhury, T. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370270 · Cited by 511
EmotionSense: a mobile phones based adaptive platform for experimental social psychology research
Rachuri, K. K., Musolesi, M., Mascolo, C., Rentfrow, P. J., Longworth, C., & Aucinas, A. (2010) · Proceedings of the 12th ACM international conference on Ubiquitous computing · doi:10.1145/1864349.1864393 · Cited by 508
Exponential Family Harmoniums with an Application to Information Retrieval
Welling, M., Rosen‐Zvi, M., & Hinton, G. E. (2004) · Cited by 423
Can Deep Learning Revolutionize Mobile Sensing?
Lane, N. D., & Georgiev, P. (2015) · Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications · doi:10.1145/2699343.2699349 · Cited by 282 · cited 2×
“Deep learning has the potential to broadly impact the fields of activity recognition and the modeling of user behavior and context; early explorations of this potential are already underway [47, 35].”
BackgroundResultSelf-citationNericell: using mobile smartphones for rich monitoring of road and traffic conditions
Mohan, P., Padmanabhan, V. N., & Ramjee, R. (2008) · Proceedings of the 6th ACM conference on Embedded network sensor systems · doi:10.1145/1460412.1460450 · Cited by 245
Passive and In-Situ assessment of mental and physical well-being using mobile sensors
Rabbi, M., Ali, S., Choudhury, T., & Berke, E. (2011) · Proceedings of the 13th international conference on Ubiquitous computing · doi:10.1145/2030112.2030164 · Cited by 243
Accurate and privacy preserving cough sensing using a low-cost microphone
Larson, E. C., Lee, T., Liu, S., Rosenfeld, M., & Patel, S. N. (2011) · Proceedings of the 13th international conference on Ubiquitous computing · doi:10.1145/2030112.2030163 · Cited by 217
Analysis of Chewing Sounds for Dietary Monitoring
Amft, O., Stäger, M., Lukowicz, P., & Tröster, G. (2005) · Lecture Notes in Computer Science · doi:10.1007/11551201_4 · Cited by 216
Darwin phones: the evolution of sensing and inference on mobile phones
Miluzzo, E., Cornelius, C. T., Ramaswamy, A., Choudhury, T., Liu, Z., & Campbell, A. T. (2010) · Proceedings of the 8th international conference on Mobile systems, applications, and services · doi:10.1145/1814433.1814437 · Cited by 200
PD Disease State Assessment in Naturalistic Environments Using Deep Learning
Hammerla, N., Fisher, J., Andras, P., Rochester, L., Walker, R., & Ploetz, T. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9484 · Cited by 179 · cited 1×
“Deep learning has the potential to broadly impact the fields of activity recognition and the modeling of user behavior and context; early explorations of this potential are already underway [47, 35].”
BackgroundSubspace Gaussian Mixture Models for speech recognition
Povey, D., Burget, L., Agarwal, M., Akyazi, P., Feng, K., Ghoshal, A., Glembek, O., Goel, N. K., Karafiat, M., Rastrow, A., et al. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5495662 · Cited by 174
Lane, N. D., Xu, Y., Lu, H., Hu, S., Choudhury, T., Campbell, A. T., & Zhao, F. (2011) · Proceedings of the 13th international conference on Ubiquitous computing · doi:10.1145/2030112.2030160 · Cited by 171
Self-citationFeature learning and deep architectures: new directions for music informatics
Humphrey, E. J., Bello, J. P., & LeCun, Y. (2013) · Journal of Intelligent Information Systems · doi:10.1007/s10844-013-0248-5 · Cited by 171
BodyBeat: a mobile system for sensing non-speech body sounds
Rahman, T., Adams, A. T., Zhang, M., Cherry, E., Zhou, B., Peng, H., & Choudhury, T. (2014) · Proceedings of the 12th annual international conference on Mobile systems, applications, and services · doi:10.1145/2594368.2594386 · Cited by 159
SpeakerSense: Energy Efficient Unobtrusive Speaker Identification on Mobile Phones
Lu, H., Brush, A. J. B., Priyantha, N. B., Karlson, A. K., & Liu, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21726-5_12 · Cited by 155
Understanding the coverage and scalability of place-centric crowdsensing
Chon, Y., Lane, N. D., Kim, Y., Zhao, F., & Cha, H. (2013) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2493432.2493498 · Cited by 135
Self-citationCrowd++: unsupervised speaker count with smartphones
Xu, C., Li, S., Liu, G., Zhang, Y., Miluzzo, E., Chen, Y.-F. R., Li, J., & Firner, B. (2013) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2493432.2493435 · Cited by 124
A deep learning approach to machine transliteration
Thomas Deselaers, B., Hasan, S., Bender, O., & Ney, H. (2009) · Conference on Machine Translation (WMT) · doi:10.3115/1626431.1626476 · Cited by 120
Lee, Y., Min, C., Hwang, C., Lee, J., Hwang, I., Ju, Y., Yoo, C., Moon, M., Lee, U., & Song, J. (2013) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2462456.2465426 · Cited by 114
An auto-encoder based approach to unsupervised learning of subword units
Badino, L., Canevari, C., Fadiga, L., et al. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6855085 · Cited by 77
DSP.Ear: leveraging co-processor support for continuous audio sensing on smartphones
Georgiev, P., Lane, N. D., Rachuri, K. K., & Mascolo, C. (2014) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/2668332.2668349 · Cited by 60
Self-citationPocket, Bag, Hand, etc. - Automatically Detecting Phone Context through Discovery
Miluzzo, E., Papandrea, M., Lane, N. D., Lu, H., & Campbell, A. T. (2010) · doi:10.71910/supsi.7225 · Cited by 54
Self-citationCommunity-Guided Learning: Exploiting Mobile Sensor Users to Model Human Behavior
Peebles, D., Lu, H., Lane, N. D., Choudhury, T., & Campbell, A. T. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7731 · Cited by 49
Self-citationTan, W.-T., Baker, M. G., Lee, B., & Samadani, R. (2013) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/2517351.2517362 · Cited by 27
Huang, Y., Slaney, M., Seltzer, M. L., & Gong, Y. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-214 · Cited by 26
Local business ambience characterization through mobile audio sensing
Wang, H., Lymberopoulos, D., & Liu, J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568027 · Cited by 22
Helping mobile apps bootstrap with fewer users
Bao, X., Bahl, P., Kansal, A., Chu, D. C. Y., Choudhury, R. R., & Wolman, A. (2012) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2370216.2370289 · Cited by 18
Park, T. H., Turner, J., Musick, M., Lee, J. H., Jacoby, C., Mydlarz, C., & Salamon, J. (2014) · EDBT/ICDT Workshops · Cited by 18
Integrating Speaker Identification and Learning with Adaptive Speech Recognition
Fink, G. A., & Plötz, T. (2004) · Cited by 6
Cited by
Deep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014
“Although some existing work adopted deep HAR on smartphone (Lane et al., 2015) and watch (Bhattacharya and Lane, 2016), they are still far from online and mobile deployment.”
BackgroundMethodInfluentialLearning IoT in Edge: Deep Learning for the Internet of Things with Edge Computing
Li, H., Ota, K., & Dong, M. (2018) · IEEE Network · doi:10.1109/mnet.2018.1700202 · Cited by 1,566
Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing
Li, E., Zeng, L., Zhou, Z., & Chen, X. (2020) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2019.2946140 · Cited by 929
DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Yao, S., Hu, S., Zhao, Y., Zhang, A., & Abdelzaher, T. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052577 · Cited by 667
“DeepEar [22] uses Deep Boltzmann Machines to improve the performance of audio sensing tasks in an environment with background noise.”
MethodDeep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606
Metadata
- Authors
- Lane, N. D., Georgiev, P., & Qendro, L.
- Venue
- Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing
- Publisher
- ACM
- Published
- 2015-09-07
- Pages
- 283-294
- OpenAlex
- W1997430507
- Cited by
- 310
- References
- 65
- FWCI
- 33.41
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.