Automotive big data: Applications, workloads and infrastructures
Abstract
Data is increasingly affecting the automotive industry, from vehicle development, to manufacturing and service processes, to online services centered around the connected vehicle. Connected, mobile and Internet of Things devices and machines generate immense amounts of sensor data. The ability to process and analyze this data to extract insights and knowledge that enable intelligent services, new ways to understand business problems, improvements of processes and decisions, is a critical capability. Hadoop is a scalable platform for compute and storage and emerged as de-facto standard for Big Data processing at Internet companies and in the scientific community. However, there is a lack of understanding of how and for what use cases these new Hadoop capabilities can be efficiently used to augment automotive applications and systems. This paper surveys use cases and applications for deploying Hadoop in the automotive industry. Over the years a rich ecosystem emerged around Hadoop comprising tools for parallel, in-memory and stream processing (most notable MapReduce and Spark), SQL and NOSQL engines (Hive, HBase), and machine learning (Mahout, MLlib). It is critical to develop an understanding of automotive applications and their characteristics and requirements for data discovery, integration, exploration and analytics. We then map these requirements to a confined technical architecture consisting of core Hadoop services and libraries for data ingest, processing and analytics. The objective of this paper is to address questions, such as: What applications and datasets are suitable for Hadoop? How can a diverse set of frameworks and tools be managed on multi-tenant Hadoop cluster? How do these tools integrate with existing relational data management systems? How can enterprise security requirements be addressed? What are the performance characteristics of these tools for real-world automotive applications? To address the last question, we utilize a standard benchmark (TPCx-HS), and two application benchmarks (SQL and machine learning) that operate on a dataset of multiple Terabytes and billions of rows.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ETSI is actively pursuing TI standardization through a work item on IPv6-based TI.
“Finally, the European Telecommunications Standards Institute (ETSI) is actively pursuing TI standardization through a work item on IPv6-based TI [9].”
From The IEEE 1918.1 “Tactile Internet” Standards Working Group and its Standards (2019) · cites this paper as [9]
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476 · see citing works
(2016) · Cited by 11,679 · see citing works
The NumPy Array: A Structure for Efficient Numerical Computation
van der Walt, S., Colbert, S. C., & Varoquaux, G. P. (2011) · Computing in Science & Engineering · doi:10.1109/mcse.2011.37 · Cited by 10,445 · see citing works
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622 · see citing works
MLlib: Machine Learning in Apache SparkPreprint
Meng, X., Bradley, J. K., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Freeman, J., Tsai, D., Amde, M., Owen, S., et al. (2015) · Journal of machine learning research · doi:10.48550/arxiv.1505.06807 · Cited by 1,898 · see citing works
Apache Hadoop YARN: yet another resource negotiator
Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., et al. (2013) · Proceedings of the 4th annual Symposium on Cloud Computing · doi:10.1145/2523616.2523633 · Cited by 1,848 · see citing works
Thusoo, A., Sarma, J. S., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P., & Murthy, R. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687609 · Cited by 1,559 · see citing works
Armbrust, M., Xin, R., Lian, C., Huai, Y., Liu, D., Bradley, J. K., Meng, X., Kaftan, T., Franklin, M. J. M., Ghodsi, A., et al. (2015) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2723372.2742797 · Cited by 1,254 · see citing works
Big Data: Principles and best practices of scalable realtime data systems
Marz, N., & Warren, J. (2015) · Cited by 1,019 · see citing works
Melnik, S., Gubarev, A., Long, J. J., Romer, G., Shivakumar, S., Tolton, M., & Vassilakis, T. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1920841.1920886 · Cited by 713 · see citing works
The Stratosphere platform for big data analytics
Alexandrov, A., Bergmann, R., Ewen, S., Freytag, J.-C., Hueske, F., Heise, A., Kao, O., Leich, M., Leser, U., Markl, V., et al. (2014) · The VLDB Journal · doi:10.1007/s00778-014-0357-y · Cited by 434 · see citing works
Hellerstein, J. M., Ré, C., Schoppmann, F., Wang, D. Z., Fratkin, E., Gorajek, A., Ng, K. S., Welton, C., Feng, X., Li, K., et al. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367510 · Cited by 382 · see citing works
Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation
Feamster, N., & Mogul, J. (2013) · Cited by 348 · see citing works
Impala: A Modern, Open-Source SQL Engine for Hadoop.
Kornacker, M., Behm, A., Bittorf, V., Bobrovytsky, T., Ching, C., Choi, A., Erickson, J., Grund, M., Hecht, D., Jacobs, M., et al. (2015) · Cited by 322 · see citing works
Soliman, M. A., Antova, L., Raghavan, V., et al. (2014) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/2588555.2595637 · Cited by 74 · see citing works
A Tale of Two Data-Intensive Paradigms: Applications, Abstractions, and Architectures
Jha, S., Qiu, J., Luckow, A., et al. (2014) · IEEE International Congress on Big Data (BigData Congress) · doi:10.1109/bigdata.congress.2014.137 · Cited by 74 · see citing works
Self-citationTraffic Signal Phase and Timing Estimation From Low-Frequency Transit Bus Data
Fayazi, S. A., Vahidi, A., Mahler, G., et al. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2323341 · Cited by 68 · see citing works
Synthetic data generation for the internet of things
Anderson, J. W., Kennedy, K., Ngo, L. B., et al. (2014) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2014.7004228 · Cited by 65 · see citing works
Self-citationDiscussion of BigBench: A Proposed Industry Standard Performance Benchmark for Big Data
Baru, C. K., Bhandarkar, M. A., Curino, C., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-15350-6_4 · Cited by 35 · see citing works
Introducing TPCx-HS: The First Industry Standard for Benchmarking Big Data Systems
Nambiar, R. O., Poess, M., Dey, A., et al. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-15350-6_1 · Cited by 35 · see citing works
Harp: Collective Communication on Hadoop
Zhang, B., Ruan, Y., & Qiu, J. (2015) · IEEE International Conference on Cloud Engineering · doi:10.1109/ic2e.2015.35 · Cited by 24 · see citing works
Profiling R on a contemporary processor
Sridharan, S., & Patel, J. M. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2735471.2735478 · Cited by 14 · see citing works
Towards a Comprehensive Set of Big Data Benchmarks
Fox, G., Shantenu, J., Qiu, J., et al. (2015) · Advances in parallel computing · doi:10.3233/978-1-61499-583-8-47 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Edge Machine Learning for AI-Enabled IoT Devices: A Review
Merenda, M., Porcaro, C., & Iero, D. (2020) · Sensors · doi:10.3390/s20092533 · Cited by 523 · see citing works
The IEEE 1918.1 “Tactile Internet” Standards Working Group and its Standards
Holland, O., Steinbach, E., Prasad, R. V., Liu, Q., Dawy, Z., Aijaz, A., Pappas, N., Chandra, K., Rao, V. S., Oteafy, S., et al. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2018.2885541 · Cited by 200 · see citing works
“Furthermore, the onboard sensors in today self-driving cars generate data flows up to 1 GB/s [35], [36].”
BackgroundDeep learning in the automotive industry: Applications and tools
Luckow, A., Cook, M., Ashcraft, N., Weill, E., Djerekarov, E., & Vorster, B. (2016) · 2016 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2016.7841045 · Cited by 164 · see citing works
Self-citationIn Search of Big Medical Data Integration Solutions - A Comprehensive Survey
Dhayne, H., Haque, R., Kilany, R., & Taher, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2927491 · Cited by 84 · see citing works
- Authors
- Luckow, A., Kennedy, K., Manhardt, F., Djerekarov, E., Vorster, B., & Apon, A.
- Affiliations
- Clemson University (US)
- Venue
- 2015 IEEE International Conference on Big Data (Big Data)
- Publisher
- IEEE
- Published
- 2015-10
- Pages
- 1201-1210
- OpenAlex
- W2202064371
- Cited by
- 92
- References
- 26
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 10 · 2023: 7 · 2022: 6 · 2021: 9 · 2020: 11 · 2019: 15 · 2018: 10 · 2017: 7 · 2016: 6
- FWCI
- 6.66
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Big Data (BigData) (publishedVersion)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › Advanced Database Systems and Queries
- Keywords
- Hadoop, connected vehicles, sensor data, MapReduce, Apache Spark, HBase, Mahout, data integration, enterprise security, performance benchmarking, machine learning benchmark, Internet of Things
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
