Machine learning with big data: challenges and approaches.
Open accessThe DOI matches its Crossref record; 108 of the 109 references it cites were found in a registry or an index.
Abstract
The Big Data revolution promises to transform how we live, work, and think by enabling process optimization, empowering insight discovery and improving decision making. The realization of this grand potential relies on the ability to extract value from such massive data through data analytics; machine learning is at its core because of its ability to learn from data and provide data driven insights, decisions, and predictions. However, traditional machine learning approaches were developed in a different era, and thus are based upon multiple assumptions, such as the data set fitting entirely into memory, what unfortunately no longer holds true in this new context. These broken assumptions, together with the Big Data characteristics, are creating obstacles for the traditional techniques. Consequently, this paper compiles, summarizes, and organizes machine learning challenges with Big Data. In contrast to other research that discusses challenges, this work highlights the cause–effect relationship by organizing challenges according to Big Data Vs or dimensions that instigated the issue: volume, velocity, variety, or veracity. Moreover, emerging machine learning approaches and techniques are discussed in terms of how they are capable of handling the various challenges with the ultimate objective of helping practitioners select appropriate solutions for their use cases. Finally, a matrix relating the challenges and approaches is presented. Through this process, this paper provides a perspective on the domain, identifies research gaps and opportunities, and provides a strong foundation and encouragement for further research in the field of machine learning with Big Data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, training these models requires large, diverse, and well-labeled datasets [94].”
From A Review of Passenger Counting in Public Transport Concepts with Solution Proposal Based on Image Processing and Machine Learning (2024) · cites this paper as [94]
In the citing paper
“it is also shifting from a single domain (e.g., domain expertise, or expert systems) to a cross-disciplinary approach through collaboration”
From Challenges and Future Directions of Big Data and Artificial Intelligence in Education (2020) · cites this paper as L’heureux et al., 2017
Citations
Cites
Data Mining: Practical Machine Learning Tools and Techniques
(2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,437
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · cited 1×
“• Regularization paths [92] for linear models supports linear regression, two-class logistic regression, and multinomial regression problems.”
MethodThe WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 1×
“However, at the moment, they support a limited number of algorithms compared to the R language [73], MATLAB [74], or Weka [75].”
MethodRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 1×
“Representation learning is also often referred to as feature learning [100].”
BackgroundEnsemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
(2011) · Cited by 5,203
Multiple comparisons among means
Dunn, O. J. (1961) · Journal of the American Statistical Association · doi:10.1080/01621459.1961.10482090 · Cited by 5,006 · cited 1×
“It suggests that if testing m hypotheses with a desired significance of α, each individual hypothesis should be tested at a significance level of α/m [43].”
BackgroundResilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · UC Berkeley · Cited by 4,891
The Hadoop Distributed File System
Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010) · 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) · doi:10.1109/msst.2010.5496972 · Cited by 4,834 · cited 8×
“In these stages, independent of the category of manipulations, processes can be embedded to capture data provenance and therefore remedy provenance challenge; an example is the Reduce and Map Provenance (RAMP) developed for MapReduce as an extension for Hadoop [64].”
MethodInfluentialHan, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662 · cited 1×
“Deep compression has been proposed as a way of speeding up processing without the loss of accuracy [105].”
Background
Research works printed but not linked: 20
- goodfellow, 2016, Deep Learning
- marz, 2014, Apache Storm
- rouse, 2011, Machine Learning Definition
- 2015, R A Language and Environment for Statistical Computing, 1
- 2014, IBM Watson Ecosystem Program
- wu, 2016, Google’s neural machine translation system Bridging the gap between human and machine translation
- 2016, Machine Learning American
- 2016, Google Cloud Machine Learning
- 2016, Matlab
- 2013, Billion Devices Will Wirelessly Connect to the Internet of Everything in 2020
- krikorian, 2010, Twitter by the Numbers Twitter
- 2016, H20
- 2016, Apache Mahout
- abadi, 2016, TensorFlow Large-Scale Machine Learning on Heterogeneous Systems
- langford, 2007, Vowpal wabbit online learning project
- rouse, 2009, Predictive Analytics Definition
- sukumar, 2014, Machine learning in the big data era: Are we there yet?, Proc 20th ACM SIGKDD Conf Knowl Discovery Data Mining Workshop Data Sci Social Good (KDD), 1
- 2016, GPU Applications Transforming Computational Research and Engineering
- torrey, 2010, Handbook of Research on Machine Learning Applications and Trends
- 2016, Apache Giraph
Cited by
Machine Learning: New Ideas and Tools in Environmental Science and Engineering
Zhong, S., Zhang, K., Bagheri, M., Burken, J. G., Gu, A., Li, B., Ma, X., Marrone, B. L., Ren, Z. J., Schrier, J., et al. (2021) · Environmental Science & Technology · doi:10.1021/acs.est.1c01339 · Cited by 945
Forecasting: theory and practice
Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M. Z., Barrow, D. K., Ben Taieb, S., Bergmeir, C., Bessa, R. J., Bijak, J., Boylan, J. E., et al. (2022) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.11.001 · Cited by 921
Modeling and forecasting building energy consumption: A review of data-driven techniques
Bourdeau, M., Zhai, X. Q., Nefzaoui, E., Guo, X., & Chatellier, P. (2019) · Sustainable Cities and Society · doi:10.1016/j.scs.2019.101533 · Cited by 792
Age of Information: A New Concept, Metric, and Tool
Kosta, A., Παππάς, Ν., & Angelakis, V. (2017) · Foundations and Trends® in Networking · doi:10.1561/1300000060 · Cited by 782
Challenges and Future Directions of Big Data and Artificial Intelligence in Education
Luan, H., Geczy, P., Lai, H., Gobert, J., Yang, S. J. H., Ogata, H., Baltes, J., Guerra, R., Li, P., & Tsai, C.-C. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.580820 · Cited by 667
“Moreover, it is also shifting from a single domain (e.g., domain expertise, or expert systems) to a cross-disciplinary approach through collaboration (Spikol et al., 2018; Krouska et al., 2019) and domain transfers (L’heureux et al., 2017).”
BackgroundKaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564
“With only six interviews, we cannot claim that this list is by any means exhaustive, but it is in line with other analyses of ML pitfalls [42].”
ResultSocial media big data analytics: A survey
Ghani, N. A., Hamid, S., Targio Hashem, I. A., & Ahmed, E. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2018.08.039 · Cited by 430
A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View
Liu, Q., Li, P., Zhao, W., Cai, W., Yu, S., & Leung, V. C. M. (2018) · IEEE Access · doi:10.1109/access.2018.2805680 · Cited by 421
“worth noticing that new intelligent systems mainly focus on learning from massive amounts of data with the goals of high efficiency, minimum computational cost and considerable predictive or classification accuracy [3], [5].”
BackgroundThapa, C., & Camtepe, S. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104130 · Cited by 393
“Ensemble Learning: Ensemble Learning implements several (similar or different) learning algorithms (models) to solve the same problem, and then combine the outputs from each algorithm (model) to get a better final result in terms of predictions or classification [183,189] of the problem.”
BackgroundMethodBaek, Y., & Kim, H. Y. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.07.019 · Cited by 372
Wu, Y., Sicard, B., & Gadsden, S. A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124678 · Cited by 357
Understanding artificial intelligence ethics and safety
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 134
Crop Prediction Model Using Machine Learning Algorithms
Elbasi, E., Zaki, C., Topcu, A. E., Abdelbaki, W., Zreikat, A. I., Cina, E., Shdefat, A., & Saker, L. (2023) · Applied Sciences · doi:10.3390/app13169288 · Cited by 329
Imoize, A. L., Adedeji, O., Tandiya, N., & Shetty, S. (2021) · Sensors · doi:10.3390/s21051709 · Cited by 281
Implementation of Machine Learning Model to Predict Heart Failure Disease
Alotaibi, F. S. (2019) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2019.0100637 · Cited by 278
“As described by [25] the size of the dataset for machine learning implementation may create biasness and would also effect on the results generated through machine learning models.”
BackgroundTrunk, A., Birkel, H., & Hartmann, E. (2020) · Business Research · doi:10.1007/s40685-020-00133-x · Cited by 252
“This, however, also increases the risk of few people possessing too much power and managers must always be aware that the use of AI can bring additional dangers and challenges, such as bias in several dimensions (Flath and Stein 2018; L’Heureux et al. 2017).”
BackgroundPerspectives on the integration between first-principles and data-driven modeling
Bradley, W., Kim, J., Kilwein, Z., Blakely, L., Eydenberg, M., Jalvin, J., Laird, C., & Boukouvala, F. (2022) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2022.107898 · Cited by 231
Transfer learning with seasonal and trend adjustment for cross-building energy forecasting
Ribeiro, M., Grolinger, K., ElYamany, H. F., Higashino, W. A., & Capretz, M. A. M. (2018) · Energy and Buildings · doi:10.1016/j.enbuild.2018.01.034 · Cited by 225
Self-citationA Review on Data Cleansing Methods for Big Data
Ridzuan, F., & Wan Zainon, W. M. N. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.11.177 · Cited by 223
Fekri, M. N., Grolinger, K., & Mir, S. (2022) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2021.107669 · Cited by 220
“Conventional ML typically assumes independent and identically distributed (IID) variables [34, 31, 35]; however, data collection in the FL setup violates this assumption.”
BackgroundSelf-citationChallenges to Use Machine Learning in Agricultural Big Data: A Systematic Literature Review
Cravero, A., Pardo, S., Sepúlveda, S., & Muñoz, L. (2022) · Agronomy · doi:10.3390/agronomy12030748 · Cited by 218
Kunang, Y. N., Nurmaini, S., Stiawan, D., & Suprapto, B. Y. (2021) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2021.102804 · Cited by 215
Secure Smart Communication Efficiency in Federated Learning: Achievements and Challenges
Pouriyeh, S., Shahid, O., Parizi, R. M., Sheng, Q. Z., Srivastava, G., Zhao, L., & Nasajpour, M. (2022) · Applied Sciences · doi:10.3390/app12188980 · Cited by 213
Vudugula, S., Chebrolu, S. K., Bhuiyan, M., & Rozony, F. Z. (2023) · doi:10.63125/s5skge53 · Cited by 274
Drift-Aware Methodology for Anomaly Detection in Smart Grid
Fenza, G., Gallo, M., & Loia, V. (2019) · IEEE Access · doi:10.1109/access.2019.2891315 · Cited by 199
“In Machine Learning, concept drift refers to an unexpected changing in the statistical properties of the predicted variable [4] that could cause a substantial decrease in the prediction performances.”
BackgroundSafat, W., Asghar, S., & Gillani, S. A. (2021) · IEEE Access · doi:10.1109/access.2021.3078117 · Cited by 195
“Therefore, time series analysis is required to generate visual patterns along with a deep learning algorithm specifically LSTM, which provides the better classification of crimes over time based on adequate measures [16].”
Artificial Intelligence in Point-of-Care Biosensing: Challenges and Opportunities
Flynn, C. D., & Chang, D. (2024) · Diagnostics · doi:10.3390/diagnostics14111100 · Cited by 194
Transformer-Based Model for Electrical Load Forecasting
L’Heureux, A., Grolinger, K., & Capretz, M. A. M. (2022) · Energies · doi:10.3390/en15144993 · Cited by 179
“[14] in the context of load forecasting.”
Self-citationMovie Recommender System Using K-Means Clustering AND K-Nearest Neighbor
Ahuja, R., Solanki, A., & Nayyar, A. (2019) · 2019 9th International Conference on Cloud Computing, Data Science & Engineering (Confluence) · doi:10.1109/confluence.2019.8776969 · Cited by 178
A mini-review of machine learning in big data analytics: Applications, challenges, and prospects
Nti, I. K., Quarcoo, J. A., Aning, J., & Fosu, G. K. (2022) · Big Data Mining and Analytics · doi:10.26599/bdma.2021.9020028 · Cited by 207
Face mask recognition system using CNN model
Kaur, G., Sinha, R., Tiwari, P. K., Yadav, S. K., Pandey, P., Raj, R., Vashisth, A., & Rakhra, M. (2021) · Neuroscience Informatics · doi:10.1016/j.neuri.2021.100035 · Cited by 197
“enable the comparison of millions of transactions and the distinction between legal and illegal transactions between buyers and sellers [30].”
Waqas, M. M., Tu, S., Halim, Z., Rehman, S. U., Abbas, G., & Abbas, Z. H. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10143-2 · Cited by 196
“Notably, advanced intelligent systems primarily emphasize learning from vast amounts of data with the goals of high efficiency, minimum computational cost, and substantial predictive or classification accuracy (Prasad and Rohokale 2020; Lheureux et al. 2017).”
BackgroundTopological data analysis in biomedicine: A review
Skaf, Y., & Laubenbacher, R. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104082 · Cited by 165
Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review
Rosendo, D., Costan, A., Valduriez, P., & Antoniu, G. (2022) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2022.04.004 · Cited by 203
“[88, 165, 84, 52] 5% Lifelong Learning or Continual Learning [133] Learning is continuous and knowledge is retained and used to solve different problems.”
BackgroundMethodInfluentialFusing Stretchable Sensing Technology with Machine Learning for Human–Machine Interfaces
Wang, M., Wang, T., Luo, Y., He, K., Pan, L., Li, Z., Cui, Z., Liu, Z., Tu, J., & Chen, X. (2021) · Advanced Functional Materials · doi:10.1002/adfm.202008807 · Cited by 164
Acciarini, C., Cappa, F., Boccardelli, P., & Oriani, R. (2023) · Technovation · doi:10.1016/j.technovation.2023.102713 · Cited by 188
Mumuni, A., & Mumuni, F. (2025) · Journal of Information and Intelligence · doi:10.1016/j.jiixd.2024.01.002 · Cited by 159
“While deep learning methods have performed incredibly well, in the era of Big Data they have been shown to lack the power necessary to handle large volumes of complex heterogeneous data [1], [2].”
MethodExploring machine learning potential for climate change risk assessment
Zennaro, F., Furlan, E., Simeoni, C., Torresan, S., Aslan, S., Critto, A., & Marcomini, A. (2021) · Earth-Science Reviews · doi:10.1016/j.earscirev.2021.103752 · Cited by 167
Deep learning for power quality
de Oliveira, R. A., & Bollen, M. (2022) · Electric Power Systems Research · doi:10.1016/j.epsr.2022.108887 · Cited by 152
Establishing Data Provenance for Responsible Artificial Intelligence Systems
Werder, K., Ramesh, B., & Zhang, R. (. (2022) · ACM Transactions on Management Information Systems · doi:10.1145/3503488 · Cited by 106
“An easily ignorable fact is the non-stationary nature of the environment and the population from which all the input data of AI-based systems are generated [59].”
BackgroundArtificial intelligence in cardiology: Hope for the future and power for the present
Karatzia, L., Aung, N., & Aksentijević, D. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.945726 · Cited by 169
A Unified Architectural Approach for Cyberattack-Resilient Industrial Control Systems
Zhou, C., Hu, B., Shi, Y., Tian, Y.-C., Li, X., & Zhao, Y. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3034595 · Cited by 141
“The main focus of data-driven industrial security is on learning knowledge from the massive amount of data with high production efficiency, minimum computational cost, and accurate prediction or classification [188].”
BackgroundRemaining Useful Life Prediction for Bearings Based on a Gated Recurrent Unit
Que, Z., Jin, X., & Xu, Z. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3054025 · Cited by 148
“These enable the rapid development of data-driven approaches for bearing RUL prediction [8].”
Yazıcı, İ., Shayea, I. A. M., & Din, J. (2023) · Engineering Science and Technology an International Journal · doi:10.1016/j.jestch.2023.101455 · Cited by 174
N.G., B. A., & S., S. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.07.020 · Cited by 139
Use and Adaptations of Machine Learning in Big Data—Applications in Real Cases in Agriculture
Cravero, A., & Sepúlveda, S. (2021) · Electronics · doi:10.3390/electronics10050552 · Cited by 157
“However, information display problems have not been resolved [24].”
BackgroundMethodInfluentialFrom inverse optimal control to inverse reinforcement learning: A historical review
Azar, N. A., Shahmansoorian, A., & Davoudi, M. (2020) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2020.06.001 · Cited by 164
Machine Learning-Based Load Balancing Algorithms in Future Heterogeneous Networks: A Survey
Güreş, E., Shayea, I. A. M., Ergen, M., Azmi, M. H., & El‐Saleh, A. A. (2022) · IEEE Access · doi:10.1109/access.2022.3161511 · Cited by 160
“It describes the structural diversity of the dataset and the types of data it contains, as well as the variety, semantic interpretation and sources of what it represents [129].”
BackgroundMachine Learning in IoT Security: Current Solutions and Future Challenges
Hussain, F., Hussain, R., Hassan, S. A., & Hossain, E. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2986444 · Cited by 129
“Also, predictive ability of an algorithm decreases with the increase in the dimensionality of data [102].”
BackgroundJagait, R. K., Fekri, M. N., Grolinger, K., & Mir, S. H. (2021) · IEEE Access · doi:10.1109/access.2021.3095420 · Cited by 128
“Ensemble learning is a machine learning paradigm that combines several base models to improve learning outcomes [6].”
BackgroundInfluentialSelf-citationHuber, M. B., Kurz, C., & Leidl, R. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0731-6 · Cited by 116
“Additional uncertainty is partly rooted in the nature of machine learning where modelers have a wide variety of algorithms and approaches to choose from [21], unless more automated approaches are implemented [22].”
BackgroundShukla, S., Hassan, M. F., Khan, M. K., Jung, L. T., & Awang, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0224934 · Cited by 143
“Neuroevolution (NE), i.e., neural network evolution, is used for training NNs [44, 55].”
BackgroundMethodInfluentialOsman, A. I., Nasr, M., Farghali, M. R. F., Bakr, S. S., Eltaweil, A. S., Rashwan, A. K., & El-Monaem, E. M. A. (2024) · Environmental Chemistry Letters · doi:10.1007/s10311-023-01695-y · Cited by 132
Abbasi, A., Javed, A. R., Chakraborty, C., Nebhen, J., Zehra, W., & Jalil, Z. (2021) · IEEE Access · doi:10.1109/access.2021.3076264 · Cited by 133
“Online learning takes a ’learn-as-you-go approach and thus enables our model to learn one instance at a time, therefore not requiring entire data to be held in memory [7].”
BackgroundRobot-Enabled Construction Assembly with Automated Sequence Planning Based on ChatGPT: RoboGPT
You, H., Ye, Y., Zhou, T., Zhu, Q., & Du, J. (2023) · Buildings · doi:10.3390/buildings13071772 · Cited by 99
Pannakkong, W., Thiwa-anont, K., Singthong, K., Parthanadee, P., & Buddhakulsomsiri, J. (2022) · Mathematical Problems in Engineering · doi:10.1155/2022/8513719 · Cited by 119
“Another advantage is that ML algorithms can handle a large amount of input data for model training [1]. main reason for ML algorithms’ growing use is that ML algorithms can improve productivity and efficiency by automating them in the usage environment [1, 2].”
BackgroundVenkatesh, R., Balasubramanian, C., & Kaliappan, M. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1398-y · Cited by 135
On the relevance of data science for flight delay research: a systematic review
de Carvalho, L. S., Sternberg, A., Gonçalves, L. M., Cruz, A. B., Soares, J., Brandão, D. N., Carvalho, D., & Ogasawara, E. (2020) · Transport Reviews · doi:10.1080/01441647.2020.1861123 · Cited by 128
“Such analysis becomes more relevant as there is an increasing presence of machine learning methods (L ’ Heureux, Grolinger, Elyamany, & Capretz, 2017) to model fl ight delays.”
BackgroundEEG datasets for seizure detection and prediction— A review
Wong, S., Simmons, A., Rivera‐Villicana, J., Barnett, S., Sivathamboo, S., Perucca, P., Ge, Z., Kwan, P., Kuhlmann, L., Vasa, R., et al. (2023) · Epilepsia Open · doi:10.1002/epi4.12704 · Cited by 133
Reviewing Federated Machine Learning and Its Use in Diseases Prediction
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Sensors · doi:10.3390/s23042112 · Cited by 114
Naphade, M., Wang, S., Anastasiu, D. C., Tang, Z., Chang, M., Yang, X., Yao, Y., Zheng, L., Chakraborty, P., Sharma, A., et al. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00482 · Cited by 133
Arrowsmith, S., Trugman, D. T., MacCarthy, J., Bergen, K. J., Lumley, D., & Magnani, M. B. (2022) · Reviews of Geophysics · doi:10.1029/2021rg000769 · Cited by 123
“While the term “big” is field-, task- and context-dependent, so “big” data are broadly distinguished from small data by a set of characteristics referred to as the five Vs: volume, velocity, variety, veracity and value (L'heureux et al., 2017).”
BackgroundA Fully Convolutional Neural Network for Wood Defect Location and Identification
He, T., Liu, Y., Xu, C., Zhou, X., Hu, Z., & Fan, J. (2019) · IEEE Access · doi:10.1109/access.2019.2937461 · Cited by 113
“In addition to using data augmentation to increase the number of the dataset, we also used dropout regularization and L2 regularization to avoid over-fitting effectively [16], [17].”
MethodEdge-Cloud Computing for IoT Data Analytics: Embedding Intelligence in the Edge with Deep Learning
Ghosh, A. M., & Grolinger, K. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.3008711 · Cited by 209
“Due to its representation capabilities, ability to learn complex models, and diversity of architectures [2], DL has been successful in many domains including various vision tasks, natural language processing, and speech recognition.”
InfluentialSelf-citationMegantara, A. A., & Ahmad, T. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00531-w · Cited by 128
“The biggest challenge of machine learning methods is how to build an appropriate model to represent the dataset [8].”
BackgroundIoT data analytics in dynamic environments: From an automated machine learning perspective
Yang, L., & Shami, A. A. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105366 · Cited by 130
Dao, D. V., Ly, H., Vu, H.-L. T., Le, T.-T., & Pham, B. T. (2020) · Materials · doi:10.3390/ma13051072 · Cited by 109
“In addition, it is well-known 325 that the accuracy of the given machine learning algorithm greatly depends on the sampling strategy 326 to construct the model [85,86].”
BackgroundAhmad, G. N., Shafiullah, Algethami, A. A., Fatima, H., & Akhter, S. M. H. (2022) · IEEE Access · doi:10.1109/access.2022.3153047 · Cited by 120
“To learn ML algorithms, we need training data [15].”
MethodSoni, S., Berry, C. P. L., Coughlin, S. B., Harandi, M., Jackson, C. B., Crowston, K., Østerlund, C. S., Patane, O., Katsaggelos, A. K., Trouille, L. E., et al. (2021) · Classical and Quantum Gravity · doi:10.1088/1361-6382/ac1ccb · Cited by 101
Robot-Enabled Construction Assembly with Automated Sequence Planning based on ChatGPT: RoboGPT
You, H., Ye, Y., Zhou, T., Zhu, Q., & Du, J. (2023) · arXiv · doi:10.48550/arxiv.2304.11018 · Cited by 100
Orabi, M. M., Emam, O. E., & Fahmy, H. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01099-5 · Cited by 116
“…model training, it has two challenges and issues: (i) the transmission of data may use up a lot of communication resources, and (ii) data owners are less likely to upload data to the central server out of security concerns because the input of raw data raises the potential of privacy breach [10].”
Naphade, M., Wang, S., Anastasiu, D. C., Tang, Z., Chang, M., Yang, X., Zheng, L., Sharma, A., Chellappa, R., & Chakraborty, P. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00321 · Cited by 109
“However, difficulties including poor data quality, the lack of annotations, and the absence of high-quality models are some of the biggest impediments to unlocking the value of the data [21].”
Background15 years of Big Data: a systematic literature review
Tosi, D., Kokaj, R., & Roccetti, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00914-9 · Cited by 110
Abubakar, A. I., Ahmad, I., Omeke, K. G., Öztürk, M., Öztürk, C., Abdel-Salam, A. M., Mollel, M. S., Abbasi, Q. H., Hussain, S., & Imran, M. A. (2023) · Drones · doi:10.3390/drones7030214 · Cited by 104
“Moreover, the availability, quality, and sufficiency of the data to be employed in training is a big issue for ML-based solutions [21],”
BackgroundRevolutionizing Groundwater Management with Hybrid AI Models: A Practical Review
Zaresefat, M., & Derakhshani, R. (2023) · Water · doi:10.3390/w15091750 · Cited by 106
Explainability in supply chain operational risk management: A systematic literature review
Nimmy, S. F., Hussain, F. K., Chakrabortty, R. K., Hussain, F. K., & Saberi, M. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107587 · Cited by 112
Bioinspired electronics for intelligent soft robots
Bang, J., Choi, S. H., Pyun, K. R., Jung, Y., Hong, S., Kim, D., Lee, Y., Won, D., Jeong, S., Shin, W., et al. (2024) · Nature Reviews Electrical Engineering · doi:10.1038/s44287-024-00081-2 · Cited by 96
“The progression in transparent electronics and actuators has been pivotal in the evolution of transparent soft robots 48,49,201,202 .”
Federated learning for big data: A survey on opportunities, applications, and future directions
Gadekallu, T. R., Pham, Q., Huynh‐The, T., Feng, H., Fang, K., Pandya, S., Liyanage, M., Wang, W. W., & Nguyen, T. T. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.113614 · Cited by 94
Omrany, H., Al-Obaidi, K. M., Hossain, M., Alduais, N. A. M., Al-Duais, H. S., & Ghaffarianhoseini, A. (2024) · Discover Cities · doi:10.1007/s44327-024-00002-w · Cited by 103
“L’heureux et al. [87] indicated that batch learning represents a method to analyse static IoT data in batches via Traditional Machine learning algorithms.”
Responsible Data Governance of Neuroscience Big Data
Fothergill, B. T., Knight, W., Stahl, B. C., & Ulnicane, I. (2019) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2019.00028 · Cited by 97
“Veracity is defined by the quality (e.g., uncertainty, precision, and noise) and provenance (the origin, tracking, and movement) of the data (Gandomi and Haider, 2015; Wang et al., 2015; L’Heureux et al., 2017).”
BackgroundInfluentialUse of artificial intelligence to enhance phenotypic drug discovery
Malandraki-Miller, S., & Riley, P. R. (2021) · Drug Discovery Today · doi:10.1016/j.drudis.2021.01.013 · Cited by 97
Ebrahimi‐Khusfi, Z., Nafarzadegan, A. R., & Dargahian, F. (2021) · Ecological Indicators · doi:10.1016/j.ecolind.2021.107499 · Cited by 89
Machine Learning for Emergency Management: A Survey and Future Outlook
Kyrkou, C., Kolios, P., Theocharides, T., & Polycarpou, M. M. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3223186 · Cited by 100
“This feature sometimes calls for adopting data processing techniques, such as data cleaning, dimensionality reduction, and instance selection, as it is required to reduce the size of the datasets and help the ML model extract and use only the useful information contained in the data [33].”
MethodA survey on machine learning based analysis of heterogeneous data in industrial automation
Kamm, S., Veekati, S. S., Müller, T., Jazdi, N., & Weyrich, M. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.103930 · Cited by 97
Aaldering, L. J., Leker, J., & Song, C. H. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.03.174 · Cited by 88
Mathrani, A., Sušnjak, T., Ramaswami, G. S., & Barczak, A. L. C. (2021) · Computers and Education Open · doi:10.1016/j.caeo.2021.100060 · Cited by 97
In Search of Big Medical Data Integration Solutions - A Comprehensive Survey
Dhayne, H., Haque, R., Kilany, R., & Taher, Y. (2019) · IEEE Access · doi:10.1109/access.2019.2927491 · Cited by 84
End-to-End Federated Learning for Autonomous Driving Vehicles
Zhang, H., Bosch, J., & Olsson, H. H. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533808 · Cited by 84
Time series big data: a survey on data stream frameworks, analysis and algorithms
Almeida, A., Brás, S., Sargento, S., & Pinto, F. C. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00760-1 · Cited by 100
“1 Big data taxonomy—information collected from [2, 5, 15, 17, 19]”
Medical Data Processing and Analysis for Remote Health and Activities Monitoring
Vitabile, S., Marks, M., Stojanovic, D. H., Pllana, S., Molina, J. M., Krzysztoń, M., Sikora, A., Jarynowski, A. K., Hosseinpour, F., Jakóbik, A., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-16272-6_7 · Cited by 90
Food Quality Monitoring System Based on Smart Contracts and Evaluation Models
Yu, B., Zhan, P., Lei, M., Zhou, F., & Wang, P. (2020) · IEEE Access · doi:10.1109/access.2020.2966020 · Cited by 88
“The time complexities of these three methods are respectively O(mn2+ n3) [70], O(m2) [71] and O(n3) [72], where m is the number of samples and n is the number of features for GC-MS data in each sample.”
BackgroundMorfino, V., & Rampone, S. (2020) · Electronics · doi:10.3390/electronics9030444 · Cited by 91
“The application limits of state-of-the-art machine learning algorithms are mostly related to the computational requirements needed for large datasets [21].”
BackgroundA combined deep learning application for short term load forecasting
Özer, İ., Efe, S. B., & ÖZBAY, H. (2021) · Alexandria Engineering Journal · doi:10.1016/j.aej.2021.02.050 · Cited by 86
Machine learning and the Internet of Things security: Solutions and open challenges
Farooq, U., Tariq, N., Asim, M., Baker, T., & Al-Shamma'a, A. I. (2022) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2022.01.015 · Cited by 85
Computer Vision for Plant Disease Recognition: A Comprehensive Review
Dang, M. H., Wang, H., Li, Y., Nguyen, T., Tightiz, L., Xuan-Mung, N., & Nguyen, T. N. (2024) · The Botanical Review · doi:10.1007/s12229-024-09299-z · Cited by 91
Information retrieval methodology for aiding scientific database search
Marcos-Pablos, S., & García‐Peñalvo, F. J. (2018) · Soft Computing · doi:10.1007/s00500-018-3568-0 · Cited by 93
Intelligent Condition Monitoring of Wind Power Systems: State of the Art Review
Benbouzid, M., Berghout, T., Sarma, N., Djurović, S., Wu, Y., & Ma, X. (2021) · Energies · doi:10.3390/en14185967 · Cited by 77
“24 of 33”
Reviewing Multimodal Machine Learning and Its Use in Cardiovascular Diseases Detection
Moshawrab, M., Adda, M., Bouzouane, A., Ibrahim, H., & Raad, A. (2023) · Electronics · doi:10.3390/electronics12071558 · Cited by 85
“These challenges have been extensively studied in the literature, and more details can be found in several articles, such as [9,21–23].”
BackgroundMethodInfluentialThe accuracy of XGBoost for insurance claim prediction
Fauzan, M. A., & Murfi, H. (2018) · Cited by 76
A Profit Function-Maximizing Inventory Backorder Prediction System Using Big Data Analytics
Hájek, P., & Abedin, M. Z. (2020) · IEEE Access · doi:10.1109/access.2020.2983118 · Cited by 80
“Therefore, more adaptive and robust models must be developed to overcome these problems [43].”
Background
Metadata
- Authors
- L'Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M.
- Affiliations
- Western University (CA)
- Suez Canal University (EG)
- Venue
- IEEE Access
- Published
- 2017
- Volume
- 5
- Pages
- 7776-7797
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2609731728
- Cited by
- 1,023
- References
- 109
- Citations by year
- 2026: 50 · 2025: 175 · 2024: 128 · 2023: 145 · 2022: 130 · 2021: 138 · 2020: 105 · 2019: 88 · 2018: 53 · 2017: 11
- FWCI
- 112.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- NSERC CRD at Western University (477530-14)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- big data challenges, volume, velocity, varieties, honesty, scalable learning, data-driven decision making, research gaps
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.