An Automated Big Data Quality Anomaly Correction Framework Using Predictive Analysis
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Big data has emerged as a fundamental component in various domains, enabling organizations to extract valuable insights and make informed decisions. However, ensuring data quality is crucial for effectively using big data. Thus, big data quality has been gaining more attention in recent years by researchers and practitioners due to its significant impact on decision-making processes. However, existing studies addressing data quality anomalies often have a limited scope, concentrating on specific aspects such as outliers or inconsistencies. Moreover, many approaches are context-specific, lacking a generic solution applicable across different domains. To the best of our knowledge, no existing framework currently automatically addresses quality anomalies comprehensively and generically, considering all aspects of data quality. To fill the gaps in the field, we propose a sophisticated framework that automatically corrects big data quality anomalies using an intelligent predictive model. The proposed framework comprehensively addresses the main aspects of data quality by considering six key quality dimensions: Accuracy, Completeness, Conformity, Uniqueness, Consistency, and Readability. Moreover, the framework is not correlated to a specific field and is designed to be applicable across various areas, offering a generic approach to address data quality anomalies. The proposed framework was implemented on two datasets and has achieved an accuracy of 98.22%. Moreover, the results have shown that the framework has allowed the data quality to be boosted to a great score, reaching 99%, with an improvement rate of up to 14.76% of the quality score.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Accelerating the XGBoost algorithm using GPU computing
Mitchell, R., & Frank, E. (2017) · PeerJ Computer Science · doi:10.7717/peerj-cs.127 · Cited by 359
Lian, B., Kartal, Y., Lewis, F. L., Mikulski, D., Hudas, G. R., Wan, Y., & Davoudi, A. A. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3213526 · Cited by 66
Machine Learning Technologies for Big Data Analytics
Gandomi, A. H., Chen, F., & Abualigah, L. (2022) · Electronics · doi:10.3390/electronics11030421 · Cited by 60
Ali, S., Poulová, P., Yasmin, F., Danish, M., Akhtar, W., & Javed, H. M. U. (2020) · Journal of Open Innovation Technology Market and Complexity · doi:10.3390/joitmc6040190 · Cited by 59
An Advanced Big Data Quality Framework Based on Weighted Metrics
Elouataoui, W., Alaoui, I. E., Mendili, S. E., & Gahi, Y. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6040153 · Cited by 59
Self-citationContext-aware Big Data Quality Assessment: A Scoping Review
Fadlallah, H., Kilany, R., Dhayne, H., Haddad, R. E., Haque, R., Taher, Y., & Jaber, A. H. (2023) · Journal of Data and Information Quality · doi:10.1145/3603707 · Cited by 50
Novel Methods for Imputing Missing Values in Water Level Monitoring Data
Khampuengson, T., & Wang, W. (2023) · Water Resources Management · doi:10.1007/s11269-022-03408-6 · Cited by 36
Miss-gradient boosting regression tree: a novel approach to imputing water treatment data
Zhang, W., Li, R., Zhao, J., Wang, J., Meng, X., & Li, Q. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04828-6 · Cited by 27
Perspective of anomaly detection in big data for data quality improvement
Keskar, V., Yadav, J., & Kumar, A. (2021) · Materials Today Proceedings · doi:10.1016/j.matpr.2021.05.597 · Cited by 23
A Secure Multi-User Database-as-a-Service Approach for Cloud Computing Privacy
Gahi, Y., & Alaoui, I. E. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.11.006 · Cited by 20
Self-citation
Research works printed but not linked: 3
- (2023, November 09). Big Data Analytics: A Comparison of Tools and Applications|SpringerLink. Available online: https://link.springer.com/chapter/10.1007/978-3-319-74500-8_54. · DOI
- Goyle, K., Xie, Q., and Goyle, V. (2023). DataAssist: A Machine Learning Approach to Data Cleaning and Preparation. arXiv.
- (2023, July 08). Data Quality for Big Data and Machine Learning|Frontiers Research Topic. Available online: https://www.frontiersin.org/research-topics/35131/data-quality-for-big-data-and-machine-learning.
Cited by
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
“Predictive approaches, such as XGBoost, can correct anomalies across multiple dimensions (accuracy, completeness, conformity, uniqueness, consistency, and readability), improving composite Data Quality Index scores from 70% to over 90% in large-scale applications [219].”
Data quality and uncertainty issues in flood prediction: a systematic review
Yu, J., Li, Y., Huang, X., & Ye, X. (2025) · International Journal of Digital Earth · doi:10.1080/17538947.2025.2495738 · Cited by 30
“However, issues related to data quality and uncertainty have not been given sufficient attention (Dottori, Di Baldassarre, and Todini 2013; Elouataoui, El Mendili, and Gahi 2023).”
A Survey on Data Quality Dimensions and Tools for Machine Learning Invited Paper
Zhou, Y., Tu, F., Sha, K., Ding, J., & Chen, H. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00023 · Cited by 43
“The quality of training data impacts the accuracy, reliability, and interpretation of ML models’ results [56], [57].”
BackgroundHasan, M. N., Papel, M. S. I., Rasel, I. H., Akter, S., Aktar, M. K., Abedin, M. Z., & Mani, L. (2025) · Edelweiss Applied Science and Technology · doi:10.55214/2576-8484.v9i7.9142 · Cited by 22
Avital, N., Egel, I., Weinstock, I., & Malka, D. (2024) · Inventions · doi:10.3390/inventions9060113 · Cited by 17
Sumarlin, T., & Kusumajaya, R. A. (2024) · Journal of Management and Informatics · doi:10.51903/jmi.v3i2.25 · Cited by 17
“Additionally, the ethical and data privacy issues raised in this research reflect concerns highlighted by (Elouataoui et al., 2023), who stressed the importance of transparency in AI algorithms to avoid bias in decision-making.”
BackgroundEnhancing Data Quality in Medical Concept Normalization Through Large Language Models
Chen, H., Li, R., Cleveland, A. D., & Ding, J. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104812 · Cited by 16
Enhancing DataOps practices through innovative collaborative models: A systematic review
Fannouch, A., Gharib, J., & Gahi, Y. (2025) · International Journal of Information Management Data Insights · doi:10.1016/j.jjimei.2025.100321 · Cited by 20
Self-citationBłachowicz, T., Wylezek, J., Sokol, Z., & Bondel, M. (2025) · Information · doi:10.3390/info16020079 · Cited by 7
Selecting the Best Regression Model for Wind Power Prediction and Management for VPP
Roy, S., Singh, D., Sinha, N., Das, D. C., & Sinha, N. (2025) · Journal of Renewable Energy and Smart Grid Technology · doi:10.69650/rast.2025.260878 · Cited by 7
“Research by Elouataoui et al. [28] and Budach et al. [29] has highlighted the significance of resolving data quality challenges, such as handling missing values and outliers, to improve the accuracy and dependability of these models.”
Reliable and Secure Anomaly Detection in Heterogeneous Federated Learning: A Comprehensive Review
Xiang, H., Wang, G., Yu, X., Di, F., Gao, R., Zhang, Y., & Han, X. (2026) · Big Data Mining and Analytics · doi:10.26599/bdma.2025.9020067 · Cited by 13
Gbemavo, D. S. J. C., Laly, J., & Adjahossou, V. N. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e40600 · Cited by 6
Active Metadata and Machine Learning based Framework for Enhancing Big Data Quality
Elouataoui, W., Mendili, S. E., & Gahi, Y. (2024) · International Conference on Networking, Information Systems & Security (NISS) · doi:10.1145/3659677.3659707 · Cited by 5
“Other recent approaches addressing Big Data quality issues were discussed in [19] [20] [21].”
Self-citationSmart Technologies in Poultry Production: Trends and Innovations
Thomas, L., Abe, J. M., Leite, M. V., Nääs, I. D. A., Souza, M. L. H., & Gomes, D. D. A. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.570 · Cited by 4
Elouataoui, W., & Gahi, Y. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc10010007 · Cited by 4
Self-citationBridging data gaps: homogenisation of climate time series in the Ethiopian Abay Basin
Ayalew, D. W., Kumar, N., Borgemeister, C., & Tischbein, B. (2025) · Theoretical and Applied Climatology · doi:10.1007/s00704-025-05950-w · Cited by 3
Improving Asset Information Management in the Oil & Gas Sector: A Comprehensive Review
Singh, A. P. A., & Gameti, N. (2024) · IEEE Pune Section International Conference (PuneCon) · doi:10.1109/punecon63413.2024.10895047 · Cited by 3
“The model achieved a high accuracy of up to 98.22%, indicating a strong potential for identifying unusual patterns within large datasets [15].”
InfluentialEnsuring Reliable Data: Innovative Approaches to Metadata Research and Practices
Tella, A., & Mandu, M. (2025) · Bibliotechnyi visnyk · doi:10.15407/bv2025.04.132 · Cited by 4
Data Quality Assessment Framework for Predictive Maintenance (PdM) in Industrial AI: Review
AlKalbani, W., AlAzri, A., & Ahmad, M. N. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3808-9_3 · Cited by 1
Rethinking Big Data Value Chains: A Comparative Framework Across Hadoop, Modern, and Cloud Stacks
Elouataoui, W., & Gahi, Y. (2025) · International Conference on Optimization and Applications (ICOA) · doi:10.1109/icoa66896.2025.11236898 · Cited by 1
“These technologies allowed users to manage large datasets across clusters, making Hadoop one of the most adopted platforms for big data analytics for several years [1].”
Self-citationBig-MDC: A Multi-Dimensions Cleaning Framework to Improve Big Data Quality
Nadjeh, N., Abdellaoui, S., & Nader, F. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00804-0
“In the big data area, only one work addressing multiple data quality dimensions exist [13].”
Automated Data Quality Monitoring and Anomaly Detection
Pal, M. (2013) · International Journal of Innovative Research in Computer and Communication Engineering · doi:10.15680/ijircce.2013.0102063
Pittala, V. S., Dhanushya, G., Brahmanaidu, K., & Krishna, K. (2025) · International Conference on Augmented Intelligence and Sustainable Systems (ICAISS) · doi:10.1109/icaiss61471.2025.11042002
Automated Data Quality Monitoring and Anomaly Detection
Pal, M. (2023) · International Journal of Innovative Research in Computer and Communication Engineering · doi:10.15680/ijircce.2023.1107084
Gonzales, R. J. L. (2025) · LATAM Revista Latinoamericana de Ciencias Sociales y Humanidades · doi:10.56712/latam.v6i5.4684
The Role of AI and Machine Learning in Predictive Safety Analytics
Selvakumar, P., Hiremath, V. M. R., Selvi, R. K., Rambabu, U., Uprikar, V. N., & Rithe, N. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-6370-7.ch010
AI-Driven Data Quality Management with Generativelearning and Anomaly Detection
Tummuri, S. S. R. (2026) · IEEE International Conference on Emerging Computing and Intelligent Technologies (ICoECIT) · doi:10.1109/icoecit68303.2026.11497864
“The structure is closed-loop based on learning normal data behavior, deviation identification, and self-refinement in terms of changing input patterns [11].”
Adaptive Ai-Driven Total Quality Management for Higher Education Excellence
Elganas, T., Moir, R., & Alkhateeb, A. (2026) · International Journal of Project Management · doi:10.47672/ijpm.2916
Evaluating the Role of Data Mining in Enhancing Decision-Making
Sikri, A. (2026) · BENTHAM SCIENCE PUBLISHERS eBooks · doi:10.2174/9798898815318126040012
Baddam, V., & Sama, S. (2026) · International Conference on Augmented Intelligence and Sustainable Systems (ICAISS) · doi:10.1109/icaiss68683.2026.11526660
Improving the Quality of Enterprise Data Management with Tree-Based Models
Büyükbıçakcı, E. (2026) · Celal Bayar Üniversitesi Fen Bilimleri Dergisi · doi:10.18466/cbayarfbe.1677875
Medición de la comprensibilidad de datos en analítica de negocio: el framework B-DataCom
España, S. A. M., Ruiz, S. L. B., & Correa, F. J. P. (2026) · Ingeniería y Competitividad · doi:10.25100/iyc.v28i2.15823 · Cited by 1
Ayyadurai, R., Maamra, O. E., Badgujar, V. S., Fahim, F. I., Mishra, B. K., & Praseptiawan, M. (2026) · IGI Global eBooks · doi:10.4018/979-8-2600-5218-1.ch002
Metadata
- Authors
- Elouataoui, W., El Mendili, S., & Gahi, Y.
- Affiliations
- Université Ibn-Tofail (MA)
- Laboratory of Engineering Sciences, National School of Applied Sciences, Ibn Tofail University, Kenitra 14000, Morocco
- Corresponding author
- Elouataoui, Widad
- Gahi, Youssef
- Venue
- Data
- Publisher
- MDPI AG
- Published
- 2023-12-01
- Volume
- 8
- Issue
- 12
- Pages
- 182
- ISSN
- 2306-5729 (ISSN-L)
- OpenAlex
- W4389261600
- Cited by
- 27
- References
- 29
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 6 · 2023: 1
- FWCI
- 4.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Data (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- big data quality, automated data cleaning, predictive modeling, data quality dimensions, accuracy, completeness, conformity, uniqueness, consistency, readability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.