CoChat

An Automated Big Data Quality Anomaly Correction Framework Using Predictive Analysis

Open access
Elouataoui, Widad, El Mendili, Saida, Gahi, Youssef · 2023 · Data

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.

Abstract

Big data has emerged as a fundamental component in various domains, enabling organizations to extract valuable insights and make informed decisions. However, ensuring data quality is crucial for effectively using big data. Thus, big data quality has been gaining more attention in recent years by researchers and practitioners due to its significant impact on decision-making processes. However, existing studies addressing data quality anomalies often have a limited scope, concentrating on specific aspects such as outliers or inconsistencies. Moreover, many approaches are context-specific, lacking a generic solution applicable across different domains. To the best of our knowledge, no existing framework currently automatically addresses quality anomalies comprehensively and generically, considering all aspects of data quality. To fill the gaps in the field, we propose a sophisticated framework that automatically corrects big data quality anomalies using an intelligent predictive model. The proposed framework comprehensively addresses the main aspects of data quality by considering six key quality dimensions: Accuracy, Completeness, Conformity, Uniqueness, Consistency, and Readability. Moreover, the framework is not correlated to a specific field and is designed to be applicable across various areas, offering a generic approach to address data quality anomalies. The proposed framework was implemented on two datasets and has achieved an accuracy of 98.22%. Moreover, the results have shown that the framework has allowed the data quality to be boosted to a great score, reaching 99%, with an improvement rate of up to 14.76% of the quality score.

Overview

Integrity signals

Registry

Crossref · 10.3390/data8120182

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 30 references

Research works printed but not linked: 3

  1. (2023, November 09). Big Data Analytics: A Comparison of Tools and Applications|SpringerLink. Available online: https://link.springer.com/chapter/10.1007/978-3-319-74500-8_54. · DOI
  2. Goyle, K., Xie, Q., and Goyle, V. (2023). DataAssist: A Machine Learning Approach to Data Cleaning and Preparation. arXiv.
  3. (2023, July 08). Data Quality for Big Data and Machine Learning|Frontiers Research Topic. Available online: https://www.frontiersin.org/research-topics/35131/data-quality-for-big-data-and-machine-learning.

Cited by

See all 33

Metadata

Authors
Elouataoui, W., El Mendili, S., & Gahi, Y.
Affiliations
  • Université Ibn-Tofail (MA)
  • Laboratory of Engineering Sciences, National School of Applied Sciences, Ibn Tofail University, Kenitra 14000, Morocco
Corresponding author
  • Elouataoui, Widad
  • Gahi, Youssef
Venue
Data
Publisher
MDPI AG
Published
2023-12-01
Volume
8
Issue
12
Pages
182
ISSN
2306-5729 (ISSN-L)
OpenAlex
W4389261600
Cited by
27
References
29
Citations by year
2026: 6 · 2025: 13 · 2024: 6 · 2023: 1
FWCI
4.80
Type
journal article
Language
en
Open access
Locations
  • Data (publishedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Information Systems › Data Quality and Management
  • Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
big data quality, automated data cleaning, predictive modeling, data quality dimensions, accuracy, completeness, conformity, uniqueness, consistency, readability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.