Big data quality framework: a holistic approach to continuous quality management
Open accessThe DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Abstract
Big Data is an essential research area for governments, institutions, and private agencies to support their analytics decisions. Big Data refers to all about data, how it is collected, processed, and analyzed to generate value-added data-driven insights and decisions. Degradation in Data Quality may result in unpredictable consequences. In this case, confidence and worthiness in the data and its source are lost. In the Big Data context, data characteristics, such as volume, multi-heterogeneous data sources, and fast data generation, increase the risk of quality degradation and require efficient mechanisms to check data worthiness. However, ensuring Big Data Quality (BDQ) is a very costly and time-consuming process, since excessive computing resources are required. Maintaining Quality through the Big Data lifecycle requires quality profiling and verification before its processing decision. A BDQ Management Framework for enhancing the pre-processing activities while strengthening data control is proposed. The proposed framework uses a new concept called Big Data Quality Profile. This concept captures quality outline, requirements, attributes, dimensions, scores, and rules. Using Big Data profiling and sampling components of the framework, a faster and efficient data quality estimation is initiated before and after an intermediate pre-processing phase. The exploratory profiling component of the framework plays an initial role in quality profiling; it uses a set of predefined quality metrics to evaluate important data quality dimensions. It generates quality rules by applying various pre-processing activities and their related functions. These rules mainly aim at the Data Quality Profile and result in quality scores for the selected quality attributes. The framework implementation and dataflow management across various quality management processes have been discussed, further some ongoing work on framework evaluation and deployment to support quality evaluation decisions conclude the paper.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, this singular focus can flatten the variety of uses and data quality requirements that are encountered at various stages of the ML development pipeline over the longer-term [57,70].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [70]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Governance frameworks that integrate ethical safeguards [119,121] are therefore vital to sustaining trust.”
From Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles (2025) · cites this paper as [121]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2011) · Cited by 5,203
Beyond Accuracy: What Data Quality Means to Data Consumers
Wang, R. Y., & Strong, D. M. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518099 · Cited by 4,672
Beyond the hype: Big data concepts, methods, and analytics
Gandomi, A., & Haider, M. (2015) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2014.10.007 · Cited by 4,226
Data-intensive applications, challenges, techniques and technologies: A survey on Big Data
Philip Chen, C. L., & Zhang, C.-Y. (2014) · Information Sciences · doi:10.1016/j.ins.2014.01.015 · Cited by 3,047
Chen, M., Mao, S., & Liu, Y. (2014) · Mobile Networks and Applications · doi:10.1007/s11036-013-0489-0 · Cited by 2,913 · cited 2×
“• Data acquisition: it consists of data collection, data transmission, and data preprocessing [1, 10].”
BackgroundMethodThe rise of big data on cloud computing: Review and open research issues
Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Ullah Khan, S. (2015) · Information Systems · doi:10.1016/j.is.2014.07.006 · Cited by 2,656
Anchoring Data Quality Dimensions in Ontological Foundations
Wand, Y., & Wang, R. Y. (1996) · Communications of the ACM · doi:10.1145/240455.240479 · Cited by 1,479 · cited 1×
“There are several DQDs, which can be organized into 4 categories according to [24, 25], intrinsic, contextual, accessibility, and representational [14, 15, 22, 24, 26, 27].”
Background(2000) · Cited by 1,467
Methodologies for data quality assessment and improvement
Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541883 · Cited by 1,288
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial
Hu, H., Wen, Y., Chua, T.-S., & Li, X. (2014) · IEEE Access · doi:10.1109/access.2014.2332453 · Cited by 1,099
Research works printed but not linked: 3
- Wielki J. The Opportunities and Challenges Connected with Implementation of the Big Data Concept. In: Mach-Król M, Olszak CM, Pełech-Pilichowski T, editors. Advances in ICT for Business. Springer International Publishing: Industry and Public Sector, Studies in Computational Intelligence; 2015. p. 171–89.
- Kleiner, A., Talwalkar, A., Sarkar, P., Jordan, M., 2012. The big data bootstrap. arXiv preprint
- Firmani, D., Mecella, M., Scannapieco, M., Batini, C., 2015. On the Meaningfulness of “Big Data Quality” (Invited Paper), in: Data Science and Engineering. Springer Berlin Heidelberg, pp. 1–15. https://doi.org/10.1007/s41019-015-0004-7 · DOI · Paper page
Cited by
A survey on dataset quality in machine learning
Gong, Y., Liu, G., Xue, Y., Li, R., & Meng, L. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107268 · Cited by 354
Nsoh, B., Katimbo, A., Guo, H., Heeren, D. M., Nakabuye, H. N., Qiao, X., Ge, Y., Rudnick, D. R., Wanyama, J., Bwambale, E., et al. (2024) · Sensors · doi:10.3390/s24237480 · Cited by 158
Theodorakopoulos, L., Theodoropoulou, A., & Stamatiou, Y. C. (2024) · Eng—Advances in Engineering · doi:10.3390/eng5030068 · Cited by 145
Data governance & quality management—Innovation and breakthroughs across different fields
Bernardo, B. M. V., Mamede, H. S., Barroso, J. M. P., & dos Santos, V. M. P. D. (2024) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2024.100598 · Cited by 129
Idemudia, C., Ige, A. B., Adebayo, V. I., & Eyieyien, O. G. (2024) · Computer Science & IT Research Journal · doi:10.51594/csitrj.v5i7.1352 · Cited by 100
Metadata
- Authors
- Taleb, I., Serhani, M. A., Bouhaddioui, C., & Dssouli, R.
- Affiliations
- Zayed University (AE)
- United Arab Emirates University (AE)
- Corresponding author
- Serhani, Mohamed Adel
- Venue
- Journal of Big Data
- Published
- 2021-05-29
- Volume
- 8
- Issue
- 1
- ISSN
- 2196-1115 (ISSN-L)
- OpenAlex
- W4210862833
- Cited by
- 149
- References
- 66
- Citations by year
- 2026: 14 · 2025: 45 · 2024: 40 · 2023: 28 · 2022: 18 · 2021: 3
- FWCI
- 13.53
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Journal Of Big Data (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- ZCHS at UAE University (12R005)
- Topics
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Social Sciences › Business, Management and Accounting › Management Information Systems › Big Data and Business Intelligence
- Keywords
- big data quality, data profiling, data quality metrics, data preprocessing, heterogeneous data sources
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Big data quality framework: a holistic approach to continuous quality management. https://app.cochat.ai/papers/10.1186/s40537-021-00468-0
- The paper
- Taleb, I., Serhani, M. A., Bouhaddioui, C., & Dssouli, R. (2021). Big data quality framework: a holistic approach to continuous quality management. Journal of Big Data. https://doi.org/10.1186/s40537-021-00468-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.