How to Address the Data Quality Issues in Regression Models: A Guided Process for Data Cleaning
Open accessAbstract
Today, data availability has gone from scarce to superabundant. Technologies like IoT, trends in social media and the capabilities of smart-phones are producing and digitizing lots of data that was previously unavailable. This massive increase of data creates opportunities to gain new business models, but also demands new techniques and methods of data quality in knowledge discovery, especially when the data comes from different sources (e.g., sensors, social networks, cameras, etc.). The data quality process of the data set proposes conclusions about the information they contain. This is increasingly done with the aid of data cleaning approaches. Therefore, guaranteeing a high data quality is considered as the primary goal of the data scientist. In this paper, we propose a process for data cleaning in regression models (DC-RM). The proposed data cleaning process is evaluated through a real datasets coming from the UCI Repository of Machine Learning Databases. With the aim of assessing the data cleaning process, the dataset that is cleaned by DC-RM was used to train the same regression models proposed by the authors of UCI datasets. The results achieved by the trained models with the dataset produced by DC-RM are better than or equal to that presented by the datasets’ authors.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Only three data cleaning techniques [9,18,42] address high dimensionality.
“Only three data cleaning techniques [9,18,42] address high dimensionality.”
From A Systematic Review of Data Quality in CPS and IoT for Industry 4.0 (2023) · cites this paper as [9]
The three techniques are offline, and none of them provide online support for reducing high dimensions in input datasets for real-time ML applications.
“The three techniques are offline, and none of them provide online support for reducing high dimensions in input datasets for real-time ML applications.”
From A Systematic Review of Data Quality in CPS and IoT for Industry 4.0 (2023) · cites this paper as [9]
Two techniques [9,18] employ the principal component analysis [53] to filter out irrelevant attributes.
“Two techniques [9,18] employ the principal component analysis [53] to filter out irrelevant attributes.”
From A Systematic Review of Data Quality in CPS and IoT for Industry 4.0 (2023) · cites this paper as [9]
Cites
A survey on feature selection methods
Chandrashekar, G., & Sahin, F. (2014) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2013.11.024 · Cited by 5,189 · see citing works
Building a Conceptual Framework: Philosophy, Definitions, and Procedure
Jabareen, Y. (2009) · International Journal of Qualitative Methods · doi:10.1177/160940690900800406 · Cited by 1,559 · see citing works
Weiskopf, N. G., & Weng, C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000681 · Cited by 1,216 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Big Data Quality Framework: A Holistic Approach to Continuous Quality Management
Taleb, I., Serhani, M. A., Bouhaddioui, C., & Dssouli, R. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00468-0 · Cited by 149 · see citing works
“In [21, 35, 42], many causes of poor data quality were enumerated, and a list of elements, which affect the quality and DQD’s was produced.”
BackgroundData Quality for Machine Learning Tasks
Gupta, N., Mujumdar, S., Patel, H., Masuda, S., Panwar, N., Bandyopadhyay, S., Mehta, S., Guttula, S., Afzal, S., Sharma Mittal, R., et al. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3470817 · Cited by 127 · see citing works
“We will discuss set of quality metrics which can affect the performance of two widely used tasks: (i) Classification tasks - Quality metrics that we will cover for classification tasks are (a) Class Overlap [8], (b) Outlier Detection, (c) Boundary Complexity [15], (d) Label Noise [17], and (e) Class Imbalance [16] and…”
MethodA Systematic Review of Data Quality in CPS and IoT for Industry 4.0
Goknil, A., Nguyen, P., Sen, S., Politaki, D., Niavis, H., Pedersen, K. J., Suyuthi, A., Anand, A., & Ziegenbein, A. (2023) · ACM Computing Surveys · doi:10.1145/3593043 · Cited by 68 · see citing works
“[9] M24 SameasM17butstatisticalpropertiesareusedtocomputeredundancy.”
BackgroundMethodInfluentialStructural Health Monitoring of Concrete Bridges Through Artificial Intelligence: A Narrative Review
Prakash, V., Debono, C. J., Musarat, M. A., Borg, R. P., Seychell, D., Ding, W., & Shu, J.-P. (2025) · Applied Sciences · doi:10.3390/app15094855 · Cited by 62 · see citing works
“The challenges in data quality issues can be solved using the following approaches [44]: • AI-based imputation methods, such as autoencoders or KNN imputation, may effectively manage missing or noisy data.”
Kleinekorte, J., Kleppich, J., Fleitmann, L., Beckert, V., Blodau, L., & Bardow, A. (2023) · ACS Sustainable Chemistry & Engineering · doi:10.1021/acssuschemeng.2c07682 · Cited by 58 · see citing works
Peplinski, M., Dilkina, B., Chen, M., Silva, S. J., Ban-Weiss, G., & Sanders, K. T. (2024) · Applied Energy · doi:10.1016/j.apenergy.2023.122413 · Cited by 44 · see citing works
Peivaste, I., Belouettar, S., Mercuri, F., Fantuzzi, N., Dehghani, H., Izadi, R., Ibrahim, H., Lengiewicz, J., Belouettar-Mathis, M., Bendine, K., et al. (2025) · Composite Structures · doi:10.1016/j.compstruct.2025.119419 · Cited by 39 · see citing works
Corrales, D. C., Ledezma, A., & Corrales, J. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106180 · Cited by 37 · see citing works
Self-citationMetcalfe, B., Acosta-Pavas, J. C., Robles-Rodríguez, C. E., Georgakilas, G., Dalamagas, T., Aceves-Lara, C., Daboussi, F., Koehorst, J., & Corrales, D. C. (2024) · Computers and Chemical Engineering · doi:10.1016/j.compchemeng.2024.108991 · Cited by 34 · see citing works
Self-citationFrom Theory to Practice: A Data Quality Framework for Classification Tasks
Corrales, D. C., Ledezma, A., & Corrales, J. C. (2018) · Symmetry · doi:10.3390/sym10070248 · Cited by 32 · see citing works
“• Build an integrated data quality framework for several knowledge discovery tasks as regression [37], clustering and association rules.”
BackgroundMethodSelf-citationEstimating Avocado Sales Using Machine Learning Algorithms and Weather Data
Rincon-Patino, J., Lasso, E., & Corrales, J. (2018) · Sustainability · doi:10.3390/su10103498 · Cited by 20
“Data cleaning process, adapted from [15].”
MethodInfluentialSelf-citationMindful Self-Compassion as an Antidote to Burnout for Mental Health Practitioners
Lyon, T. R., & Galbraith, A. (2023) · Healthcare · doi:10.3390/healthcare11202715 · Cited by 20 · see citing works
“When responses across a scale from a single respondent do not vary, it may indicate non-engagement in the survey, which can negatively impact the reliability or validity of the results [71].”
BackgroundEthical and Socially-Aware Data Labels
Beretta, E., Vetrò, A., Lepri, B., & Martin, J. C. D. (2018) · Symposium on Information Management and Big Data · doi:10.1007/978-3-030-11680-4_30 · Cited by 18 · see citing works
Developing machine-learning regression model with Logical Analysis of Data (LAD)
Khalifa, R., Yacout, S., & Bassetto, S. (2020) · Computers & industrial engineering · doi:10.1016/j.cie.2020.106947 · Cited by 16
Cerebral Palsy classification based on multi-feature analysis using machine learning
al-Sowi, A. M., Almasri, N., Hammo, B. H., & Al-Qwaqzeh, F. A.-Z. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101197 · Cited by 14
Content-Based Image Recognition and Tagging by Deep Learning Methods
Christy, A., & Dhanalakshmi, K. (2021) · Wireless personal communications · doi:10.1007/s11277-021-09159-8 · Cited by 14
Optimizing Lung Cancer Detection: The Synergy of Support Vector Machine and Random Forest
Pbv, R., Kumar, T. N. S. K. M., Praveen, S., Sindhura, S., Al-Dmour, N. A., & Islam, S. (2024) · 2024 International Conference on Decision Aid Sciences and Applications (DASA) · doi:10.1109/dasa63652.2024.10836455 · Cited by 10
Measuring Trustworthiness in Neuro-Symbolic Integration
Agiollo, A., & Omicini, A. (2023) · Annals of Computer Science and Information Systems · doi:10.15439/2023f6019 · Cited by 6 · see citing works
“…Metrics: Given the impact of data quality on the optimisation process of ML and DL systems, several quality metrics are available, namely: (i) class overlap [24], (ii) boundary complexity [25], (iii) label noise [26], (iv) class imbalance [27], (v) missing value analysis [28], and many more.”
BackgroundChen, C., Tan, Y., Liu, Y., Sun, M., Wang, Z. L., Zheng, X., Shen, Z., & Zhao, M. (2026) · Environmental Research · doi:10.1016/j.envres.2026.123853 · Cited by 8 · see citing works
Nishat, M. M., Ahsan, A., & Olsson, N. O. E. (2025) · Journal of Information Technology in Construction · doi:10.36680/j.itcon.2025.033 · Cited by 8 · see citing works
“According to Corrales et al., if an activity in the PM program is not linked to a calendar, it is viewed as poor planning or activity reporting (Corrales et al, 2018).”
Employing Various Data Cleaning Techniques to Achieve Better Data Quality using Python
Dasari, D., & Varma, P. (2022) · 2022 6th International Conference on Electronics, Communication and Aerospace Technology · doi:10.1109/iceca55336.2022.10009079 · Cited by 7
“After that, a description about data quality of each element is presented [5].”
BackgroundMethodAn exploratory analysis of regression methods for predicting faults in software systems
Rathore, S. S. (2021) · Soft Computing · doi:10.1007/s00500-021-06048-x · Cited by 6 · see citing works
“The experimental study is performed on 44 fault datasets, their versions collected from the PROMISE data repository (Cukic 2005; Cheikhi and Abran 2013), and absolute relative error (ARE), root mean square error (RSME), and fault-percentile-average (FPA) is used as the performance measures.”
MethodAUTO-CDD: automatic cleaning dirty data using machine learning techniques
H., J. M. Z., Hossen, A., Hossen, J., Raja, J., Thangavel, B., Sayeed, S., & K., T. (2019) · TELKOMNIKA (Telecommunication Computing Electronics and Control) · doi:10.12928/telkomnika.v17i4.12780 · Cited by 6
Performance Prediction using Neural Network and Confidence Intervals: a Gas Turbine application
Cisotto, S., & Herzallah, R. (2018) · 2018 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2018.8621919 · Cited by 5
Laasma, A., Aiken, D., Kasepõld, K., Hilmola, O., & Tapaninen, U. (2025) · Journal of Marine Science and Engineering · doi:10.3390/jmse13040688 · Cited by 4
“These methods rely on machine learning techniques to analyze ship energy performance [23,24].”
Quality matters: A decadal systematic exploration of data quality in IoT environment
Kaur, T., & Kaur, P. D. (2025) · Data & Knowledge Engineering · doi:10.1016/j.datak.2025.102533 · Cited by 4 · see citing works
Afriliana, N., Król, D., & Gaol, F. (2021) · International Conference on Conceptual Structures · doi:10.1007/978-3-030-77967-2_33 · Cited by 3 · see citing works
Testing for Data Quality Assessment: ACase Study from the Industry 4.0 Perspective
Król, D., & Czarnecki, T. (2021) · International Conference on Computational Collective Intelligence · doi:10.1007/978-3-030-88113-9_6 · Cited by 3 · see citing works
FramePests: A Comprehensive Framework for Crop Pests Modeling and Forecasting
Lasso, E., Motisi, N., Avelino, J., & Corrales, J. C. (2021) · IEEE Access · doi:10.1109/access.2021.3104537 · Cited by 8 · see citing works
“A framework for the data cleaning process can be consulted for regression [15] and classification [16] tasks.”
MethodSelf-citationZhu, C.-F., Babu, B. R., & Ramaiah, G. B. (2025) · Evolutionary Systematics · doi:10.1007/s12530-025-09723-8 · Cited by 2
“The scheduling time of the method in (Corrales et al. 2018a, b) is the next, and the scheduling time of different data amounts ranges from 3 to 5.6 min.”
InfluentialData Quality as a Service Framework for AI-Enabled Industrial Internet of Things
Goknil, A., Nguyen, P., Husom, E. J., Sen, S., Tverdal, S., Peysson, F., Politaki, D., & González-Velázquez, R. (2024) · IoT · doi:10.1145/3703790.3703794 · Cited by 2 · see citing works
“Notable contributions include methods for repairing data using sensory substitution [49], addressing data corruption through computational dependencies [31], and utilizing clustering techniques for outlier detection [11, 57].”
Thompson, Y. T., Li, Y., & Silovsky, J. F. (2023) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-023-09875-y · Cited by 2 · see citing works
Prediction of Downhole Pressure while Tripping
Mohammad, A., Karunakaran, S., Panchalingam, M., & Davidrajuh, R. (2022) · International Conference on Computational Intelligence and Communication Networks · doi:10.1109/cicn56167.2022.10008376 · Cited by 2
UAV Parameter Estimation Through Machine Learning
Fernandez, A. E., Everett, L., & Cedeno, M. (2021) · Journal of Strategic Innovation and Sustainability · doi:10.33423/jsis.v16i3.4447 · Cited by 2
“Filter performs feature reduction based on discriminating criteria independent of the Machine Learning algorithm (Corrales, Corrales, & Ledezma, 2018).”
MethodFramework for a Symmetric Integration Approach
Muntean, M., Brândaș, C., & Cîrstea, T. (2019) · Symmetry · doi:10.3390/sym11020224 · Cited by 2 · see citing works
“Corrales (2018) synthesized the diverse approaches regarding data quality issues [6].”
BackgroundAndreswari, R., Fauzi, R., Valensia, L., & Chanifah, S. (2022) · SHS Web of Conferences · doi:10.1051/shsconf/202213903018 · Cited by 2
Qarqour, A., Arora, S., Heisenberg, G., Rabe, M., & Kleinert, T. (2024) · International Conference on Knowledge Discovery and Information Retrieval · doi:10.5220/0012876200003838 · Cited by 1
“The choice of more trees for the RFR compared to the RFC reflects the increased complexity needed in regression models to capture data variability and nuances accurately (Corrales et al., 2018).”
Machine Learning Models for Emergency Department Layout Prediction Using Graph Metrics
Sarhan, O., Wahed, M. A., & Rushdi, M. A. (2026) · Egyptian Informatics Journal · doi:10.1016/j.eij.2026.100989
Xiao, M.-F., Gan, W.-X., Zhang, Y., & Zhao, L. (2025) · Scientific Research Bulletin · doi:10.71052/srb2024/eigq8039
Factors Influencing Water Point Functionality in Liberia: A Regression and Bayesian Network Analysis
Li, H., McManus, C., & Cronk, R. (2025) · Sustainability · doi:10.3390/su17198928
Nishchemenko, D., & Volochchuk, O. (2025) · Cybersecurity: Education, Science, Technique · doi:10.28925/2663-4023.2025.28.844
Flexible Techniques to Detect Typical Hidden Errors in Large Longitudinal Datasets
Bruni, R., Daraio, C. C., & Leo, S. D. (2024) · Symmetry · doi:10.3390/sym16050529
Microwave dielectric spectroscopy for determination of ethanol concentration in brandy
Hernández-Gómez, E., Olvera-Cervantes, J., & Pava-Chipol, N. (2022) · International Conference on Electrical Engineering, Computing Science and Automatic Control · doi:10.1109/cce56709.2022.9975987
“The sample or neighboring data is the one that the values of its variables are the closest to a specific data [19].”
BackgroundA Data Compacting Technique to reduce the NetFlow size in Botnet Detection with BotCluster
Wang, C.-Y., Fuh, S.-H., Lo, T.-C., Cheng, Q.-J., Chen, Y.-C., Cho, F.-M., Chang, J.-B., & Shieh, C. (2019) · BDCAT · doi:10.1145/3365109.3368778
“Most researches [2-15], aimed to reduce the size of the dataset and hold representative knowledge, can be cataloged into two types, such as dimensionality reduction and data quality adjustment.”
Background
- Authors
- Corrales, D., Corrales, J., & Ledezma, A.
- Affiliations
- Universidad del Cauca (CO)
- Universidad Carlos III de Madrid (ES)
- Corresponding author
- Corrales, David
- Venue
- Symmetry
- Publisher
- MDPI AG
- Published
- 2018-04-06
- Volume
- 10
- Issue
- 4
- Pages
- 99
- ISSN
- 2073-8994 (ISSN-L)
- OpenAlex
- W2795539379
- Cited by
- 58
- References
- 79
- Citations by year
- 2026: 2 · 2025: 11 · 2024: 6 · 2023: 9 · 2022: 8 · 2021: 9 · 2020: 4 · 2019: 5 · 2018: 4
- FWCI
- 4.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Symmetry (publishedVersion, cc-by)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- e-Archivo (Carlos III University of Madrid) (submittedVersion, cc-by-nc-nd)
- Funders
- Departamento Administrativo de Ciencia, Tecnología e Innovación (COLCIENCIAS)
- Universidad del Cauca
- Universidad Carlos III de Madrid (TRA2015-63708-R, TRA2016-78886-C3-1-R)
- Ministerio de Asuntos Económicos y Transformación Digital, Gobierno de España
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Keywords
- data cleaning, data quality, regression models, heterogeneous data sources, sensor data, social media data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
