The Impact of Data Quality on Software Testing Effort Prediction
Open accessAbstract
Background: This paper investigates the impact of data quality on the performance of models predicting effort on software testing. Data quality was reflected by training data filtering strategies (data variants) covering combinations of Data Quality Rating, UFP Rating, and a threshold of valid cases. Methods: The experiment used the ISBSG dataset and 16 machine learning models. A process of three-fold cross-validation repeated 20 times was used to train and evaluate each model with each data variant. Model performance was assessed using absolute errors of prediction. A ‘win–tie–loss’ procedure, based on the Wilcoxon signed-rank test, was applied to identify the best models and data variants. Results: Most models, especially the most accurate, performed the best on a complete dataset, even though it contained cases with low data ratings. The detailed results include the rankings of the following: (1) models for particular data variants, (2) data variants for particular models, and (3) the best-performing combinations of models and data variants. Conclusions: Arbitrary and restrictive data selection to only projects with Data Quality Rating and UFP Rating of ‘A’ or ‘B’, commonly used in the literature, does not seem justified. It is recommended not to exclude cases with low data ratings to achieve better accuracy of most predictive models for testing effort prediction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Another dominant topic in the DL papers is completeness, which the ML community almost exclusively refers to as missing data.
“Another dominant topic is completeness to which the ML community almost exclusively refers to as missing data119,125–128,133–135,182.”
From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [127]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · see citing works
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,493 · see citing works
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957 · see citing works
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
Modern Applied Statistics with S
Venables, W., & Ripley, B. (2010) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 16,188 · see citing works
Pattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,446 · see citing works
An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression
Altman, N. (1992) · The American Statistician · doi:10.1080/00031305.1992.10475879 · Cited by 5,721 · see citing works
Symbolic Description of Factorial Models for Analysis of Variance
Wilkinson, G. N., & Rogers, C. E. (1973) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2346786 · Cited by 906 · see citing works
A Systematic Review of Software Development Cost Estimation Studies
Jørgensen, M., & Shepperd, M. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256943 · Cited by 736 · see citing works
Induction of model trees for predicting continuous classes
Wang, Y., & Witten, I. H. (1996) · Research Commons (University of Waikato) · Cited by 686 · see citing works
Systematic literature review of machine learning based software development effort estimation models
Wen, J., Li, S., Lin, Z., Hu, Y., & Huang, C. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.09.002 · Cited by 531 · see citing works
Evaluating prediction systems in software project estimation
Shepperd, M., & MacDonell, S. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2011.12.008 · Cited by 408 · see citing works
A review of automatic selection methods for machine learning algorithms and hyper-parameter values
Luo, G. (2016) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-016-0125-6 · Cited by 357 · see citing works
Ranking and Clustering Software Cost Estimation Models through a Multiple Comparisons Algorithm
Mittas, N., & Angelis, L. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.45 · Cited by 227 · see citing works
Exploiting the Essential Assumptions of Analogy-Based Effort Estimation
Kocagüneli, E., Menzies, T., Bener, A., & Keung, J. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.27 · Cited by 201 · see citing works
Why comparative effort prediction studies may be invalid
Kitchenham, B. A., & Mendes, E. (2009) · doi:10.1145/1540438.1540444 · Cited by 144 · see citing works
Mahmood, Y., Kama, N., Azmi, A., Khan, A. S., & Ali, M. (2021) · Software Practice and Experience · doi:10.1002/spe.3009 · Cited by 138 · see citing works
Data sets and data quality in software engineering
Liebchen, G., & Shepperd, M. (2008) · Predictive Models in Software Engineering · doi:10.1145/1370788.1370799 · Cited by 107 · see citing works
Kocagüneli, E., Menzies, T., Keung, J., Cok, D. R., & Madachy, R. J. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.88 · Cited by 98 · see citing works
A Baseline Model for Software Effort Estimation
Whigham, P. A., Owen, C. A., & MacDonell, S. G. (2015) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2738037 · Cited by 94 · see citing works
A systematic literature review of software effort prediction using machine learning methods
Ali, A., & Gravino, C. (2019) · Journal of Software Evolution and Process · doi:10.1002/smr.2211 · Cited by 91 · see citing works
López‐Martín, C. (2014) · Applied Soft Computing · doi:10.1016/j.asoc.2014.10.033 · Cited by 70 · see citing works
Mendes, E., Lokan, C., Harrison, R., & Triggs, C. M. (2005) · IEEE International Software Metrics Symposium (METRICS) · doi:10.1109/metrics.2005.4 · Cited by 65 · see citing works
Linear Programming as a Baseline for Software Effort Estimation
Sarro, F., & Petrozziello, A. (2018) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3234940 · Cited by 55 · see citing works
Software development effort estimation: a systematic mapping study
Carbonera, C. E., Farias, K., & Bischoff, V. (2020) · IET Software · doi:10.1049/iet-sen.2018.5334 · Cited by 54 · see citing works
Replicating studies on cross- vs single-company effort models using the ISBSG Database
Mendes, E., & Lokan, C. (2007) · Empirical Software Engineering · doi:10.1007/s10664-007-9045-5 · Cited by 53 · see citing works
Stable rankings for different effort models
Menzies, T., Jalali, O., Hihn, J., Baker, D., & Lum, K. (2010) · Automated Software Engineering · doi:10.1007/s10515-010-0070-z · Cited by 51 · see citing works
On the value of outlier elimination on software effort estimation research
Seo, Y., & Bae, D. (2012) · Empirical Software Engineering · doi:10.1007/s10664-012-9207-y · Cited by 47 · see citing works
Minku, L. L., & Yao, X. (2013) · doi:10.1145/2499393.2499396 · Cited by 42 · see citing works
Fernández-Diego, M., & González-Ladrón-de-Guevara, F. (2014) · Information and Software Technology · doi:10.1016/j.infsof.2014.01.003 · Cited by 40 · see citing works
A Taxonomy of Data Quality Challenges in Empirical Software Engineering
Bosu, M. F., & MacDonell, S. G. (2013) · Australasian Software Engineering Conference (ASWEC) · doi:10.1109/aswec.2013.21 · Cited by 39 · see citing works
Minku, L. L. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09686-w · Cited by 38 · see citing works
A genetic algorithm based framework for software effort prediction
Murillo-Morera, J., Quesada–López, C., Castro-Herrera, C., & Jenkins, M. (2017) · Journal of Software Engineering Research and Development · doi:10.1186/s40411-017-0037-x · Cited by 38 · see citing works
Bosu, M. F., & Macdonell, S. G. (2019) · Journal of Data and Information Quality · doi:10.1145/3328746 · Cited by 36 · see citing works
Comparative study of random search hyper-parameter tuning for software effort estimation
Villalobos-Arias, L., & Quesada–López, C. (2021) · doi:10.1145/3475960.3475986 · Cited by 30 · see citing works
Machine learning techniques for software testing effort prediction
López‐Martín, C. (2021) · Software Quality Journal · doi:10.1007/s11219-020-09545-8 · Cited by 27 · see citing works
On the Relationship between Different Size Measures in the Software Life Cycle
Gencel, Ç., Heldal, R., & Lind, K. E. (2009) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2009.51 · Cited by 27 · see citing works
Sequential Model Optimization for Software Effort Estimation
Xia, T., Shu, R., Shen, X., & Menzies, T. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3047072 · Cited by 25 · see citing works
Software Testing Effort Estimation and Related Problems
Bluemke, I., & Malanowska, A. (2021) · ACM Computing Surveys · doi:10.1145/3442694 · Cited by 22 · see citing works
Gautam, S. S., & Singh, V. (2022) · IEEE Access · doi:10.1109/access.2022.3200149 · Cited by 18 · see citing works
Evaluating the Quality of Datasets in Software Engineering
Rosli, M. M., Tempero, E. D., & Luxton-Reilly, A. (2018) · Advanced Science Letters · doi:10.1166/asl.2018.12920 · Cited by 16 · see citing works
Effort and Cost in Software Engineering
Huijgens, H., Arie Van Deursen, A., Minku, L. L., & Lokan, C. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084249 · Cited by 15 · see citing works
Shepperd, M. (2011) · Workshop on Emerging Trends in Software Metrics · doi:10.1145/1985374.1985376 · Cited by 14 · see citing works
Influence of Outliers on Estimation Accuracy of Software Development Effort
Kenichi, O. N. O., Tsunoda, M., Monden, A., & Matsumoto, K. (2020) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2020mpp0005 · Cited by 12 · see citing works
Fernández-Diego, M., Martínez-Gómez, M., & Torralba-Martínez, J.-M. (2010) · doi:10.1145/1868328.1868348 · Cited by 11 · see citing works
Application of mutual information-based sequential feature selection to ISBSG mixed data
Fernández-Diego, M., & González-Ladrón-de-Guevara, F. (2017) · Software Quality Journal · doi:10.1007/s11219-017-9391-5 · Cited by 8 · see citing works
Stability of user satisfaction prediction in software projects
Radliński, Ł. (2020) · Procedia Computer Science · doi:10.1016/j.procs.2020.09.308 · Cited by 7 · see citing works
Self-citationPrediction of software quality with Machine Learning-Based ensemble methods
Ceran, A., Ar, Y., Tanrıöver, Ö. Ö., & Ceran, S. S. (2022) · Materials Today Proceedings · doi:10.1016/j.matpr.2022.11.229 · Cited by 6 · see citing works
Predicting User Satisfaction in Software Projects using Machine Learning Techniques
Radliński, Ł. (2020) · International Conference on Evaluation of Novel Approaches to Software Engineering · doi:10.5220/0009391803740381 · Cited by 5 · see citing works
Self-citationPredicting Aggregated User Satisfaction in Software Projects
Radliński, Ł. (2018) · Foundations of Computing and Decision Sciences · doi:10.1515/fcds-2018-0017 · Cited by 4 · see citing works
Self-citationRadliński, Ł. (2015) · Applied Computer Science · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Radli ński, Ł. Predicting User Satisfaction in Software Projects using Machine Learning Techniques. In Proceedings of the 15th International Conference on Evaluation of Novel Approaches to Software Engineering, Prague, Czech Republic, 5-6 May 2020;
Cited by
Sorted by citations
The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01196-4 · Cited by 169 · see citing works
Sánchez-García, Á. J., López‐Martín, C., & Abran, A. (2023) · IEEE Access · doi:10.1109/access.2023.3337809 · Cited by 11 · see citing works
Radliński, Ł., & Swacha, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112618 · Cited by 3 · see citing works
Self-citationThe Trade-off Between Data Volume and Quality in Predicting User Satisfaction in Software Projects
Radliński, Ł. (2024) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa64295.2024.00080 · Cited by 1 · see citing works
Self-citationComparison of data set sample selection algorithms for data science: a systematic review
Fernandez-Sanchez, A., Gestal, M., Bolón‐Canedo, V., de la Calle, J. D., & Sierra, A. P. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12124-w · Cited by 1 · see citing works
Sakhrawi, Z., Labidi, T., Sellami, A., & Bouassida, N. (2025) · Cluster Computing · doi:10.1007/s10586-025-05384-z
A Method for Intelligent Optimization Algorithms to Automatically Generate LLM Test Data
Xie, W., Qian, J., Li, Y., Huang, J., Zheng, Y., & Liu, Y. (2025) · Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS) · doi:10.1109/acctcs66275.2025.00047
Radliński, Ł. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02898-9
Self-citation
- Authors
- Radliński, Ł.
- Affiliations
- West Pomeranian University of Technology in Szczecin (PL)
- Corresponding author
- Radliński, Łukasz
- ORCID
- Radliński, Łukasz
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2023-03-31
- Volume
- 12
- Issue
- 7
- Pages
- 1656
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W4362475637
- Cited by
- 8
- References
- 55
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 2 · 2023: 1
- FWCI
- 1.88
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- data quality, training data filtering, ISBSG dataset, model performance comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
