The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Reliability and validity in comparative studies of software prediction models
Open accessAbstract
Empirical studies on software prediction models do not converge with respect to the question "which prediction model is best?" The reason for this lack of convergence is poorly understood. In this simulation study, we have examined a frequently used research procedure comprising three main ingredients: a single data sample, an accuracy indicator, and cross validation. Typically, these empirical studies compare a machine learning model with a regression model. In our study, we use simulation and compare a machine learning and a regression model. The results suggest that it is the research procedure itself that is unreliable. This lack of reliability may strongly contribute to the lack of convergence. Our findings thus cast some doubt on the conclusions of any study of competing software prediction models that used this research procedure as a basis of model comparison. Thus, we need to develop more reliable research procedures before we can have confidence in the conclusions of comparative studies of software prediction models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reliability and Validity Assessment
Carmines, E., & Zeller, R. (1979) · doi:10.4135/9781412985642 · Cited by 5,884
Reliability and Validity Assessment
(2011) · The Encyclopedia of Political Science · doi:10.4135/9781608712434.n1341 · Cited by 5,040
A leisurely look at the bootstrap, the jackknife, and cross-validation
Efron, B., & Gong, G. (1983) · The American Statistician · doi:10.1080/00031305.1983.10483087 · Cited by 3,229
Machine Learning, Neural and Statistical Classification
Fulkerson, B. (1995) · Technometrics · doi:10.1080/00401706.1995.10484383 · Cited by 2,209
Software engineering metrics and models
Conte, S. D., Dunsmore, H. E., & Shen, V. Y. (1986) · Benjamin-Cummings Publishing Co., Inc. eBooks · Cited by 1,062
Estimating software project effort using analogies
Shepperd, M., & Schofield, C. (1997) · IEEE Transactions on Software Engineering · doi:10.1109/32.637387 · Cited by 954
Self-citationEncyclopedia of Statistical Sciences
(2004) · Wiley eBooks · doi:10.1002/0471667196 · Cited by 724
Software engineering metrics and models
Vanamstel, J. J. (1987) · European Journal of Operational Research · doi:10.1016/0377-2217(87)90230-x · Cited by 520
Machine learning approaches to estimating software development effort
Srinivasan, K., & Fisher, D. (1995) · IEEE Transactions on Software Engineering · doi:10.1109/32.345828 · Cited by 483
A simulation study of the model evaluation criterion mmre
Foss, T., Stensrud, E., Kitchenham, B., & Myrtveit, I. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1245300 · Cited by 480
Self-citationWhat accuracy statistics really measure
Kitchenham, B. A., Pickard, L. M., MacDonell, S. G., & Shepperd, M. J. (2001) · IEE Proceedings - Software · doi:10.1049/ip-sen:20010506 · Cited by 437
Self-citationExperience with the accuracy of software maintenance task effort prediction models
Jorgensen, M. (1995) · IEEE Transactions on Software Engineering · doi:10.1109/32.403791 · Cited by 288
Comparing software prediction techniques using simulation
Shepperd, M., & Kadoda, G. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.965341 · Cited by 280
Self-citationAn assessment and comparison of common software cost estimation modeling techniques
Briand, L. C., El Emam, K., Surmann, D., Wieczorek, I., & Maxwell, K. D. (1999) · Proceedings of the 21st international conference on Software engineering · doi:10.1145/302405.302647 · Cited by 277
An investigation of machine learning based prediction systems
Mair, C., Kadoda, G., Lefley, M., Phalp, K., Schofield, C., Shepperd, M., & Webster, S. (2000) · Journal of Systems and Software · doi:10.1016/s0164-1212(00)00005-4 · Cited by 270
Self-citationSoftware cost estimation with incomplete data
Strike, K., El Emam, K., & Madhavji, N. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.962560 · Cited by 252
Examining the Feasibility of a Case-Based Reasoning Model for Software Effort Estimation1
Mukhopadhyay, T., Vicinanza, S. S., & Prietula, M. J. (1992) · MIS Quarterly · doi:10.2307/249573 · Cited by 241
A pattern recognition approach for software engineering data analysis
Briand, L. C., Basili, V. R., & Thomas, W. M. (1992) · IEEE Transactions on Software Engineering · doi:10.1109/32.177363 · Cited by 239
An Empirical Study of Analogy-based Software Effort Estimation
Walkerden, F., & Jeffery, R. (1999) · Empirical Software Engineering · doi:10.1023/a:1009872202035 · Cited by 237
A replicated assessment and comparison of common software cost modeling techniques
Briand, L. C., Langley, T., & Wieczorek, I. (2000) · Proceedings of the 22nd international conference on Software engineering - ICSE '00 · doi:10.1145/337180.337223 · Cited by 220
Using public domain metrics to estimate software development effort
Jeffery, R., Ruhe, M., & Wieczorek, I. (2001) · Proceedings Seventh International Software Metrics Symposium · doi:10.1109/metric.2001.915512 · Cited by 186
On the problem of the software cost function
Dolado, J. J. (2001) · Information and Software Technology · doi:10.1016/s0950-5849(00)00137-3 · Cited by 185
A controlled experiment to assess the benefits of estimating with analogy and regression models
Myrtveit, I., & Stensrud, E. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799947 · Cited by 183
Self-citationBriand, L. C., Brasili, V. R., & Hetmanski, C. J. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.256851 · Cited by 179
Robust regression for developing software estimation models
Miyazaki, Y., Terakado, M., Ozaki, K., & Nozaki, H. (1994) · Journal of Systems and Software · doi:10.1016/0164-1212(94)90110-4 · Cited by 178
Resource Estimation in Software Engineering
Briand, L., & Wieczorek, I. (2002) · Wiley eBooks · doi:10.1002/0471028959.sof282 · Cited by 149
A procedure for analyzing unbalanced datasets
Kitchenham, B. A. (1998) · IEEE Transactions on Software Engineering · doi:10.1109/32.677185 · Cited by 117
Shepperd, M., & Cartwright, M. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.965339 · Cited by 110
Self-citationIdentifying high performance ERP projects
Stensrud, E., & Myrtveit, I. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1199070 · Cited by 98
Self-citationInter-item correlations among function points
Kitchenham, B. A., & Känsälä, K. (1993) · doi:10.5555/257572.257677 · Cited by 77
Effort estimation and prediction of object-oriented systems
Nesi, P., & Querci, T. (1998) · Journal of Systems and Software · doi:10.1016/s0164-1212(97)10021-8 · Cited by 70
An investigation of analysis techniques for software datasets
Pickard, L. M., Kitchenham, B. A., & Linkman, S. (2003) · IEEE International Software Metrics Symposium (METRICS) · doi:10.1109/metric.1999.809734 · Cited by 67
Software cost estimation using an Albus perceptron (CMAC)
Samson, B., Ellison, D., & Dugard, P. (1997) · Information and Software Technology · doi:10.1016/0950-5849(96)01124-x · Cited by 66
A replicated assessment and comparison of common software cost modeling techniques
Briand, L. C., Langley, T., & Wieczorek, I. (2002) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2000.870428 · Cited by 45
The question of scale economies in software—why cannot researchers agree?
Kitchenham, B. A. (2002) · Information and Software Technology · doi:10.1016/s0950-5849(01)00204-x · Cited by 43
Making software cost data available for meta-analysis
Mair, C. (2004) · doi:10.1049/ic:20040397 · Cited by 3
Cited by
Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
Evaluating defect prediction approaches: a benchmark and an extensive comparison
D’Ambros, M., Lanza, M., & Robbes, R. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9173-9 · Cited by 626
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574
Systematic literature review of machine learning based software development effort estimation models
Wen, J., Li, S., Lin, Z., Hu, Y., & Huang, C. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2011.09.002 · Cited by 526
Metadata
- Authors
- Myrtveit, I., Stensrud, E., & Shepperd, M.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2005-05
- Volume
- 31
- Issue
- 5
- Pages
- 380-391
- ISSN
- 0098-5589 (ISSN-L)
- OpenAlex
- W2123279272
- Cited by
- 259
- References
- 36
- FWCI
- 39.65
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.