The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Evaluating prediction systems in software project estimation
Open accessAbstract
Context: Software engineering has a problem in that when we empirically evaluate competing prediction systems we obtain conflicting results. Objective: To reduce the inconsistency amongst validation study results and provide a more formal foundation to interpret results with a particular focus on continuous prediction systems. Method: A new framework is proposed for evaluating competing prediction systems based upon (1) an unbiased statistic, Standardised Accuracy, (2) testing the result likelihood relative to the baseline technique of random 'predictions', that is guessing, and (3) calculation of effect sizes. Results: Previously published empirical evaluations of prediction systems are re-examined and the original conclusions shown to be unsafe. Additionally, even the strongest results are shown to have no more than a medium effect size relative to random guessing. Conclusions: Biased accuracy statistics such as MMRE are deprecated. By contrast this new empirical validation framework leads to meaningful results. Such steps will assist in performing future meta-analyses and in providing more robust and usable recommendations to practitioners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Another look at measures of forecast accuracy
Hyndman, R. J., & Koehler, A. B. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.001 · Cited by 5,757
Introduction to Lattices and Order
Davey, B. A., & Priestley, H. A. (2002) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809088 · Cited by 4,901
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537
Randomization, Bootstrap and Monte Carlo Methods in Biology
Manly, B. F. J. (2018) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315273075 · Cited by 4,194
Meta-analysis in social research
(1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90011-8 · Cited by 3,853
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
Forecasting methods and applications
(1993) · Computers & Operations Research · doi:10.1016/0305-0548(93)90040-p · Cited by 2,393
Experimentation in Software Engineering: An Introduction
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2011) · Lund University Publications (Lund University) · Cited by 2,214
Randomization, Bootstrap and Monte Carlo Methods in Biology.
Kemp, A. W., & Manly, B. F. J. (1997) · Biometrics · doi:10.2307/2533527 · Cited by 1,927
Parametric measures of effect size.
Rosenthal, R. (1994) · Cited by 1,847
A sequential algorithm for training text classifiers
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,731
Holder, R. L., Mosteller, F., & Tukey, J. W. (1979) · Applied Statistics · doi:10.2307/2346739 · Cited by 1,676
Osborn, J. F., Mosteller, F., & Tukey, J. W. (1978) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344493 · Cited by 1,446
Estimating software project effort using analogies
Shepperd, M., & Schofield, C. (1997) · IEEE Transactions on Software Engineering · doi:10.1109/32.637387 · Cited by 954
Self-citationThe case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931
A systematic review of software development cost estimation studies
Jørgensen, M., & Shepperd, M. (2007) · Brunel University Research Archive (BURA) (Brunel University London) · doi:10.1109/tse.2007.3 · Cited by 742
Self-citationA Systematic Review of Software Development Cost Estimation Studies
Jorgensen, M., & Shepperd, M. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256943 · Cited by 729
Self-citationExperimentation in software engineering: An introduction
(2000) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(00)90203-7 · Cited by 696
Forecasting: Methods and Applications
Briggs, W. M., Makridakis, S., Wheelwright, S. C., Hyndman, R. J., & Diebold, F. X. (1999) · Journal of the American Statistical Association · doi:10.2307/2669717 · Cited by 626
(2011) · Choice Reviews Online · doi:10.5860/choice.48-5742 · Cited by 626
A review of studies on expert estimation of software development effort
Jørgensen, M. (2004) · Journal of Systems and Software · doi:10.1016/s0164-1212(02)00156-5 · Cited by 618
A simulation study of the model evaluation criterion mmre
Foss, T., Stensrud, E., Kitchenham, B., & Myrtveit, I. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1245300 · Cited by 480
What accuracy statistics really measure
Kitchenham, B. A., Pickard, L. M., MacDonell, S. G., & Shepperd, M. J. (2001) · IEE Proceedings - Software · doi:10.1049/ip-sen:20010506 · Cited by 437
Self-citationCross versus Within-Company Cost Estimation Studies: A Systematic Review
Kitchenham, B. A., Mendes, E., & Travassos, G. H. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.1001 · Cited by 424
A General Software Defect-Proneness Prediction Framework
Song, Q., Jia, Z., Shepperd, M., Ying, S., & Liu, J. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.90 · Cited by 374
Self-citationA systematic review of effect size in software engineering experiments
Kampenes, V. B., Dybå, T., Hannay, J. E., & Sjøberg, D. I. K. (2007) · Information and Software Technology · doi:10.1016/j.infsof.2007.02.015 · Cited by 370
Experience with the accuracy of software maintenance task effort prediction models
Jorgensen, M. (1995) · IEEE Transactions on Software Engineering · doi:10.1109/32.403791 · Cited by 288
An assessment and comparison of common software cost estimation modeling techniques
Briand, L. C., El Emam, K., Surmann, D., Wieczorek, I., & Maxwell, K. D. (1999) · Proceedings of the 21st international conference on Software engineering · doi:10.1145/302405.302647 · Cited by 277
Reliability and validity in comparative studies of software prediction models
Myrtveit, I., Stensrud, E., & Shepperd, M. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.58 · Cited by 259
Self-citationForecasting: Methods and Applications, 3rd Ed
Makridakis, S., Wheelwright, S. C., & Hyndman, R. J. (1997) · Cited by 245
How large are software cost overruns? A review of the 1994 CHAOS report
Jørgensen, M., & Moløkken-Østvold, K. (2006) · Information and Software Technology · doi:10.1016/j.infsof.2005.07.002 · Cited by 227
Regression error characteristic curves
Bi, J., & Bennett, K. P. (2003) · Cited by 198
A controlled experiment to assess the benefits of estimating with analogy and regression models
Myrtveit, I., & Stensrud, E. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799947 · Cited by 183
Further Comparison of Cross-company and Within-company Effort Estimation Models for Web Applications
Mendes, E., & Kitchenham, B. (2004) · Cited by 133
Software effort estimation by analogy and “regression toward the mean”
Jørgensen, M., Indahl, U. G., & Sjøberg, D. I. K. (2003) · Journal of Systems and Software · doi:10.1016/s0164-1212(03)00066-9 · Cited by 127
Mair, C., & Shepperd, M. (2005) · International Symposium on Empirical Software Engineering · doi:10.1109/isese.2005.1541858 · Cited by 91
Self-citationSoftware project economics: a roadmap
Shepperd, M. (2007) · Future of Software Engineering (FOSE) · doi:10.1109/fose.2007.23 · Cited by 74
Self-citationShepperd, M., & Cartwright, M. (2005) · IEEE International Software Metrics Symposium (METRICS) · doi:10.1109/metrics.2005.5 · Cited by 22
Self-citationHuman judgement and software metrics
Mair, C., & Shepperd, M. (2011) · Workshop on Emerging Trends in Software Metrics · doi:10.1145/1985374.1985393 · Cited by 20
Self-citationFuture of Software Engineering @ ICSE 2023
Hu, X., & Dam, H. K. (2023) · IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00005 · Cited by 10
Cited by
An empirical analysis of data preprocessing for machine learning-based software cost estimation
Huang, J., Li, Y.-F., & Xie, M. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.07.004 · Cited by 223
Neural network models for software development effort estimation: a comparative study
Nassif, A. B., Azzeh, M., Capretz, L. F., & Ho, D. (2016) · Neural Computing and Applications · doi:10.1007/s00521-015-2127-1 · Cited by 122
On using grey literature and google scholar in systematic literature reviews in software engineering
Yasin, A., Fatima, R., Wen, L., Afzal, W., Azhar, M., & Torkar, R. (2020) · IEEE Access · doi:10.1109/access.2020.2971712 · Cited by 85
The Impact of Data Quality on Software Testing Effort Prediction
Radliński, Ł. (2023) · Electronics · doi:10.3390/electronics12071656 · Cited by 8
Metadata
- Authors
- Shepperd, M., & MacDonell, S.
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2012-08
- Volume
- 54
- Issue
- 8
- Pages
- 820-827
- ISSN
- 0950-5849 (ISSN-L)
- OpenAlex
- W2097670073
- Cited by
- 405
- References
- 44
- FWCI
- 58.54
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
