Testing scientific software: A systematic literature review
Open accessAbstract
Context Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code.Objective This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software.Method We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software.Results We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them.Conclusions Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
One of the challenges to be addressed is the identification of metamorphic relations that should be satisfied by an ML system.
“One of the challenges to be addressed is the identification of metamorphic relations that should be satisfied by an ML system (Kanewala and Bieman, 2014).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kanewala and Bieman, 2014
As in testing scientific software, perfect oracles may not be available for ML systems.
“As in testing scientific software (Kanewala and Bieman, 2014), perfect oracles may not be available for ML systems.”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kanewala and Bieman, 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2004) · Cited by 5,198 · see citing works
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · see citing works
Introduction to Software Testing
Bierig, R., Brown, S., Galván, E., & Timoney, J. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108974073.004 · Cited by 1,056 · see citing works
On testing non-testable programs
Weyuker, E. J. (1982) · The Computer Journal · doi:10.1093/comjnl/25.4.465 · Cited by 606 · see citing works
A systematic review of search-based testing for non-functional system properties
Afzal, W., Torkar, R., & FELDT, R. H. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.12.005 · Cited by 447 · see citing works
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 394 · see citing works
Applying Systematic Reviews to Diverse Study Types: An Experience Report
Dyba, T., Dingsoyr, T., & Hanssen, G. K. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.59 · Cited by 391 · see citing works
A Visual Text Mining approach for Systematic Reviews
Malheiros, V., Höhn, E. N., de Pinho, R., Mendonça, M., & Maldonado, J. C. (2007) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2007.21 · Cited by 334 · see citing works
A systematic review on regression test selection techniques
Engström, E., Runeson, P., & Skoglund, M. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.07.001 · Cited by 312 · see citing works
How do scientists develop and use scientific software?
Hannay, J. E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., & Wilson, G. (2009) · doi:10.1109/secse.2009.5069155 · Cited by 283 · see citing works
A Scientist's Nightmare: Software Problem Leads to Five Retractions
Miller, G. (2006) · Science · doi:10.1126/science.314.5807.1856 · Cited by 205 · see citing works
A systematic literature review to identify and classify software requirement errors
Walia, G., & Carver, J. C. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.01.004 · Cited by 194 · see citing works
Software Development Environments for Scientific and Engineering Software: A Series of Case Studies
Carver, J. C., Kendall, R. P., Squires, S., & Post, D. E. (2007) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2007.77 · Cited by 167 · see citing works
Analysis of faults in an N-version software experiment
Brilliant, S. S., KNIGHT, J. C., & Leveson, N. G. (1990) · IEEE Transactions on Software Engineering · doi:10.1109/32.44387 · Cited by 165 · see citing works
Proceedings of the 30th international conference on Software engineering
(2008) · doi:10.1145/1368088 · Cited by 163 · see citing works
An innovative approach for testing bioinformatics programs using metamorphic testing
Chen, T. Y., Ho, J. W. K., Liu, H., & Xie, X. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-24 · Cited by 159 · see citing works
Properties of Machine Learning Applications for Use in Metamorphic Testing
Murphy, C., Kaiser, G. E., & Hu, L. (2008) · doi:10.7916/d8xk8pfd · Cited by 155 · see citing works
Engineering the Software for Understanding Climate Change
Easterbrook, S., & Johns, T. C. (2009) · Computing in Science & Engineering · doi:10.1109/mcse.2009.193 · Cited by 141 · see citing works
How accurate is scientific software?
Hatton, L., & Roberts, A. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.328993 · Cited by 140 · see citing works
When Software Engineers Met Research Scientists: A Case Study
Segal, J. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3865-y · Cited by 117 · see citing works
Metamorphic testing of programs on partial differential equations: a case study
Chen, T. Y., Feng, J., & Tse, T. H. (2003) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.2002.1045022 · Cited by 113 · see citing works
The T experiments: errors in scientific software
Hatton, L. (1997) · IEEE Computational Science and Engineering · doi:10.1109/99.609829 · Cited by 110 · see citing works
Dealing with Risk in Scientific Software Development
Sanders, R., & Kelly, D. F. (2008) · IEEE Software · doi:10.1109/ms.2008.84 · Cited by 104 · see citing works
Pseudo-oracles for non-testable programs
Davis, M. D., & Weyuker, E. J. (1981) · ACM Annual Conference · doi:10.1145/800175.809889 · Cited by 98 · see citing works
Using machine learning techniques to detect metamorphic relations for programs without test oracles
Kanewala, U., & Bieman, J. M. (2013) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2013.6698899 · Cited by 95 · see citing works
Self-citationSome Problems of Professional End User Developers
Segal, J. (2007) · Proceedings/Proceedings -- IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2007.17 · Cited by 87 · see citing works
Girls Teaching Girls: Free-Choice Collaborative Learning Through Social Computing
Peters, M. R. (2007) · Proceedings/Proceedings -- IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2007.50 · Cited by 80 · see citing works
Exploring xp for scientific research
Wood, W. A., & Kleb, W. L. (2003) · IEEE Software · doi:10.1109/ms.2003.1196317 · Cited by 77 · see citing works
Margolis, H. (1996) · Cited by 77 · see citing works
Chaste: using agile programming techniques to develop computational biology software
Pitt‐Francis, J., Bernabéu, M. O., Cooper, J., Garny, A., Momtahan, L., Osborne, J. M., Pathmanathan, P., Rodríguez, B., Whiteley, J., & Gavaghan, D. J. (2008) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2008.0096 · Cited by 75 · see citing works
Post, D. E., & Kendall, R. P. (2004) · The International Journal of High Performance Computing Applications · doi:10.1177/1094342004048534 · Cited by 74 · see citing works
A survey of scientific software development
Nguyen-Hoan, L., Flint, S. R., & Sankaranarayana, R. S. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852802 · Cited by 72 · see citing works
Software Testing and Verification in Climate Model Development
Clune, T. L., & Rood, R. B. (2011) · IEEE Software · doi:10.1109/ms.2011.117 · Cited by 68 · see citing works
Murphy, C., Shen, K., & Kaiser, G. E. (2009) · International Conference on Software Testing, Verification, and Validation · doi:10.1109/icst.2009.19 · Cited by 67 · see citing works
Agile methods in biomedical software development: a multi-site experience report
Kane, D. W., Hohman, M. M., Cerami, E. G., McCormick, M. W., Kuhlmman, K. F., & Byrd, J. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-273 · Cited by 66 · see citing works
Guderlei, R., & Mayer, J. (2007) · Proceedings of the ... International Conference on Quality Software/Proceedings · doi:10.1109/qsic.2007.4385527 · Cited by 66 · see citing works
Introduction to software testing
(2008) · Choice Reviews Online · doi:10.5860/choice.45-6834 · Cited by 63 · see citing works
An Approach to Software Testing of Machine Learning Applications
Murphy, C., Kaiser, G. E., & Arias, M. (2007) · doi:10.7916/d8r49znr · Cited by 60 · see citing works
Segal, J. (2009) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-009-9096-9 · Cited by 60 · see citing works
On Random Testing of Image Processing Applications
Mayer, J., & Guderlei, R. (2006) · Proceedings of the ... International Conference on Quality Software/Proceedings · doi:10.1109/qsic.2006.45 · Cited by 49 · see citing works
Testing for trustworthiness in scientific software
Hook, D., & Kelly, D. F. (2009) · doi:10.1109/secse.2009.5069163 · Cited by 48 · see citing works
What Do We Know about Scientific Software Development's Agile Practices?
Sletholt, M. T., Hannay, J. E., Pfahl, D., & Langtangen, H. P. (2011) · Computing in Science & Engineering · doi:10.1109/mcse.2011.113 · Cited by 47 · see citing works
Models of scientific software development
Segal, J. (2008) · Open Research Online (The Open University) · Cited by 46 · see citing works
Easterbrook, S. (2010) · FSE/SDP Workshop on Future of Software Engineering Research (FoSER) · doi:10.1145/1882362.1882383 · Cited by 44 · see citing works
Overview of the Software Design of the Community Climate System Model
Drake, J. B., Jones, P. W., & Carr, G. R. (2005) · The International Journal of High Performance Computing Applications · doi:10.1177/1094342005056094 · Cited by 43 · see citing works
A theory-driven testing methodology for developing scientific software
Lane, P. C. R., & Gobet, F. R. (2012) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2012.695443 · Cited by 41 · see citing works
Symbolic Path-Oriented Test Data Generation for Floating-Point Programs
Bagnara, R., Carlier, M., Gori, R., et al. (2013) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2013.17 · Cited by 39 · see citing works
Scientists and software engineers: A tale of two cultures
Segal, J. (2008) · Open Research Online (The Open University) · Cited by 37 · see citing works
A Learning-Based Approach to Unit Testing of Numerical Software
Meinke, K., & Niu, F. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-16573-3_16 · Cited by 36 · see citing works
Design and use of reference data sets for testing scientific software
Cox, M. G., & Harris, P. M. (1999) · Analytica Chimica Acta · doi:10.1016/s0003-2670(98)00481-4 · Cited by 35 · see citing works
Barely sufficient software engineering: 10 practices to improve your CSE software
Heroux, M. A., & Willenbring, J. (2009) · doi:10.1109/secse.2009.5069157 · Cited by 35 · see citing works
The ASC-Alliance Projects: A Case Study of Large-Scale Parallel Scientific Code Development
Hochstein, L., & Basili, V. R. (2008) · Computer · doi:10.1109/mc.2008.101 · Cited by 35 · see citing works
On effective testing of health care simulation software
Murphy, C., Raunak, M. S., King, A., et al. (2011) · doi:10.1145/1987993.1988003 · Cited by 34 · see citing works
Scientific Software Development at a Research Facility
Ackroyd, K. S., Kinder, S. H., Mant, G. R., et al. (2008) · IEEE Software · doi:10.1109/ms.2008.93 · Cited by 34 · see citing works
Assessing climate model software quality: a defect density analysis of three models
Pipitone, J., & Easterbrook, S. (2012) · Geoscientific model development · doi:10.5194/gmd-5-1009-2012 · Cited by 34 · see citing works
Automated continuous verification for numerical simulation
Farrell, P. E., Piggott, M. D., Gorman, G., et al. (2011) · Geoscientific model development · doi:10.5194/gmd-4-435-2011 · Cited by 34 · see citing works
Some challenges facing software engineers developing software for scientists
Segal, J. (2009) · doi:10.1109/secse.2009.5069156 · Cited by 32 · see citing works
Software Engineering for Scientists
Kelly, D. F., Smith, S., & Meng, N. J. (2011) · Computing in Science & Engineering · doi:10.1109/mcse.2011.86 · Cited by 31 · see citing works
Five Recommended Practices for Computational Scientists Who Write Software
Kelly, D. F., Hook, D., & Sanders, R. (2009) · Computing in Science & Engineering · doi:10.1109/mcse.2009.139 · Cited by 30 · see citing works
Metamorphic Testing of Stochastic Optimisation
Yoo, S. (2010) · International Conference on Software Testing, Verification and Validation Workshops · doi:10.1109/icstw.2010.26 · Cited by 30 · see citing works
Scientific Software Testing: Analysis with Four Dimensions
Kelly, D. F., Thorsteinson, S., & Hook, D. (2010) · IEEE Software · doi:10.1109/ms.2010.88 · Cited by 29 · see citing works
Grid-Based Problem Solving Environments
Gaffney, P. W., & Pool, J. C. T. (2007) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-0-387-73659-4 · Cited by 27 · see citing works
On testing image processing applications with statistical methods
Mayer, J. (2005) · Cited by 25 · see citing works
Component‐Based Software Engineering
Cao, F., Gray, J., & Bryant, B. R. (2007) · Wiley Encyclopedia of Computer Science and Engineering · doi:10.1002/9780470050118.ecse066 · Cited by 24 · see citing works
Parameterizing random test data according to equivalence classes
Murphy, C., Kaiser, G. E., & Arias, M. (2007) · doi:10.1145/1292414.1292425 · Cited by 20 · see citing works
Modeling Input Space for Testing Scientific Computational Software: A Case Study
Vilkomir, S., Swain, W. T., Poore, J. H., & Clarno, K. T. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69389-5_34 · Cited by 20 · see citing works
Improving scientific software component quality through assertions
Dahlgren, T. L., & Dévanbu, P. (2005) · doi:10.1145/1145319.1145341 · Cited by 19 · see citing works
Some Challenges Facing Scientific Software Developers: The Case of Molecular Biology
Morris, C. A., & Segal, J. (2009) · IEEE International Conference on eScience · doi:10.1109/e-science.2009.38 · Cited by 18 · see citing works
Examining random and designed tests to detect code mistakes in scientific software
Kelly, D. F., Gray, R., & Shao, Y. (2010) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.002 · Cited by 17 · see citing works
Automated Software Testing for Matlab
Eddins, S. L. (2009) · Computing in Science & Engineering · doi:10.1109/mcse.2009.186 · Cited by 16 · see citing works
System Testing a Scientific Framework Using a Regression-Test Environment
Remmel, H., Paech, B., Bastian, P., et al. (2011) · Computing in Science & Engineering · doi:10.1109/mcse.2011.115 · Cited by 15 · see citing works
Dubois, P. F. (2012) · Computing in Science & Engineering · doi:10.1109/mcse.2012.84 · Cited by 14 · see citing works
A case study on testing CMM uncertainty simulation software (VCMM)
Abackerli, A. J., Pereira, P. H. C., & Calônego, N. (2010) · Journal of the Brazilian Society of Mechanical Sciences and Engineering · doi:10.1590/s1678-58782010000100002 · Cited by 12 · see citing works
Performance-Driven Interface Contract Enforcement for Scientific Components
Dahlgren, T. L. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73551-9_11 · Cited by 12 · see citing works
Specification-guided Golden Run for Analysis of Robustness Testing Results
Lemos, G. S., & Martins, E. (2012) · doi:10.1109/sere.2012.28 · Cited by 9 · see citing works
The need for a rigorous development and testing methodology for medical software
Reupke, W. A., Srinivasan, E., Rigterink, P. V., et al. (2003) · doi:10.1109/ecbs.1988.5439 · Cited by 9 · see citing works
Improving the Development Process for CSE Software
Heroux, M. A., Willenbring, J., & Phenow, M. N. (2007) · Proceedings - Euromicro Workshop on Parallel and Distributed Processing/Proceedings · doi:10.1109/pdp.2007.51 · Cited by 7 · see citing works
Proceedings of the ACM '82 conference on - ACM 82
(1982) · doi:10.1145/800174 · Cited by 7 · see citing works
Proceedings of the 7th international conference on Computational Science, Part III: ICCS 2007
Shi, Y., van Albada, G. D., Dongarra, J. J., et al. (2007) · Cited by 5 · see citing works
A Test Harness TH for Numerical Applications and Libraries
Smith, B. T. (2007) · Springer eBooks · doi:10.1007/978-0-387-73659-4_13 · Cited by 4 · see citing works
Creating a flexible environment for testing scientific software
Smith, M. C., Kelsey, R. L., Riese, J. M., et al. (2004) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.542345 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [35]Kanewala U Bieman JM Techniques for testing scientific programs without an oracle 2013 Proc. 5th International Workshop on Software Engineering for Computational Science and Engineering. IEEE 48 57
- [38]Kelly D Sanders R Assessing the quality of scientific software 2008 First International Workshop on Software Engineering for Computational Science and Engineering
- [52]Morris C Some lessons learned reviewing scientific code Proc 2008 First International Workshop on Software Engineering for Computational Science and Engineering
- [65]Sanders R Kelly D The challenge of testing scientific software 2008 7 Proceedings Conference for the Association for Software Testing (CAST) 30 36
Cited by
Sorted by citations
Towards FAIR Principles for Research Software
Lamprecht, A.-L., Garcia, L., Kuzak, M., Martinez, C., Arcila, R., Martin Del Pico, E., Dominguez Del Angel, V., van de Sandt, S., Ison, J., Martinez, P. A., et al. (2020) · Data Science · doi:10.3233/ds-190026 · Cited by 322 · see citing works
The ML test score: A rubric for ML production readiness and technical debt reduction
Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8258038 · Cited by 310 · see citing works
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
When and what to automate in software testing? A multi-vocal literature review
Garousi, V., & Mäntylä, M. V. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.015 · Cited by 182 · see citing works
Threats to scientific software from over-reliance on AI code assistants
O’Brien, G. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00845-2 · Cited by 6 · see citing works
- Authors
- Kanewala, U., & Bieman, J. M.
- Affiliations
- Colorado State University (US)
- Corresponding author
- Kanewala, Upulee
- ORCID
- Kanewala, Upulee
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2014-10
- Volume
- 56
- Issue
- 10
- Pages
- 1219-1232
- ISSN
- 0950-5849 (ISSN-L)
- PMC
- PMC4128280
- OpenAlex
- W2017425457
- Cited by
- 150
- References
- 81
- Citations by year
- 2026: 15 · 2025: 10 · 2024: 10 · 2023: 16 · 2022: 13 · 2021: 14 · 2020: 13 · 2019: 21 · 2018: 12 · 2017: 10 · 2016: 11 · 2015: 5
- FWCI
- 3.79
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Information and Software Technology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health
- National Institute of General Medical Sciences (R01GM096192)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- research software, software testing, oracle problem, software defects, publication retraction, code clone detection, climate models, software engineering challenges
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
