The DOI matches its Crossref record; 109 of the 109 references it cites were found in a registry or an index.
A Survey on Metamorphic Testing
Open accessAbstract
A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program's input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output “morphs” into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449
(2004) · Cited by 5,198
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
Regression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stv.430 · Cited by 1,322
Automated analysis of feature models 20 years later: A literature review
Benavides, D., Segura, S., & Ruiz-Cortés, A. (2010) · Information Systems · doi:10.1016/j.is.2010.01.001 · Cited by 1,204
Self-citationThe oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
The Daikon system for dynamic detection of likely invariants
Ernst, M. D., Perkins, J. H., Guo, P. J., McCamant, S., Pacheco, C., Tschantz, M. S., & Xiao, C. (2007) · Science of Computer Programming · doi:10.1016/j.scico.2007.01.015 · Cited by 1,080
Symbolic execution for software testing: three decades later
Cadar, C., & Sen, K. (2013) · Communications of the ACM · doi:10.1145/2408776.2408795 · Cited by 776
Self-testing/correcting with applications to numerical problems
Blum, M., Luby, M., & Rubinfeld, R. (1993) · Journal of Computer and System Sciences · doi:10.1016/0022-0000(93)90044-w · Cited by 712
An orchestrated survey of methodologies for automated software test case generation
Anand, S., Burke, E. K., Chen, T. Y., Clark, J., Cohen, M. B., Grieskamp, W., Harman, M., Harrold, M. J., McMinn, P., Bertolino, A., et al. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.02.061 · Cited by 668
On testing non-testable programs
Weyuker, E. J. (1982) · The Computer Journal · doi:10.1093/comjnl/25.4.465 · Cited by 606
Metamorphic Testing: A New Approach for Generating Next Test Cases
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 448
Data diversity: an approach to software fault tolerance
Ammann, P., & KNIGHT, J. C. (1988) · IEEE Transactions on Computers · doi:10.1109/12.2185 · Cited by 412
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G., Xu, B., & Chen, T. Y. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 389
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388 · cited 1×
“A closely related idea was presented by Le et al. [119].”
BackgroundDynamically discovering likely program invariants to support program evolution
Ernst, M. D., Cockrell, J., Griswold, W. G., & Notkin, D. S. (1999) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/302405.302467 · Cited by 357
Self-testing/correcting with applications to numerical problems
Blum, M., Luby, M., & Rubinfeld, R. (1990) · Proceedings of the twenty-second annual ACM symposium on Theory of computing - STOC '90 · doi:10.1145/100216.100225 · Cited by 303
iCanCloud: A Flexible and Scalable Cloud Infrastructure Simulator
Núñez, A., Vázquez‐Poletti, J. L., Caminero, A. C., Castañé, G. G., Carretero, J., & Llorente, I. M. (2012) · Journal of Grid Computing · doi:10.1007/s10723-012-9208-5 · Cited by 301
Software manual for the elementary functions
(1981) · Computer Languages · doi:10.1016/0096-0551(81)90076-x · Cited by 218
Achievements, Open Problems and Challenges for Search Based Software Testing
Harman, M., Jia, Y., & Zhang, Y. (2015) · 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2015.7102580 · Cited by 209
How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?
Liu, H., Kuo, F.-C., Towey, D., & Chen, T. Y. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.46 · Cited by 206
Analyzing the past to prepare for the future
(2002) · MIS Quarterly · doi:10.5555/2017160.2017162 · Cited by 179
Fault-based testing without the need of oracles
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2003) · Information and Software Technology · doi:10.1016/s0950-5849(02)00129-5 · Cited by 179
Properties of Machine Learning Applications for Use in Metamorphic Testing
Murphy, C., Kaiser, G. E., & Hu, L. (2008) · doi:10.7916/d8xk8pfd · Cited by 164
Metamorphic testing for software quality assessment: A study of search engines
Zhou, Z. Q., Xiang, S., & Chen, T. Y. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2478001 · Cited by 163
An innovative approach for testing bioinformatics programs using metamorphic testing
Chen, T. Y., Ho, J. W., Liu, H., & Xie, X. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-24 · Cited by 159
A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications
Chan, W. K., Cheung, S.-C., & Leung, K. R. P. H. (2007) · International Journal of Web Services Research · doi:10.4018/jwsr.2007040103 · Cited by 127
Kanewala, U., Bieman, J. M., & Ben‐Hur, A. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1594 · Cited by 126
Search-based inference of polynomial metamorphic relations
Zhang, J. M., Chen, J., Hao, D., Xiong, Y., Xie, B., Zhang, L., & Mei, H. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642994 · Cited by 121
Using machine learning techniques to detect metamorphic relations for programs without test oracles
Kanewala, U., & Bieman, J. M. (2013) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2013.6698899 · Cited by 117
Metamorphic slice: An application in spectrum-based fault localization
Xie, X., Wong, W. E., Chen, T. Y., & Xu, B. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.08.008 · Cited by 112
Metamorphic testing of programs on partial differential equations: a case study
Chen, T. Y., Feng, J., & Tse, T. H. (2003) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.2002.1045022 · Cited by 112
Case studies on the selection of useful relations in metamorphic testing
Chen, T. Y., Huang, D., Tse, T.-H., & Zhou, Z. Q. (2004) · Swinburne figshare (Swinburne University of Technology) · Cited by 108
Application of Metamorphic Testing to Supervised Classifiers
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2009) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2009.26 · Cited by 105
Arnaud Gotlieb, B., & Botella, B. (2004) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.2003.1245319 · Cited by 96
METRIC: METamorphic Relation Identification based on the Category-choice framework
Chen, T. Y., Poon, P., & Xie, X. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.07.037 · Cited by 95
Metamorphic Testing and Its Applications
Zhou, Z. Q., Huang, D., Tse, T. H., Yang, Z., Huang, H., & Chen, T. Y. (2004) · Cited by 92
Software Testing, Verification and Reliability
Stringfellow, C., Andrews, A., Wohlin, C., & Petersson, H. (2006) · Software Testing Verification and Reliability · doi:10.1002/(issn)1099-1689 · Cited by 89
Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/566171.566202 · Cited by 87
Automatic system testing of programs without test oracles
Murphy, C., Shen, K., & Kaiser, G. E. (2009) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/1572272.1572295 · Cited by 85
Automated functional testing of online search services
Zhou, Z. Q., Zhang, S., Hagenbuchner, M., Tse, T. H., Kuo, F., & Chen, T. Y. (2010) · Software Testing Verification and Reliability · doi:10.1002/stvr.437 · Cited by 85
Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.23 · Cited by 83
Testing context-sensitive middleware-based software applications
Tse, T. H., Yau, S. S., Chan, W. K., Lu, H. Z., & Chen, T. Y. (2004) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.2004.1342879 · Cited by 82
An Empirical Study on the Selection of Good Metamorphic Relations
Mayer, J., & Guderlei, R. (2006) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2006.24 · Cited by 79
Guderlei, R., & Mayer, J. (2007) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2007.4385527 · Cited by 76
Automated metamorphic testing on the analyses of feature models
Segura, S., Hierons, R. M., Benavides, D., & Ruiz–Cortés, A. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.11.002 · Cited by 71
Self-citationMetamorphic Testing and Beyond
Chen, T. Y., Kuo, F., Tse, T. H., & Zhou, Z. Q. (2005) · International Workshop on Software Technology and Engineering Practice (STEP) · doi:10.1109/step.2003.18 · Cited by 71
Murphy, C., Shen, K., & Kaiser, G. E. (2009) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2009.19 · Cited by 64
INTEGRATION TESTING OF CONTEXT-SENSITIVE MIDDLEWARE-BASED APPLICATIONS: A METAMORPHIC APPROACH
Chan, W. K., Chen, T. Y., Lu, H. Z., Tse, T. H., & Yau, S. S. (2006) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194006002951 · Cited by 61
BioWeka—extending the Weka framework for bioinformatics
Gewehr, J. E., Szugat, M., & Zimmer, R. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btl671 · Cited by 60
Towards a Metamorphic Testing Methodology for Service-Oriented Software Applications
Chan, W. K., Cheung, S.-C., & Leung, K. R. P. H. (2006) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2005.67 · Cited by 54
Spectrum-Based Fault Localization: Testing Oracles are No Longer Mandatory
Xie, X., Wong, W. E., Chen, T. Y., & Xu, B. (2011) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2011.20 · Cited by 54
Metamorphic testing and testing with special values
Chen, T. Y., Kuo, F., Liu, Y., & Tang, A. (2024) · Swinburne figshare (Swinburne University of Technology) · doi:10.25916/sut.26291794 · Cited by 51
Cao, Y., Zhou, Z. Q., & Chen, T. Y. (2013) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2013.43 · Cited by 50
On Random Testing of Image Processing Applications
Mayer, J., & Guderlei, R. (2006) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2006.45 · Cited by 49
Cross-checking oracles from intrinsic software redundancy
Carzaniga, A., Goffi, A., Gorla, A., Mattavelli, A., & Pezzè, M. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568287 · Cited by 48
Metamorphic model-based testing applied on NASA DAT: an experience report
Lindvall, M., Ganesan, D., Ardal, R., & Wiegand, R. E. (2015) · doi:10.5555/2819009.2819030 · Cited by 47
Automated Test Data Generation on the Analyses of Feature Models: A Metamorphic Testing Approach
Segura, S., Hierons, R. M., Benavides, D., & Ruiz–Cortés, A. (2010) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2010.20 · Cited by 47
Self-citationMetamorphic Model-Based Testing Applied on NASA DAT -- An Experience Report
Lindvall, M., Ganesan, D., Ardal, R., & Wiegand, R. E. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.348 · Cited by 45
Metamorphic Testing for Web Services: Framework and a Case Study
Sun, C., Wang, G., Mu, B., Liu, H., Wang, Z., & Chen, T. Y. (2011) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2011.65 · Cited by 44
Effective Statistical Fault Localization Using Program Slices
Lei, Y., Mao, X., Dai, Z., & Wang, C. (2012) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2012.9 · Cited by 44
A Metamorphic Relation-Based Approach to Testing Web Services Without Oracles
Sun, C., Wang, G., Mu, B., Liu, H., Wang, Z., & Chen, T. Y. (2012) · International Journal of Web Services Research · doi:10.4018/jwsr.2012010103 · Cited by 39
An empirical comparison between direct and indirect test result checking approaches
Hu, P., Zhang, Z., Chan, W. K., & Tse, T. H. (2006) · doi:10.1145/1188895.1188901 · Cited by 37
On effective testing of health care simulation software
Murphy, C., Raunak, M. S., King, A., Chen, S., Imbriano, C., Kaiser, G. E., Lee, I., Sokolsky, O. V., Clarke, L. A., & Osterweil, L. J. (2011) · doi:10.1145/1987993.1988003 · Cited by 37
Fault-based testing in the absence of an oracle
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.2001.960614 · Cited by 36
Techniques for testing scientific programs without an oracle
Kanewala, U., & Bieman, J. M. (2013) · International Workshop on Software Engineering for Computational Science and Engineering (SE-CSE) · doi:10.1109/secse.2013.6615099 · Cited by 35
Wu, P. (2005) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2005.166 · Cited by 34
Automated functional testing of web search engines in the absence of an oracle
Zho, Z. Q. Z., Tse, T. H., Kuo, F., & Chen, T. Y. (2007) · Swinburne figshare (Swinburne University of Technology) · Cited by 33
Search-based synthesis of equivalent method sequences
Goffi, A., Gorla, A., Mattavelli, A., Pezzè, M., & Tonella, P. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635888 · Cited by 33
An effective testing method for end-user programmers
Chen, T. Y., Kuo, F., & Zhou, Z. Q. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083236 · Cited by 33
TOWARDS AUTOMATIC TESTING OF IMAGING SOFTWARE BY MEANS OF RANDOM AND METAMORPHIC TESTING
Guderlei, R., & Mayer, J. (2007) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194007003471 · Cited by 32
Metamorphic Testing of Stochastic Optimisation
Yoo, S. (2010) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2010.26 · Cited by 29
A methodology for validating cloud models using metamorphic testing
Núñez, A., & Hierons, R. M. (2014) · Annals of Telecommunications · doi:10.1007/s12243-014-0442-7 · Cited by 27
Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking
Zhang, Z., Chan, W. K., Tse, T. H., & Pei-Feng, H. U. (2009) · Journal of Software · doi:10.3724/sp.j.1001.2009.00578 · Cited by 26
Techniques for Automatic Detection of Metamorphic Relations
Kanewala, U. (2014) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2014.62 · Cited by 25
Impacts of Test Suite's Class Imbalance on Spectrum-Based Fault Localization Techniques
Rao, P. C., Zheng, Z., Chen, T. Y., Wang, N., & Cai, K. (2013) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2013.18 · Cited by 25
Metamorphic Testing: A Simple Method for Alleviating the Test Oracle Problem
Chen, T. Y. (2015) · doi:10.1109/ast.2015.18 · Cited by 23
Automated metamorphic testing of variability analysis tools
Segura, S., Durán, A., Sánchez, A. B., Berre, D. L., Lonca, E., & Ruiz–Cortés, A. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1566 · Cited by 22
Self-citationMetamorphic testing of a Monte Carlo modeling program
Ding, J., Wu, T., Wu, D., Lu, J. Q., & Hu, X. (2011) · International Workshop on Automation of Software Test (AST) · doi:10.1145/1982595.1982597 · Cited by 21
Chan, W. K., Ho, J. C. F., & Tse, T. H. (2009) · Software Testing Verification and Reliability · doi:10.1002/stvr.408 · Cited by 20
An Effective Iterative Metamorphic Testing Algorithm Based on Program Path Analysis
Dong, G., Nie, C., Xu, B., & Wang, L. (2007) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2007.4385510 · Cited by 19
Test oracles based on metamorphic relations for image processing applications
Jameel, T., Lin, M., & Liu, C. (2015) · ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD) · doi:10.1109/snpd.2015.7176238 · Cited by 18
Testing of Heuristic Methods: A Case Study of Greedy Algorithm
Barus, A., Chen, T. Y., Grant, D. D., Kuo, F., & Lau, M. F. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22386-0_19 · Cited by 16
Concolic Metamorphic Debugging
Jin, H., Jiang, Y., Liu, N., Xu, C., Ma, X., & Lu, J. J. (2015) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2015.79 · Cited by 15
Metamorphic Testing Using Geometric Interrogation Technique and Its Application
Sim, K. Y., Pao, W. K. S., & Lin, C. (1970) · ECTI Transactions on Computer and Information Technology (ECTI-CIT) · doi:10.37936/ecti-cit.200512.51837 · Cited by 15
An effective testing method for end-user programmers
Chen, T. Y., Kuo, F., & Zhou, Z. Q. (2005) · doi:10.1145/1083231.1083236 · Cited by 15
Metamorphic testing of decision support systems: a case study
Kuo, F., Zhou, Z. Q., Ma, J., & Zhang, G. (2010) · IET Software · doi:10.1049/iet-sen.2009.0084 · Cited by 14
Metamorphic testing: a simple method for alleviating the test oracle problem
Chen, T. Y. (2015) · Automation of Software Test · doi:10.5555/2819261.2819278 · Cited by 14
Empirical Evaluation of Approaches to Testing Applications without Test Oracles
Murphy, C., & Kaiser, G. E. (2010) · doi:10.7916/d88g8tmc · Cited by 13
Automating unit and integration testing with partial oracles
Just, R., & Schweiggert, F. (2011) · Software Quality Journal · doi:10.1007/s11219-011-9151-x · Cited by 13
A Formal Model for Metamorphic Relation Decomposition
Hui, Z., & Huang, S. (2013) · World Congress on Software Engineering (WCSE) · doi:10.1109/wcse.2013.14 · Cited by 13
Metamorphic Testing Integer Overflow Faults of Mission Critical Program: A Case Study
Hui, Z., Huang, S., Ren, Z., & Yao, Y. (2013) · Mathematical Problems in Engineering · doi:10.1155/2013/381389 · Cited by 12
Evaluating Testing Strategies for Imaging Software by Means of Mutation Analysis
Just, R., & Schweiggert, F. (2009) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2009.20 · Cited by 12
An optimized method for generating cases of metamorphic testing
Chen, L., Cai, L., Liu, J., Liu, Z., Wei, S., & Liu, P. (2012) · Cited by 12
Achievements and Challenges of Metamorphic Testing
Hui, Z., & Huang, S. (2013) · World Congress on Software Engineering (WCSE) · doi:10.1109/wcse.2013.16 · Cited by 11
A property-based testing framework for encryption programs
Sun, C., Wang, Z., & Wang, G. (2014) · Frontiers of Computer Science · doi:10.1007/s11704-014-3040-y · Cited by 11
Metamorphic Testing: Applications and Integration with Other Methods: Tutorial Synopsis
Chen, T. Y., Kuo, F., Towey, D., & Zhou, Z. Q. (2012) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2012.21 · Cited by 11
Oracles Are Hardly Attain'd, and Hardly Understood: Confessions of Software Testing Researchers
Chan, W. K., & Tse, T. H. (2013) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2013.16 · Cited by 10
Automating software tests with partial oracles in integrated environments
Just, R., & Schweiggert, F. (2010) · doi:10.1145/1808266.1808280 · Cited by 10
Aruna, C., & Prasad, R. S. R. (2014) · International Conference on Advances in Computing, Communications and Informatics (ICACCI) · doi:10.1109/icacci.2014.6968586 · Cited by 9
Testing an Open Source Suite for Open Queuing Network Modelling Using Metamorphic Testing Technique
Chen, T. Y., Kuo, F., Merkel, R., & Tam, W. K. (2009) · IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) · doi:10.1109/iceccs.2009.28 · Cited by 9
Using runtime testing to detect defects in applications without test oracles
Murphy, C. (2008) · doi:10.1145/1496653.1496659 · Cited by 8
Application of Metamorphic Testing to a Case Study in Web Services Compositions
de Castro-Cabrera, M. C., & Medina‐Bulo, I. (2012) · Communications in computer and information science · doi:10.1007/978-3-642-35755-8_13 · Cited by 7
Automatic generation of cost-effective test oracles
Goffi, A. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591078 · Cited by 6
A Methodology for Designing Energy-aware Systems for Computational Science
Cañizares, P. C., Núñez, A., Nunez, M. A., & Pardo, J. J. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.05.438 · Cited by 5
Detecting faults in technical indicator computations for financial market analysis
Sim, K. Y., Low, C. S., & Kuo, F. (2010) · International Conference on Information Science and Engineering · doi:10.1109/icise.2010.5689221 · Cited by 5
Research on Metamorphic Testing: A Case Study in Integer Bugs Detection
Yi, Y., Changyou, Z., Huang, S., & Zhengping, R. (2013) · International Conference on Intelligent System Design and Engineering Application (ISDEA) · doi:10.1109/isdea.2013.516 · Cited by 4
Research on Metamorphic Testing for Oracle Problem of Integer Bugs
Yao, Y., Huang, S., & Ji, M.-Y. (2012) · Advances in intelligent and soft computing · doi:10.1007/978-3-642-30126-1_16 · Cited by 3
Research on Metamorphic Testing: A Case Study in Integer Bugs Detection
Yao, Y., Changyou, Z., Huang, S., & Zhengping, R. (2013) · Research Journal of Applied Sciences Engineering and Technology · doi:10.19026/rjaset.6.3677 · Cited by 2
Cited by
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Metamorphic Testing: A Review of Challenges and Opportunities
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 665
Zhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611
Proceedings of the 19th Python in Science Conference
(2020) · Proceedings of the Python in Science Conference · doi:10.25080/majora-342d178e-02b · Cited by 284
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Metadata
- Authors
- Segura, S., Fraser, G., Sanchez, A. B., & Ruiz-Cortes, A.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2016-09-01
- Volume
- 42
- Issue
- 9
- Pages
- 805-824
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2324595780
- Cited by
- 563
- References
- 109
- FWCI
- 127.86
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.