Metamorphic testing for software quality assessment: A study of search engines
Open accessAbstract
Metamorphic testing is a testing technique that can be used to verify the functional correctness of software in the absence of an ideal oracle. This paper extends metamorphic testing into a user-oriented approach to software verification, validation, and quality assessment, and conducts large scale empirical studies with four major web search engines: Google, Bing, Chinese Bing, and Baidu. These search engines are very difficult to test and assess using conventional approaches owing to the lack of an objective and generally recognized oracle. The results are useful for both search engine developers and users, and demonstrate that our approach can effectively alleviate the oracle problem and challenges surrounding a lack of specifications when verifying, validating, and evaluating large and complex software systems.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Violation of a metamorphic relation points to a defect.
“Violation of a metamorphic relation points to a defect (Zhou et al., 2015).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Zhou et al., 2015
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,588 · see citing works
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,385 · see citing works
(2017) · Cited by 16,319 · see citing works
Cohen, J. (1992) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep10768783 · Cited by 5,870 · see citing works
Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research
Levine, T. R., & Hullett, C. R. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00828.x · Cited by 998 · see citing works
An Efficient Boosting Algorithm for Combining Preferences
Freund, Y., Iyer, R. D., Schapire, R. E., & Singer, Y. (1998) · Cited by 806 · see citing works
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633 · see citing works
Metamorphic Testing: A New Approach for Generating Next Test CasesPreprint
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 458 · see citing works
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 394 · see citing works
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388 · see citing works
Named entity recognition in query
Guo, J., Xu, G., Cheng, X., & Li, H. (2009) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1571941.1571989 · Cited by 384 · see citing works
Software testing and analysis: process, principles, and techniques
Pezzè, M., & Young, M. (2008) · Choice Reviews Online · doi:10.5860/choice.46-0935 · Cited by 342 · see citing works
How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?
Liu, H., Kuo, F., Towey, D., & Chen, T. Y. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.46 · Cited by 207 · see citing works
Web Engineering: Modelling and Implementing Web Applications
Rossi, G. H., Schwabe, D., Pastor, Ó., & Olsina, L. A. (2007) · Human-computer interaction series · doi:10.1007/978-1-84628-923-1 · Cited by 184 · see citing works
Fault-based testing without the need of oracles
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · Information and Software Technology · doi:10.1016/s0950-5849(02)00129-5 · Cited by 179 · see citing works
Random Testing: Theoretical Results and Practical Implications
Arcuri, A., Iqbal, M. Z., & Briand, L. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.121 · Cited by 178 · see citing works
Estimating the Query Difficulty for Information Retrieval
Carmel, D., & Yom‐Tov, E. (2010) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00235ed1v01y201004icr015 · Cited by 162 · see citing works
An innovative approach for testing bioinformatics programs using metamorphic testing
Chen, T. Y., Ho, J. W. K., Liu, H., & Xie, X. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-24 · Cited by 159 · see citing works
Software Product Quality Control
Wagner, S. (2013) · Springer eBooks · doi:10.1007/978-3-642-38571-1 · Cited by 133 · see citing works
Search-based inference of polynomial metamorphic relations
Zhang, J. M., Chen, J., Hao, D., Xiong, Y., Xie, B., Zhang, L., & Mei, H. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642994 · Cited by 116 · see citing works
Metamorphic slice: An application in spectrum-based fault localization
Xie, X., Wong, W. E., Chen, T. Y., & Xu, B. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.08.008 · Cited by 112 · see citing works
Case studies on the selection of useful relations in metamorphic testing
Chen, T. Y., Huang, D., Tse, T.-H., & Zhou, Z. Q. (2004) · Swinburne figshare (Swinburne University of Technology) · Cited by 108 · see citing works
Fundamentals of Software Engineering
O’Regan, G. (2019) · Undergraduate topics in computer science · doi:10.1007/978-3-030-28494-7_2 · Cited by 101 · see citing works
An upper bound on software testing effectiveness
Chen, T. Y., & Merkel, R. (2008) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1363102.1363107 · Cited by 99 · see citing works
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/566171.566202 · Cited by 86 · see citing works
Automatic system testing of programs without test oracles
Murphy, C., Shen, K., & Kaiser, G. E. (2009) · doi:10.1145/1572272.1572295 · Cited by 85 · see citing works
Automated functional testing of online search services
Zhou, Z. Q., Zhang, S., Hagenbuchner, M., Tse, T. H., Kuo, F., & Chen, T. Y. (2010) · Software Testing Verification and Reliability · doi:10.1002/stvr.437 · Cited by 85 · see citing works
Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.23 · Cited by 84 · see citing works
Ieong, S., Mishra, N., Sadikov, E., & Zhang, L. (2012) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2124295.2124345 · Cited by 55 · see citing works
Cao, Y., Zhou, Z. Q., & Chen, T. Y. (2013) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2013.43 · Cited by 50 · see citing works
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/566172.566202 · Cited by 48 · see citing works
Metamorphic model-based testing applied on NASA DAT: an experience report
Lindvall, M., Ganesan, D., Ardal, R., & Wiegand, R. E. (2015) · doi:10.5555/2819009.2819030 · Cited by 47 · see citing works
Metamorphic Model-Based Testing Applied on NASA DAT -- An Experience Report
Lindvall, M., Ganesan, D., Ardal, R., & Wiegand, R. E. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.348 · Cited by 45 · see citing works
Oracles for Distributed Testing
Hierons, R. M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.45 · Cited by 42 · see citing works
Automated functional testing of web search engines in the absence of an oracle
Zho, Z. Q. Z., Tse, T. H., Kuo, F., & Chen, T. Y. (2007) · Swinburne figshare (Swinburne University of Technology) · Cited by 33 · see citing works
An effective testing method for end-user programmers
Chen, T. Y., Kuo, F., & Zhou, Z. Q. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083236 · Cited by 32 · see citing works
Soboroff, I. M. (2006) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1148170.1148220 · Cited by 30 · see citing works
A methodology for validating cloud models using metamorphic testing
Núñez, A., & Hierons, R. M. (2014) · Annals of Telecommunications · doi:10.1007/s12243-014-0442-7 · Cited by 25 · see citing works
Automated metamorphic testing of variability analysis tools
Segura, S., Durán, A., Sánchez, A. B., Berre, D. L., Lonca, E., & Ruiz–Cortés, A. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1566 · Cited by 21 · see citing works
An effective testing method for end-user programmers
Chen, T. Y., Kuo, F., & Zhou, Z. Q. (2005) · doi:10.1145/1083231.1083236 · Cited by 15 · see citing works
Mining Iterative Generators and Representative Rules for Software Specification Discovery
Lo, D., Li, J., wong, I., & Khoo, S. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2010.24 · Cited by 13 · see citing works
If You Ask Nicely, I will Answer: Semantic Search and Today's Search Engines
Signorini, A., & Imieliński, T. (2009) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2009.31 · Cited by 12 · see citing works
T-DOC: A Tool for the Automatic Generation of Testing Documentation for OSS Products
Morasca, S., Taibi, D., & Tosi, D. (2010) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-3-642-13244-5_16 · Cited by 9 · see citing works
Extraction and Evaluation of Candidate Named Entities in Search Engine Queries
Alasiry, A., Levene, M., & Poulovassilis, A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35063-4_35 · Cited by 8 · see citing works
Structured web search in small domain
Qader, M. A., Rahman, S., & Hasan, M. (2014) · doi:10.1109/iccitechn.2014.6997348 · Cited by 1 · see citing works
Flagged first, then by citations
3 of the 48 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Chen, T. Y., Kuo, F., Liu, H., Poon, P., Towey, D., Tse, T. H., & Zhou, Z. Q. (2018) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 691 · see citing works
A Survey on Metamorphic Testing
Segura, S., Fraser, G., Sánchez, A. B., & Ruiz–Cortés, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 571 · see citing works
Metamorphic testing of driverless cars
Zhou, Z. Q., & Sun, L. (2019) · Communications of the ACM · doi:10.1145/3241979 · Cited by 200 · see citing works
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
Metamorphic Relations for Enhancing System Understanding and Use
Zhou, Z. Q., Sun, L., Chen, T. Y., & Towey, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876433 · Cited by 127 · see citing works
Metamorphic Testing: Testing the Untestable
Segura, S., Towey, D., Zhou, Z. Q., & Chen, T. Y. (2018) · IEEE Software · doi:10.1109/ms.2018.2875968 · Cited by 126 · see citing works
Metamorphic Testing of RESTful Web APIs
Segura, S., Parejo, J. A., Troya, J., & Ruiz–Cortés, A. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2764464 · Cited by 126 · see citing works
Structure-invariant testing for machine translation
He, P., Meister, C., & Su, Z. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380339 · Cited by 89 · see citing works
Metamorphic Testing for Cybersecurity
Chen, T. Y., Kuo, F., Ma, W., Susilo, W., Towey, D., Voas, J. M., & Zhou, Z. Q. (2016) · Computer · doi:10.1109/mc.2016.176 · Cited by 85 · see citing works
Big Data Systems: A Software Engineering Perspective
Davoudian, A., & Liu, M. (2021) · ACM Computing Surveys · doi:10.1145/3408314 · Cited by 88 · see citing works
Metamorphic testing of RESTful web APIs
Segura, S., Parejo, J. A., Troya, J., & Ruiz–Cortés, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182528 · Cited by 77 · see citing works
Xie, X., Zhang, Z., Chen, T. Y., Liu, Y., Poon, P.-L., & Xu, B. (2020) · IEEE Transactions on Reliability · doi:10.1109/tr.2020.2972266 · Cited by 62 · see citing works
(7794023), X. X., (7695560), Z. Z., (9865748), T. C., (4702669), Y. L., (9827297), P. P., & (7787783), B. X. (2020) · Figshare · Cited by 57 · see citing works
Testing Machine Translation via Referential Transparency
He, P., Meister, C., & Su, Z. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00047 · Cited by 51 · see citing works
RBF-MLMR: A Multi-Label Metamorphic Relation Prediction Approach Using RBF Neural Network
Zhang, P., Zhou, X., Pelliccione, P., & LEUNG, H. K. N. (2017) · IEEE Access · doi:10.1109/access.2017.2758790 · Cited by 48 · see citing works
METRIC+: A Metamorphic Relation Identification Technique Based on Input plus Output Domains
Sun, C., Fu, A., Poon, P., Xie, X., Liu, H., & Chen, T. Y. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2934848 · Cited by 48 · see citing works
A metamorphic testing approach for supporting program repair without the need for a test oracle
Jiang, M., Chen, T. Y., Kuo, F., Towey, D., & Ding, Z. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.04.002 · Cited by 46 · see citing works
Adaptive metamorphic testing with contextual bandits
Spieker, H., & Arnaud Gotlieb, B. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110574 · Cited by 45 · see citing works
A Machine Learning Based Framework for Verification and Validation of Massive Scale Image Data
Ding, J., Hu, X., & Gudivada, V. N. (2017) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2017.2680460 · Cited by 43 · see citing works
Machine translation testing via pathological invariance
Gupta, S., He, P., Meister, C., & Su, Z. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409756 · Cited by 43 · see citing works
Theoretical and Empirical Analyses of the Effectiveness of Metamorphic Relation Composition
Qiu, K., Zheng, Z., Chen, T. Y., & Poon, P. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3009698 · Cited by 42 · see citing works
Zhang, Z., Wang, P., Guo, H., Wang, Z., Zhou, Y., & Huang, Z. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106701 · Cited by 42 · see citing works
Zhou, Z. Q., Tse, T. H., & Witheridge, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2915065 · Cited by 41 · see citing works
Testing multiple linear regression systems with metamorphic testing
Luu, Q.-H., Lau, M. F., Ng, S., & Chen, T. Y. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111062 · Cited by 41 · see citing works
SemMT: A Semantic-Based Testing Approach for Machine Translation Systems
Cao, J., Li, M., Li, Y., Wen, M., Cheung, S.-C., & Chen, H. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3490488 · Cited by 43 · see citing works
Automated inference of likely metamorphic relations for model transformations
Troya, J., Segura, S., & Ruiz–Cortés, A. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.05.043 · Cited by 39 · see citing works
Testing Your Question Answering Software via Asking Recursively
Chen, S., Jin, S., & Xie, X. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678670 · Cited by 38 · see citing works
Exploring the usefulness of unlabelled test cases in software fault localization
Zhang, X., Zheng, Z., & Cai, K. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.07.027 · Cited by 36 · see citing works
Metamorphic Testing of Navigation Software: A Pilot Study with Google Maps
Brown, J., Zhou, Z. Q., & Chow, Y.-W. (2018) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2018.713 · Cited by 34 · see citing works
A monte carlo method for metamorphic testing of machine translation services
Pesu, D., Zhou, Z. Q., Zhen, J., & Towey, D. (2018) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/3193977.3193980 · Cited by 34 · see citing works
Increasing Validity of Simulation Models Through Metamorphic Testing
Olsen, M., & Raunak, M. S. (2018) · IEEE Transactions on Reliability · doi:10.1109/tr.2018.2850315 · Cited by 32 · see citing works
An application of metamorphic testing for testing scientific software
Ding, J., Zhang, D., & Hu, X. (2016) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/2896971.2896981 · Cited by 31 · see citing works
Metamorphic Relation Patterns for Query-Based Systems
Segura, S., Durán, A., Troya, J., & Ruiz–Cortés, A. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00012 · Cited by 28 · see citing works
Metamorphic Relations for Detection of Performance Anomalies
Johnston, O., Jarman, D., Berry, J., Zhou, Z. Q., & Chen, T. Y. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00017 · Cited by 27 · see citing works
Perfect is the enemy of test oracle
Ibrahimzada, A. R., Varli, Y., Tekinoglu, D., & Jabbarvand, R. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549086 · Cited by 26 · see citing works
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Models
Meng, L., Li, Y., Lin, C., Wang, Z., Wu, D., Zhou, Y., & Xu, B. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00045 · Cited by 25 · see citing works
A Template-Based Approach to Describing Metamorphic Relations
Segura, S., Durán, A., Troya, J., & Ruiz–Cortés, A. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.3 · Cited by 24 · see citing works
Iqbal, M., Han, J. C., Zhou, Z. Q., Towey, D., & Chen, T. Y. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107104 · Cited by 22 · see citing works
Metamorphic testing 20 years later
Segura, S., & Zhou, Z. Q. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3183468 · Cited by 21 · see citing works
Web application testing—Challenges and opportunities
Balsam, S., & Mishra, D. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112186 · Cited by 20 · see citing works
A Framework for Ensuring the Quality of a Big Data Service
Ding, J., Zhang, D., & Hu, X. (2016) · IEEE International Conference on Services Computing · doi:10.1109/scc.2016.18 · Cited by 20 · see citing works
Metamorphic testing for Adobe data analytics software
Jarman, D., Zhou, Z. Q., & Chen, T. Y. (2017) · Research Online (University of Wollongong) · doi:10.5555/3103620.3103630 · Cited by 20 · see citing works
Kıraç, F., Aktemur, B., & Sözer, H. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.06.023 · Cited by 19 · see citing works
Hierarchical metamorphic relations for testing scientific software
Lin, X.-Y., Simon, M. A., & Niu, N. (2018) · doi:10.1145/3194747.3194750 · Cited by 19 · see citing works
Input Test Suites for Program Repair: A Novel Construction Method Based on Metamorphic Relations
Jiang, M., Chen, T. Y., Zhou, Z. Q., & Ding, Z. (2020) · IEEE Transactions on Reliability · doi:10.1109/tr.2020.3003313 · Cited by 19 · see citing works
Metamorphic relation prioritization for effective regression testing
Srinivasan, M., & Kanewala, U. (2022) · Software Testing Verification and Reliability · doi:10.1002/stvr.1807 · Cited by 18 · see citing works
Metamorphic Testing for Adobe Data Analytics Software
Jarman, D., Zhou, Z. Q., & Chen, T. Y. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.1 · Cited by 18 · see citing works
On the effectiveness of testing sentiment analysis systems with metamorphic testing
Jiang, M., Chen, T. Y., & Wang, S. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106966 · Cited by 17 · see citing works
Metamorphic Testing of an Artificially Intelligent Chess Game
Liaqat, A., Sindhu, M. A., & Siddiqui, G. F. (2020) · IEEE Access · doi:10.1109/access.2020.3024929 · Cited by 17 · see citing works
Metamorphic Relations for Data Validation: A Case Study of Translated Text Messages
Yan, B., Yecies, B., & Zhou, Z. Q. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00018 · Cited by 17 · see citing works
Metamorphic Exploration of an Unsupervised Clustering Program
Yang, S., Towey, D., & Zhou, Z. Q. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00015 · Cited by 17 · see citing works
Metamorphic Relation Generation: State of the Art and Research Directions
Li, R., Liu, H., Poon, P., Towey, D., Sun, C., Zheng, Z., Zhou, Z. Q., & Chen, T. Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708521 · Cited by 16 · see citing works
The Impact of a Dot: Case Studies of a Noise Metamorphic Relation Pattern
Wu, C., Sun, L., & Zhou, Z. Q. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00011 · Cited by 15 · see citing works
Automatic Self-Validation for Code Coverage Profilers
Yang, Y., Jiang, Y., Zuo, Z., Wang, Y., Sun, H., Lu, H., Zhou, Y., & Xu, B. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00018 · Cited by 16 · see citing works
Using metamorphic relations to verify and enhance Artcode classification
Xu, L., Towey, D., French, A. P., Benford, S., Zhou, Z. Q., & Chen, T. Y. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111060 · Cited by 13 · see citing works
Automated generation of metamorphic relations for query-based systems
Segura, S., Alonso, J. C. G., Martin-Lopez, A., Durán, A., Troya, J., & Ruiz–Cortés, A. (2022) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/3524846.3527338 · Cited by 13 · see citing works
Enhancing supervised classifications with metamorphic relations
Xu, L., Towey, D., French, A. P., Benford, S., Zhou, Z. Q., & Chen, T. Y. (2018) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/3193977.3193978 · Cited by 13 · see citing works
Metamorphic testing of OpenStreetMap
Almendros-Jiménez, J. M., Becerra-Terón, A., Merayo, M. G., & Nunez, M. A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106631 · Cited by 12 · see citing works
A Metamorphic Testing Approach for Assessing Question Answering Systems
Tu, K., Jiang, M., & Ding, Z. (2021) · Mathematics · doi:10.3390/math9070726 · Cited by 12 · see citing works
Evaluating Natural Language Inference Models: A Metamorphic Testing Approach
Jiang, M., Bao, H., Tu, K., Zhang, X., & Ding, Z. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00033 · Cited by 11 · see citing works
Metamorphic testing: A new student engagement approach for a new software testing paradigm
Towey, D., Chen, T. Y., Kuo, F., Liu, H., & Zhou, Z. Q. (2016) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale.2016.7851797 · Cited by 11 · see citing works
Metamorphic testing of named entity recognition systems: A case study
Xu, Y., Zhou, Z. Q., Zhang, X., Wang, J., & Jiang, M. (2022) · IET Software · doi:10.1049/sfw2.12058 · Cited by 10 · see citing works
One step further: evaluating interpreters using metamorphic testing
Fan, M. Y., Wei, J., Jin, W., Zhou, X., Wei, W., & Liu, T. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534225 · Cited by 10 · see citing works
On the Investigation of Essential Diversities for Deep Learning Testing Criteria
Zhang, Z., & Xie, X. (2019) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2019.00056 · Cited by 10 · see citing works
Evaluation of Chinese Natural Language Processing System Based on Metamorphic Testing
Jin, L., Ding, Z., & Zhou, H. (2022) · Mathematics · doi:10.3390/math10081276 · Cited by 9 · see citing works
Hazott, C., Stögmüller, F., & Grosse, D. J. (2024) · Integration · doi:10.1016/j.vlsi.2024.102320 · Cited by 9 · see citing works
Testing bidirectional model transformation using metamorphic testing
He, X., Chen, X., Cai, S., Zhang, Y., & Huang, G. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.07.010 · Cited by 9 · see citing works
On Applying Metamorphic Testing: An Empirical Study on Academic Search Engines
de Andrade, S. A., Santos, Í., Brito, C., Junior, M., Souza, S. R. S., & Delamaro, M. E. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00010 · Cited by 9 · see citing works
A Machine Learning Approach for Developing Test Oracles for Testing Scientific Software
Ding, J., & Zhang, D. (2016) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2016-137 · Cited by 9 · see citing works
Metamorphic Testing for Processor Verification: A RISC-V Case Study at the Instruction Level
Riese, F., Herdt, V., Grose, D., & Drechsler, R. (2021) · IFIP/IEEE International Conference on Very Large Scale Integration · doi:10.1109/vlsi-soc53125.2021.9606997 · Cited by 8 · see citing works
Machine Translation Testing via Syntactic Tree Pruning
Zhang, Q., Zhai, J., Fang, C., Liu, J., Sun, W., Hu, H., & Wang, Q. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3640329 · Cited by 8 · see citing works
Rahman, K., Kahanda, I., & Kanewala, U. (2020) · doi:10.1145/3387940.3392250 · Cited by 8 · see citing works
An Interleaving Guided Metamorphic Testing Approach for Concurrent Programs
Sun, C., Dai, H., Geng, N., Liu, H., Chen, T. Y., Wu, P., Cai, Y., & Wang, J. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3607182 · Cited by 8 · see citing works
Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program
Ding, J., & Hu, X. (2016) · Software Quality Journal · doi:10.1007/s11219-016-9337-3 · Cited by 8 · see citing works
A template-based approach to describing metamorphic relations
Segura, S., Durán, A., Troya, J., & Ruiz–Cortés, A. (2017) · doi:10.5555/3103620.3103626 · Cited by 8 · see citing works
Zhang, Y., Chen, T. Y., Pike, M., Towey, D., Ying, Z., & Zhou, Z. Q. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107828 · Cited by 7 · see citing works
Ying, Z., Towey, D., Bellotti, A. G., Chen, T. Y., & Zhou, Z. Q. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107528 · Cited by 7 · see citing works
Scenario‐Driven Metamorphic Testing for Autonomous Driving Simulators
Zhang, Y., Towey, D., Pike, M., Han, J. C., Zhou, Z. Q., Yin, C., Wang, Q., & Chen, X. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1892 · Cited by 7 · see citing works
Towards effective metamorphic testing by algorithm stability for linear classification programs
Yang, Y., Li, Z., Wang, H., Xu, C., & Ma, X. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111012 · Cited by 7 · see citing works
Using Metamorphic Testing to Improve the Quality of Tags in OpenStreetMap
Almendros-Jiménez, J. M., Becerra-Terón, A., Merayo, M. G., & Nunez, M. A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3156182 · Cited by 7 · see citing works
qaAskeR$$^+$$: a novel testing method for question answering software via asking recursive questions
Xie, X., Jin, S., & Chen, S. (2023) · Automated Software Engineering · doi:10.1007/s10515-023-00380-2 · Cited by 7 · see citing works
Harnessing Multiple Source Test Cases in Metamorphic Testing: A Case Study in Bioinformatics
Tang, J. Y. S., Yang, A., Chen, T. Y., & Ho, J. W. K. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.4 · Cited by 7 · see citing works
Metamorphic Relation Patterns for Metamorphic Testing, Exploration and Robustness
Ying, Z., Towey, D., Bellotti, A. G., Chua, C., & Zhou, Z. Q. (2025) · Software Testing Verification and Reliability · doi:10.1002/stvr.70003 · Cited by 6 · see citing works
Evaluation of Metamorphic Testing for Edge Detection in MRI Brain Diagnostics
Jafari, F., Nadeem, A., & Zaman, Q. (2022) · Applied Sciences · doi:10.3390/app12178684 · Cited by 6 · see citing works
Verifying Embedded Graphics Libraries leveraging Virtual Prototypes and Metamorphic Testing
Hazott, C., Stögmüller, F., & Grosse, D. J. (2024) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac58780.2024.10473799 · Cited by 6 · see citing works
Balancing Effectiveness and Flakiness of Non-Deterministic Machine Learning Tests
Xia, C. S., Dutta, S., Misailovíc, S., Marinov, D., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00154 · Cited by 6 · see citing works
Metamorphic Testing of Mapping Software
Brown, J., Zhou, Z. Q., & Chow, Y.-W. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-28430-5_1 · Cited by 6 · see citing works
Identifying Metamorphic Relations: A Data Mutation Directed Approach
Sun, C., Jin, H., Wu, S., Fu, A., Wang, Z., & Chan, W. K. (2022) · Software Practice and Experience · doi:10.1002/spe.3280 · Cited by 5 · see citing works
MRGS‐ART: Metamorphic Relation and Group Selection Based on Adaptive Random Testing
Ying, Z., Towey, D., Bellotti, A. G., & Zhou, Z. Q. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1908 · Cited by 5 · see citing works
Identifying the Failure-Revealing Test Cases in Metamorphic Testing: A Statistical Approach
Zheng, Z., Ren, D., Liu, H., Chen, T. Y., & Li, T. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695990 · Cited by 5 · see citing works
MetaMFL: Metamorphic Multiple Fault Localization Without Test Oracles
Fu, L., Wu, Z., Lei, Y., & Yan, M. L. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3504400 · Cited by 5 · see citing works
ADVISOR: An Adjustable Framework for Test Oracle Automation of Visual Output Systems
Genc, A. E., Sözer, H., Kıraç, F., & Aktemur, B. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2019.2957507 · Cited by 5 · see citing works
Combining Metamorphic Testing and Machine Learning to Enhance OpenStreetMap
Méndez, M., Becerra-Terón, A., Almendros-Jiménez, J. M., Merayo, M. G., & Nunez, M. A. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3379366 · Cited by 5 · see citing works
Addressing Data Quality Problems with Metamorphic Data Relations
Auer, F., & Felderer, M. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00019 · Cited by 5 · see citing works
A Revisit of the Integration of Metamorphic Testing and Test Suite Based Automated Program Repair
Jiang, M., Chen, T. Y., Kuo, F., Ding, Z., Choi, E.-H., & Mizuno, O. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.5 · Cited by 5 · see citing works
A metamorphic testing approach for event sequences
Chen, J., Wang, Y., Guo, Y., & Jiang, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0212476 · Cited by 4 · see citing works
Tabular‐expression‐based method for constructing metamorphic relations
Li, J., Liu, L., & Zhang, P. (2020) · Software Practice and Experience · doi:10.1002/spe.2818 · Cited by 4 · see citing works
Metamorphic Testing for Reliability in System of Systems
Chua, K. K. A., Bae, D., & Jee, E. (2021) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec53868.2021.00046 · Cited by 4 · see citing works
Tao, C., Guo, H., Zhang, J., & Huang, Z. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2444 · Cited by 4 · see citing works
Metamorphic relation automation: Rationale, challenges, and solution directions
Altamimi, E., Elkawakjy, A., & Catal, C. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2509 · Cited by 4 · see citing works
- Authors
- Zhou, Z. Q., Xiang, S., & Chen, T. Y.
- Affiliations
- University of Wollongong (AU)
- Swinburne University of Technology (AU)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2016-03-01
- Volume
- 42
- Issue
- 3
- Pages
- 264-284
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2293505944
- Cited by
- 163
- References
- 48
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 17 · 2023: 9 · 2022: 25 · 2021: 25 · 2020: 15 · 2019: 18 · 2018: 12 · 2017: 17 · 2016: 10
- FWCI
- 54.68
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Research Online (University of Wollongong) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Swinburne figshare (Swinburne University of Technology) (submittedVersion)
- Funders
- Australian Research Council (LP100200208)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- metamorphic testing, search engines, oracle problem, software quality assessment, software verification, software validation, functional correctness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
