The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Metamorphic Testing for Software Quality Assessment: A Study of Search Engines
Open accessAbstract
Metamorphic testing is a testing technique that can be used to verify the functional correctness of software in the absence of an ideal oracle. This paper extends metamorphic testing into a user-oriented approach to software verification, validation, and quality assessment, and conducts large scale empirical studies with four major web search engines: Google, Bing, Chinese Bing, and Baidu. These search engines are very difficult to test and assess using conventional approaches owing to the lack of an objective and generally recognized oracle. The results are useful for both search engine developers and users, and demonstrate that our approach can effectively alleviate the oracle problem and challenges surrounding a lack of specifications when verifying, validating, and evaluating large and complex software systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Violation of a metamorphic relation points to a defect (Zhou et al., 2015).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Zhou et al., 2015
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,421
(2017) · Cited by 16,319
Cohen, J. (1992) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep10768783 · Cited by 5,870
Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research
Levine, T. R., & Hullett, C. R. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00828.x · Cited by 998
An Efficient Boosting Algorithm for Combining Preferences
Freund, Y., Iyer, R. D., Schapire, R. E., & Singer, Y. (1998) · Cited by 806
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
Metamorphic Testing: A New Approach for Generating Next Test CasesPreprint
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 603
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 394
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388
Research works printed but not linked: 6
- denyer, 2014, Microsoft denies censoring results in the ‘uncensored’ version of Chinese-language Bing, The Washington Post
- kristof, 2009, Boycott Microsoft Bing, The New York Times
- chen, 0, Metamorphic testing: Applications and integration with other methods, Proc 12th Int Conf Quality Softw, 285
- 0, Standard—Adoption of ISO/IEC 25010-2010 Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models
- 0, IEEE Standards Association IEEE Standard—Adoption of ISO/IEC 15026-2 2011 Systems and Software Engineering—Systems and Software Assurance—Part 2 Assurance Case
- 2011, “Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models ’
Cited by
Sorted by citations
Items 101–163 of 163
Metamorphic Testing in Autonomous System Simulations
Adigun, J. G., Eisele, L., & Felderer, M. (2022) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa56994.2022.00059 · Cited by 4
Metamorphic Testing and Serverless Computing: A Basic Architecture
Yusyn, Y., & Заболотня, Т. (2022) · Informatica · doi:10.31449/inf.v46i6.4184 · Cited by 4
Applying Metamorphic Testing to e-Commerce Product Search Engines
Nagai, S., & Tsuchiya, T. (2018) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc.2018.00030 · Cited by 4
Drivence: Realistic Driving Sequence Synthesis for Testing Multi-Sensor Fusion Perception Systems
Gao, X., Wang, Z., Feng, Y., Wang, C., Zhou, Z., Huang, Y., Ma, L., Chen, Z., & Xu, B. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3650399 · Cited by 4
Can Large Language Models Discover Metamorphic Relations? A Large-Scale Empirical Study
Zhang, J., Sun, C., Liu, H., & Dong, S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00011 · Cited by 5
Automated Functional Testing of Search Engines using Metamorphic Testing
Wang, X., Yi, G., & Wang, Y. (2021) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c55045.2021.00014 · Cited by 3
Metamorphic Testing of Relation Extraction Models
Yuhe, S., Ding, Z., Huang, H., Zou, S., & Jiang, M. (2023) · Algorithms · doi:10.3390/a16020102 · Cited by 3
Metamorphic Testing for Verification and Fault Localization in Industrial Control Systems
Sudheerbabu, G., Ahmad, T., Truşcan, D., & Vain, J. (2023) · Springer eBooks · doi:10.1007/978-3-031-42212-6_5 · Cited by 5
MT-PART: Metamorphic-Testing-Based Adaptive Random Testing Through Partitioning
Ying, Z., Towey, D., Chen, T. Y., & Zhou, Z. Q. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00158 · Cited by 3
Testing convolutional neural network based deep learning systems: a statistical metamorphic approach
Rehman, F. U., & Izurieta, C. I. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2658 · Cited by 5
Learning I/O Variables from Scientific Software’s User Manuals
Peng, Z., Lin, X.-Y., Santhoshkumar, S. N., Niu, N., & Kanewala, U. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08760-8_42 · Cited by 3
Improve the Quality of ARC Systems Based on the Metamorphic Testing
Zhang, J., Jing, X., Zhang, W., Wang, H., & Dong, Y. (2016) · International Symposium on System and Software Reliability (ISSSR) · doi:10.1109/isssr.2016.029 · Cited by 4
A revisit of the integration of metamorphic testing and test suite based automated program repair
Jiang, M., Chen, T. Y., Kuo, F., Ding, Z., Choi, E.-H., & Mizuno, O. (2017) · doi:10.5555/3103620.3103628 · Cited by 3
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning ModelsPreprint
Meng, L., Li, Y., Chen, L., Wang, Z., Wu, D., Zhou, Y., & Xu, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04333 · Cited by 3
Summary of: Adaptive Metamorphic Testing with Contextual Bandits
Spieker, H., & Arnaud Gotlieb, B. (2021) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst49551.2021.00037 · Cited by 2
Less Is More: Feature Engineering for Fairness and Performance of Machine Learning Software
Meng, L., Li, Y., Chen, L., Ma, M., Zhou, Y., & Xu, B. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3730577 · Cited by 2
Proposal of a Method to Measure Test Suite Quality Attributes for White-Box Testing
Saputra, M. C., & Katayama, T. (2021) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2021.0120535 · Cited by 2
Stability evaluation for text localization systems via metamorphic testing
Yan, R., Wang, S., Yan, Y., Gao, H., & Yan, J. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111040 · Cited by 2
Wang, F., Ben, K., & Zhang, X. (2022) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/3524846.3527339 · Cited by 2
Evaluation of the Limit of Detection in Network Dataset Quality Assessment with PerQoDA
Wasielewska, K., Soukup, D., Čejka, T., & Camacho, J. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-23633-4_13 · Cited by 2
Boosting Metamorphic Relation Prediction via Code Representation Learning: An Empirical Study
Zheng, X., Jiang, M., & Zhou, Z. Q. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1889 · Cited by 2
Combating Missed Recalls in E-commerce Search: A CoT-Prompting Testing Approach
Wu, S., Hu, Y., Wang, Y., Gu, J., Meng, J., Fan, L., Luan, Z., Wang, X., & Zhou, Y. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663842 · Cited by 3
Metamorphic Testing of Code Search Engines
Ding, Z., Zhang, Q., & Jiang, M. (2020) · International Symposium on Theoretical Aspects of Software Engineering (TASE) · doi:10.1109/tase49443.2020.00032 · Cited by 2
Bin, Y., Chen, X., Li, W., & Tian, J. (2022) · Journal of Circuits Systems and Computers · doi:10.1142/s021812662250133x · Cited by 2
Cho, E., Shin, Y.-J., Hyun, S., Kim, H., & Bae, D. (2022) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec57359.2022.00033 · Cited by 2
Exploring Software Security Test Generation Techniques: Challenges and Opportunities
Alenezi, M., Akour, M., & Basit, H. A. (2021) · International Journal of Education and Information Technologies · doi:10.46300/9109.2021.15.11 · Cited by 2
Using Category Partition to Detect Metamorphic Relations
Pedram, S., & Labiche, Y. (2024) · doi:10.1145/3679006.3685068 · Cited by 1
MetBench: A Numerical Expression Metamorphic Relations Benchmark Data Set
Zhao, Y., Li, M., Su, Y., Zi, K., & Zeng, C. (2024) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa63982.2024.00026 · Cited by 1
Hoard, B. R. (2025) · Cogent Engineering · doi:10.1080/23311916.2025.2522652 · Cited by 1
Jiang, M., Hu, B., & Zhang, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107855 · Cited by 1
A framework for the acceptance testing of geospatial search engines
Herrera-Murillo, D. J., Nogueras-Iso, J., Power, P. A., Latre, M. Á., & Lopez-Pellicer, F. J. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106692 · Cited by 1
ISVMT: An approach of indicator systems validation based on metamorphic testing and data mutation
Ma, G., Yang, B., Xia, X., & Xu, L. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107944 · Cited by 1
An Approach for Iteratively Generating Adequate Tests in Metamorphic Testing: A Case Study
Ding, J., & Zhang, D. (2016) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2016.48 · Cited by 1
Iqbal, M. J. (2023) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met59151.2023.00008 · Cited by 1
Classification of Graphical User Interfaces Through Gaze-Based Features
Sharma, R., & Cecotti, H. (2019) · Communications in computer and information science · doi:10.1007/978-981-13-9181-1_1 · Cited by 1
Metamorphic Testing for Plant Identification Mobile Applications Based on Test Contexts
Guo, H., Tao, C., & Huang, Z. (2020) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-64214-3_15 · Cited by 1
Orchid: Building Dynamic Test Oracles with Training Bias for Improving Deep Neural Network Models
Wang, H., & Chan, W. K. (2021) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa52907.2021.00022 · Cited by 1
Yusyn, Y., & Заболотня, Т. (2022) · Herald of Khmelnytskyi National University Economic sciences · doi:10.31891/2307-5732-2022-305-1-107-115 · Cited by 1
Fu, S., Yang, X., Li, M., & Wang, F. (2021) · International Academic Exchange Conference on Science and Technology Innovation (IAECST) · doi:10.1109/iaecst54258.2021.9695750 · Cited by 1
Use of Metamorphic Relations as Knowledge Carriers to Train Deep Neural NetworksPreprint
Chen, T. Y., Poon, P., Qiu, K., Zheng, Z., & Zhou, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04718 · Cited by 2
Software quality: A Historical and Synthetic Content AnalysisPreprint
Kokol, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14598 · Cited by 1
Embedded Software Quality Evaluation Based on BP Neural Network and Causal InferencePreprint
Zheng, W., Zhu, J., Cai, F., Chen, Q., & Yang, F. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4098742 · Cited by 1
A Quality Evaluation Approach to Search Engines of Shopping Platforms
Chen, H., Chuanqi, T., & Gao, J. Z. (2017) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice.2017.45
Case studies of metamorphic tesing
Ma, W. (2017) · Research Online (University of Wollongong)
SENTHILNATHAN, S. (2017) · International Journal of Advanced Research in Computer Science · doi:10.26483/ijarcs.v8i9.5229
Automated program repair: a perspective from metamorphic testing
Jiang, M. (2017) · Swinburne figshare (Swinburne University of Technology) · doi:10.25916/sut.26278918
Li-kai, H., & Liu, J. (2019)
Xie, X., Poon, P., & Pullum, L. (2019) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3356773.3356810
Jarman, D., Smith, R., Johnston, O., Towey, D., & Zhou, Z. Q. (2020) · doi:10.1145/3387940.3391482
Using Metamorphic Relations to Verify and Enhance Artcode ClassificationPreprint
Xu, L., Towey, D., French, A. P., Benford, S., Zhou, Z. Q., & Chen, T. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02694
A Case Study of Testing an Image Recognition
Tao, C., & Tao, C. (2021) · doi:10.18293/jvlc2021-n2-194
SABER: Identifying SimilAr BEhavioR for Program Comprehension
Sridhar, A., Qiao, G., & Kaiser, G. E. (2020)
Metamorphic Relations Recognition Method Based on Node Features and Path Features
Zhang, T., Liu, J., Yang, X., Yan, S., & Li, M. (2022) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2219/1/012059
Zhu, J., Long, T., Wang, W., & Memon, A. M. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558941
Multi-dimensional Metamorphic Testing Method in Vehicle Intelligent Identification System
(2022) · Proceedings of WCSE 2022 Spring Event: 2022 9th International Conference on Industrial Engineering and Applications · doi:10.18178/wcse.2022.04.069
Emerging Aspects of Software Fault Localization
Tse, T. H., Lo, D., Gorce, A., Perscheid, M., Hirschfeld, R., & Wong, W. E. (2023) · doi:10.1002/9781119880929.ch13
Applying metamorphic testing to reliability calculating programs
Asaji, T., & Tsuchiya, T. (2023) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc59308.2023.00026
Using Metamorphic Testing to Identify Authentication Vulnerabilities in Android Mobile Applications
Júnior, M. C. (2024) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2024.tde-20082024-083506
LLMs in Debate: Does Arguing Make Them Better at Detecting Metamorphic Relations?
Bose, D. B., Alebachew, Y. B., & Brown, C. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00019
Ayundyahrini, M., Widianti, T., Firdaus, H., Azzumar, M., Ega, A. V., Rakhmawati, T., Damayanti, S., Sumaedi, S., Dinaseviani, A., Syahlani, N., et al. (2026) · International Journal of Quality & Reliability Management · doi:10.1108/ijqrm-09-2025-0310
Probing the Black Box: A Behavioral Framework for the Deep Verification of Random Forests
Zhang, Z., & Klabjan, D. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402443
Toha, A. H. C., liu, H., & Guo, M. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00370
Breaking the Test Oracle Barrier
Hazott, C., & Daniel, G. (2026) · Springer eBooks · doi:10.1007/978-3-032-23584-8_3
Metadata
- Authors
- Zhou, Z. Q., Xiang, S., & Chen, T. Y.
- Affiliations
- University of Wollongong (AU)
- Swinburne University of Technology (AU)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2016-03-01
- Volume
- 42
- Issue
- 3
- Pages
- 264-284
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2293505944
- Cited by
- 163
- References
- 48
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 17 · 2023: 9 · 2022: 25 · 2021: 25 · 2020: 15 · 2019: 18 · 2018: 12 · 2017: 17 · 2016: 10
- FWCI
- 54.68
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Research Online (University of Wollongong) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Swinburne figshare (Swinburne University of Technology) (submittedVersion)
- Funders
- Australian Research Council (LP100200208)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- metamorphic testing, search engines, oracle problem, software quality assessment, software verification, software validation, functional correctness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
