The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Metamorphic Testing for Software Quality Assessment: A Study of Search Engines
Open accessAbstract
Metamorphic testing is a testing technique that can be used to verify the functional correctness of software in the absence of an ideal oracle. This paper extends metamorphic testing into a user-oriented approach to software verification, validation, and quality assessment, and conducts large scale empirical studies with four major web search engines: Google, Bing, Chinese Bing, and Baidu. These search engines are very difficult to test and assess using conventional approaches owing to the lack of an objective and generally recognized oracle. The results are useful for both search engine developers and users, and demonstrate that our approach can effectively alleviate the oracle problem and challenges surrounding a lack of specifications when verifying, validating, and evaluating large and complex software systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Violation of a metamorphic relation points to a defect (Zhou et al., 2015).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Zhou et al., 2015
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,421
(2017) · Cited by 16,319
Cohen, J. (1992) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep10768783 · Cited by 5,870
Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research
Levine, T. R., & Hullett, C. R. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00828.x · Cited by 998
An Efficient Boosting Algorithm for Combining Preferences
Freund, Y., Iyer, R. D., Schapire, R. E., & Singer, Y. (1998) · Cited by 806
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
Metamorphic Testing: A New Approach for Generating Next Test CasesPreprint
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 603
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 394
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388
Research works printed but not linked: 6
- denyer, 2014, Microsoft denies censoring results in the ‘uncensored’ version of Chinese-language Bing, The Washington Post
- kristof, 2009, Boycott Microsoft Bing, The New York Times
- chen, 0, Metamorphic testing: Applications and integration with other methods, Proc 12th Int Conf Quality Softw, 285
- 0, Standard—Adoption of ISO/IEC 25010-2010 Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models
- 0, IEEE Standards Association IEEE Standard—Adoption of ISO/IEC 15026-2 2011 Systems and Software Engineering—Systems and Software Assurance—Part 2 Assurance Case
- 2011, “Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models ’
Cited by
Sorted by citations
Chen, T. Y., Kuo, F., Liu, H., Poon, P., Towey, D., Tse, T. H., & Zhou, Z. Q. (2018) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 691
A Survey on Metamorphic Testing
Segura, S., Fraser, G., Sánchez, A. B., & Ruiz–Cortés, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 571
Metamorphic testing of driverless cars
Zhou, Z. Q., & Sun, L. (2019) · Communications of the ACM · doi:10.1145/3241979 · Cited by 200
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Metamorphic Relations for Enhancing System Understanding and Use
Zhou, Z. Q., Sun, L., Chen, T. Y., & Towey, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876433 · Cited by 127
Metamorphic Testing: Testing the Untestable
Segura, S., Towey, D., Zhou, Z. Q., & Chen, T. Y. (2018) · IEEE Software · doi:10.1109/ms.2018.2875968 · Cited by 126
Metamorphic Testing of RESTful Web APIs
Segura, S., Parejo, J. A., Troya, J., & Ruiz–Cortés, A. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2764464 · Cited by 126
Structure-invariant testing for machine translation
He, P., Meister, C., & Su, Z. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380339 · Cited by 94
Metamorphic Testing for Cybersecurity
Chen, T. Y., Kuo, F., Ma, W., Susilo, W., Towey, D., Voas, J. M., & Zhou, Z. Q. (2016) · Computer · doi:10.1109/mc.2016.176 · Cited by 85
Big Data Systems: A Software Engineering Perspective
Davoudian, A., & Liu, M. (2021) · ACM Computing Surveys · doi:10.1145/3408314 · Cited by 88
Metamorphic testing of RESTful web APIs
Segura, S., Parejo, J. A., Troya, J., & Ruiz–Cortés, A. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182528 · Cited by 77
Xie, X., Zhang, Z., Chen, T. Y., Liu, Y., Poon, P.-L., & Xu, B. (2020) · IEEE Transactions on Reliability · doi:10.1109/tr.2020.2972266 · Cited by 80
(7794023), X. X., (7695560), Z. Z., (9865748), T. C., (4702669), Y. L., (9827297), P. P., & (7787783), B. X. (2020) · Figshare · Cited by 57
Testing Machine Translation via Referential Transparency
He, P., Meister, C., & Su, Z. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00047 · Cited by 56
RBF-MLMR: A Multi-Label Metamorphic Relation Prediction Approach Using RBF Neural Network
Zhang, P., Zhou, X., Pelliccione, P., & LEUNG, H. K. N. (2017) · IEEE Access · doi:10.1109/access.2017.2758790 · Cited by 48
METRIC+: A Metamorphic Relation Identification Technique Based on Input plus Output Domains
Sun, C., Fu, A., Poon, P., Xie, X., Liu, H., & Chen, T. Y. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2934848 · Cited by 48
A metamorphic testing approach for supporting program repair without the need for a test oracle
Jiang, M., Chen, T. Y., Kuo, F., Towey, D., & Ding, Z. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.04.002 · Cited by 46
Adaptive metamorphic testing with contextual bandits
Spieker, H., & Arnaud Gotlieb, B. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110574 · Cited by 46
A Machine Learning Based Framework for Verification and Validation of Massive Scale Image Data
Ding, J., Hu, X., & Gudivada, V. N. (2017) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2017.2680460 · Cited by 46
Machine translation testing via pathological invariance
Gupta, S., He, P., Meister, C., & Su, Z. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409756 · Cited by 46
Theoretical and Empirical Analyses of the Effectiveness of Metamorphic Relation Composition
Qiu, K., Zheng, Z., Chen, T. Y., & Poon, P. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3009698 · Cited by 42
Zhang, Z., Wang, P., Guo, H., Wang, Z., Zhou, Y., & Huang, Z. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106701 · Cited by 45
Zhou, Z. Q., Tse, T. H., & Witheridge, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2915065 · Cited by 40
Testing multiple linear regression systems with metamorphic testing
Luu, Q.-H., Lau, M. F., Ng, S., & Chen, T. Y. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111062 · Cited by 41
SemMT: A Semantic-Based Testing Approach for Machine Translation Systems
Cao, J., Li, M., Li, Y., Wen, M., Cheung, S.-C., & Chen, H. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3490488 · Cited by 46
Automated inference of likely metamorphic relations for model transformations
Troya, J., Segura, S., & Ruiz–Cortés, A. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.05.043 · Cited by 39
Testing Your Question Answering Software via Asking Recursively
Chen, S., Jin, S., & Xie, X. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678670 · Cited by 43
Exploring the usefulness of unlabelled test cases in software fault localization
Zhang, X., Zheng, Z., & Cai, K. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.07.027 · Cited by 36
Metamorphic Testing of Navigation Software: A Pilot Study with Google Maps
Brown, J., Zhou, Z. Q., & Chow, Y.-W. (2018) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2018.713 · Cited by 34
A monte carlo method for metamorphic testing of machine translation services
Pesu, D., Zhou, Z. Q., Zhen, J., & Towey, D. (2018) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/3193977.3193980 · Cited by 34
Increasing Validity of Simulation Models Through Metamorphic Testing
Olsen, M., & Raunak, M. S. (2018) · IEEE Transactions on Reliability · doi:10.1109/tr.2018.2850315 · Cited by 32
An application of metamorphic testing for testing scientific software
Ding, J., Zhang, D., & Hu, X. (2016) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/2896971.2896981 · Cited by 31
Metamorphic Relation Patterns for Query-Based Systems
Segura, S., Durán, A., Troya, J., & Ruiz–Cortés, A. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00012 · Cited by 28
Metamorphic Relations for Detection of Performance Anomalies
Johnston, O., Jarman, D., Berry, J., Zhou, Z. Q., & Chen, T. Y. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00017 · Cited by 27
Perfect is the enemy of test oracle
Ibrahimzada, A. R., Varli, Y., Tekinoglu, D., & Jabbarvand, R. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549086 · Cited by 26
Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Models
Meng, L., Li, Y., Lin, C., Wang, Z., Wu, D., Zhou, Y., & Xu, B. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00045 · Cited by 28
A Template-Based Approach to Describing Metamorphic Relations
Segura, S., Durán, A., Troya, J., & Ruiz–Cortés, A. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.3 · Cited by 24
Iqbal, M., Han, J. C., Zhou, Z. Q., Towey, D., & Chen, T. Y. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107104 · Cited by 23
Metamorphic testing 20 years later
Segura, S., & Zhou, Z. Q. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3183468 · Cited by 21
Web application testing—Challenges and opportunities
Balsam, S., & Mishra, D. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112186 · Cited by 20
A Framework for Ensuring the Quality of a Big Data Service
Ding, J., Zhang, D., & Hu, X. (2016) · IEEE International Conference on Services Computing · doi:10.1109/scc.2016.18 · Cited by 20
Metamorphic testing for Adobe data analytics software
Jarman, D., Zhou, Z. Q., & Chen, T. Y. (2017) · Research Online (University of Wollongong) · doi:10.5555/3103620.3103630 · Cited by 20
Kıraç, F., Aktemur, B., & Sözer, H. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.06.023 · Cited by 19
Hierarchical metamorphic relations for testing scientific software
Lin, X.-Y., Simon, M. A., & Niu, N. (2018) · doi:10.1145/3194747.3194750 · Cited by 19
Input Test Suites for Program Repair: A Novel Construction Method Based on Metamorphic Relations
Jiang, M., Chen, T. Y., Zhou, Z. Q., & Ding, Z. (2020) · IEEE Transactions on Reliability · doi:10.1109/tr.2020.3003313 · Cited by 19
Metamorphic relation prioritization for effective regression testing
Srinivasan, M., & Kanewala, U. (2022) · Software Testing Verification and Reliability · doi:10.1002/stvr.1807 · Cited by 18
Metamorphic Testing for Adobe Data Analytics Software
Jarman, D., Zhou, Z. Q., & Chen, T. Y. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.1 · Cited by 18
On the effectiveness of testing sentiment analysis systems with metamorphic testing
Jiang, M., Chen, T. Y., & Wang, S. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106966 · Cited by 17
Metamorphic Testing of an Artificially Intelligent Chess Game
Liaqat, A., Sindhu, M. A., & Siddiqui, G. F. (2020) · IEEE Access · doi:10.1109/access.2020.3024929 · Cited by 17
Metamorphic Relations for Data Validation: A Case Study of Translated Text Messages
Yan, B., Yecies, B., & Zhou, Z. Q. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00018 · Cited by 17
Metamorphic Exploration of an Unsupervised Clustering Program
Yang, S., Towey, D., & Zhou, Z. Q. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00015 · Cited by 17
Metamorphic Relation Generation: State of the Art and Research Directions
Li, R., Liu, H., Poon, P., Towey, D., Sun, C., Zheng, Z., Zhou, Z. Q., & Chen, T. Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708521 · Cited by 16
The Impact of a Dot: Case Studies of a Noise Metamorphic Relation Pattern
Wu, C., Sun, L., & Zhou, Z. Q. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00011 · Cited by 15
Automatic Self-Validation for Code Coverage Profilers
Yang, Y., Jiang, Y., Zuo, Z., Wang, Y., Sun, H., Lu, H., Zhou, Y., & Xu, B. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00018 · Cited by 16
Using metamorphic relations to verify and enhance Artcode classification
Xu, L., Towey, D., French, A. P., Benford, S., Zhou, Z. Q., & Chen, T. Y. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111060 · Cited by 13
Automated generation of metamorphic relations for query-based systems
Segura, S., Alonso, J. C. G., Martin-Lopez, A., Durán, A., Troya, J., & Ruiz–Cortés, A. (2022) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/3524846.3527338 · Cited by 13
Enhancing supervised classifications with metamorphic relations
Xu, L., Towey, D., French, A. P., Benford, S., Zhou, Z. Q., & Chen, T. Y. (2018) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/3193977.3193978 · Cited by 13
Metamorphic testing of OpenStreetMap
Almendros-Jiménez, J. M., Becerra-Terón, A., Merayo, M. G., & Nunez, M. A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106631 · Cited by 12
A Metamorphic Testing Approach for Assessing Question Answering Systems
Tu, K., Jiang, M., & Ding, Z. (2021) · Mathematics · doi:10.3390/math9070726 · Cited by 15
Evaluating Natural Language Inference Models: A Metamorphic Testing Approach
Jiang, M., Bao, H., Tu, K., Zhang, X., & Ding, Z. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00033 · Cited by 11
Metamorphic testing: A new student engagement approach for a new software testing paradigm
Towey, D., Chen, T. Y., Kuo, F., Liu, H., & Zhou, Z. Q. (2016) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale.2016.7851797 · Cited by 11
Metamorphic testing of named entity recognition systems: A case study
Xu, Y., Zhou, Z. Q., Zhang, X., Wang, J., & Jiang, M. (2022) · IET Software · doi:10.1049/sfw2.12058 · Cited by 12
One step further: evaluating interpreters using metamorphic testing
Fan, M. Y., Wei, J., Jin, W., Zhou, X., Wei, W., & Liu, T. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534225 · Cited by 10
On the Investigation of Essential Diversities for Deep Learning Testing Criteria
Zhang, Z., & Xie, X. (2019) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2019.00056 · Cited by 10
Evaluation of Chinese Natural Language Processing System Based on Metamorphic Testing
Jin, L., Ding, Z., & Zhou, H. (2022) · Mathematics · doi:10.3390/math10081276 · Cited by 9
Hazott, C., Stögmüller, F., & Grosse, D. J. (2024) · Integration · doi:10.1016/j.vlsi.2024.102320 · Cited by 9
Testing bidirectional model transformation using metamorphic testing
He, X., Chen, X., Cai, S., Zhang, Y., & Huang, G. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.07.010 · Cited by 9
On Applying Metamorphic Testing: An Empirical Study on Academic Search Engines
de Andrade, S. A., Santos, Í., Brito, C., Junior, M., Souza, S. R. S., & Delamaro, M. E. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00010 · Cited by 9
A Machine Learning Approach for Developing Test Oracles for Testing Scientific Software
Ding, J., & Zhang, D. (2016) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2016-137 · Cited by 9
Metamorphic Testing for Processor Verification: A RISC-V Case Study at the Instruction Level
Riese, F., Herdt, V., Grose, D., & Drechsler, R. (2021) · IFIP/IEEE International Conference on Very Large Scale Integration · doi:10.1109/vlsi-soc53125.2021.9606997 · Cited by 8
Machine Translation Testing via Syntactic Tree Pruning
Zhang, Q., Zhai, J., Fang, C., Liu, J., Sun, W., Hu, H., & Wang, Q. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3640329 · Cited by 8
Rahman, K., Kahanda, I., & Kanewala, U. (2020) · doi:10.1145/3387940.3392250 · Cited by 8
An Interleaving Guided Metamorphic Testing Approach for Concurrent Programs
Sun, C., Dai, H., Geng, N., Liu, H., Chen, T. Y., Wu, P., Cai, Y., & Wang, J. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3607182 · Cited by 8
Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program
Ding, J., & Hu, X. (2016) · Software Quality Journal · doi:10.1007/s11219-016-9337-3 · Cited by 8
A template-based approach to describing metamorphic relations
Segura, S., Durán, A., Troya, J., & Ruiz–Cortés, A. (2017) · doi:10.5555/3103620.3103626 · Cited by 8
Zhang, Y., Chen, T. Y., Pike, M., Towey, D., Ying, Z., & Zhou, Z. Q. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107828 · Cited by 7
Ying, Z., Towey, D., Bellotti, A. G., Chen, T. Y., & Zhou, Z. Q. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107528 · Cited by 7
Scenario‐Driven Metamorphic Testing for Autonomous Driving Simulators
Zhang, Y., Towey, D., Pike, M., Han, J. C., Zhou, Z. Q., Yin, C., Wang, Q., & Chen, X. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1892 · Cited by 7
Towards effective metamorphic testing by algorithm stability for linear classification programs
Yang, Y., Li, Z., Wang, H., Xu, C., & Ma, X. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111012 · Cited by 7
Using Metamorphic Testing to Improve the Quality of Tags in OpenStreetMap
Almendros-Jiménez, J. M., Becerra-Terón, A., Merayo, M. G., & Nunez, M. A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3156182 · Cited by 7
qaAskeR$$^+$$: a novel testing method for question answering software via asking recursive questions
Xie, X., Jin, S., & Chen, S. (2023) · Automated Software Engineering · doi:10.1007/s10515-023-00380-2 · Cited by 8
Harnessing Multiple Source Test Cases in Metamorphic Testing: A Case Study in Bioinformatics
Tang, J. Y. S., Yang, A., Chen, T. Y., & Ho, J. W. K. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.4 · Cited by 7
Metamorphic Relation Patterns for Metamorphic Testing, Exploration and Robustness
Ying, Z., Towey, D., Bellotti, A. G., Chua, C., & Zhou, Z. Q. (2025) · Software Testing Verification and Reliability · doi:10.1002/stvr.70003 · Cited by 6
Evaluation of Metamorphic Testing for Edge Detection in MRI Brain Diagnostics
Jafari, F., Nadeem, A., & Zaman, Q. (2022) · Applied Sciences · doi:10.3390/app12178684 · Cited by 6
Verifying Embedded Graphics Libraries leveraging Virtual Prototypes and Metamorphic Testing
Hazott, C., Stögmüller, F., & Grosse, D. J. (2024) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac58780.2024.10473799 · Cited by 6
Balancing Effectiveness and Flakiness of Non-Deterministic Machine Learning Tests
Xia, C. S., Dutta, S., Misailovíc, S., Marinov, D., & Zhang, L. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00154 · Cited by 7
Metamorphic Testing of Mapping Software
Brown, J., Zhou, Z. Q., & Chow, Y.-W. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-28430-5_1 · Cited by 6
Identifying Metamorphic Relations: A Data Mutation Directed Approach
Sun, C., Jin, H., Wu, S., Fu, A., Wang, Z., & Chan, W. K. (2022) · Software Practice and Experience · doi:10.1002/spe.3280 · Cited by 5
MRGS‐ART: Metamorphic Relation and Group Selection Based on Adaptive Random Testing
Ying, Z., Towey, D., Bellotti, A. G., & Zhou, Z. Q. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1908 · Cited by 5
Identifying the Failure-Revealing Test Cases in Metamorphic Testing: A Statistical Approach
Zheng, Z., Ren, D., Liu, H., Chen, T. Y., & Li, T. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695990 · Cited by 5
MetaMFL: Metamorphic Multiple Fault Localization Without Test Oracles
Fu, L., Wu, Z., Lei, Y., & Yan, M. L. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3504400 · Cited by 5
ADVISOR: An Adjustable Framework for Test Oracle Automation of Visual Output Systems
Genc, A. E., Sözer, H., Kıraç, F., & Aktemur, B. (2019) · IEEE Transactions on Reliability · doi:10.1109/tr.2019.2957507 · Cited by 5
Combining Metamorphic Testing and Machine Learning to Enhance OpenStreetMap
Méndez, M., Becerra-Terón, A., Almendros-Jiménez, J. M., Merayo, M. G., & Nunez, M. A. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3379366 · Cited by 5
Addressing Data Quality Problems with Metamorphic Data Relations
Auer, F., & Felderer, M. (2019) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2019.00019 · Cited by 5
A Revisit of the Integration of Metamorphic Testing and Test Suite Based Automated Program Repair
Jiang, M., Chen, T. Y., Kuo, F., Ding, Z., Choi, E.-H., & Mizuno, O. (2017) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1109/met.2017.5 · Cited by 5
A metamorphic testing approach for event sequences
Chen, J., Wang, Y., Guo, Y., & Jiang, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0212476 · Cited by 4
Tabular‐expression‐based method for constructing metamorphic relations
Li, J., Liu, L., & Zhang, P. (2020) · Software Practice and Experience · doi:10.1002/spe.2818 · Cited by 4
Metamorphic Testing for Reliability in System of Systems
Chua, K. K. A., Bae, D., & Jee, E. (2021) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec53868.2021.00046 · Cited by 4
Tao, C., Guo, H., Zhang, J., & Huang, Z. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2444 · Cited by 4
Metamorphic relation automation: Rationale, challenges, and solution directions
Altamimi, E., Elkawakjy, A., & Catal, C. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2509 · Cited by 4
Metadata
- Authors
- Zhou, Z. Q., Xiang, S., & Chen, T. Y.
- Affiliations
- University of Wollongong (AU)
- Swinburne University of Technology (AU)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2016-03-01
- Volume
- 42
- Issue
- 3
- Pages
- 264-284
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2293505944
- Cited by
- 163
- References
- 48
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 17 · 2023: 9 · 2022: 25 · 2021: 25 · 2020: 15 · 2019: 18 · 2018: 12 · 2017: 17 · 2016: 10
- FWCI
- 54.68
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Research Online (University of Wollongong) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Swinburne figshare (Swinburne University of Technology) (submittedVersion)
- Funders
- Australian Research Council (LP100200208)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- metamorphic testing, search engines, oracle problem, software quality assessment, software verification, software validation, functional correctness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
