The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Metamorphic Testing for Software Quality Assessment: A Study of Search Engines
Open accessAbstract
Metamorphic testing is a testing technique that can be used to verify the functional correctness of software in the absence of an ideal oracle. This paper extends metamorphic testing into a user-oriented approach to software verification, validation, and quality assessment, and conducts large scale empirical studies with four major web search engines: Google, Bing, Chinese Bing, and Baidu. These search engines are very difficult to test and assess using conventional approaches owing to the lack of an objective and generally recognized oracle. The results are useful for both search engine developers and users, and demonstrate that our approach can effectively alleviate the oracle problem and challenges surrounding a lack of specifications when verifying, validating, and evaluating large and complex software systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Violation of a metamorphic relation points to a defect (Zhou et al., 2015).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Zhou et al., 2015
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,421
(2017) · Cited by 16,319
Cohen, J. (1992) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep10768783 · Cited by 5,870
Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research
Levine, T. R., & Hullett, C. R. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00828.x · Cited by 998
An Efficient Boosting Algorithm for Combining Preferences
Freund, Y., Iyer, R. D., Schapire, R. E., & Singer, Y. (1998) · Cited by 806
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
Metamorphic Testing: A New Approach for Generating Next Test CasesPreprint
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 603
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 394
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388
Named entity recognition in query
Guo, J., Xu, G., Cheng, X., & Li, H. (2009) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1571941.1571989 · Cited by 384
Software testing and analysis: process, principles, and techniques
Pezzè, M., & Young, M. (2008) · Choice Reviews Online · doi:10.5860/choice.46-0935 · Cited by 342
How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?
Liu, H., Kuo, F., Towey, D., & Chen, T. Y. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.46 · Cited by 207
Web Engineering: Modelling and Implementing Web Applications
Rossi, G. H., Schwabe, D., Pastor, Ó., & Olsina, L. A. (2007) · Human-computer interaction series · doi:10.1007/978-1-84628-923-1 · Cited by 184
Fault-based testing without the need of oracles
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · Information and Software Technology · doi:10.1016/s0950-5849(02)00129-5 · Cited by 179
Random Testing: Theoretical Results and Practical Implications
Arcuri, A., Iqbal, M. Z., & Briand, L. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.121 · Cited by 178
Estimating the Query Difficulty for Information Retrieval
Carmel, D., & Yom‐Tov, E. (2010) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00235ed1v01y201004icr015 · Cited by 162
An innovative approach for testing bioinformatics programs using metamorphic testing
Chen, T. Y., Ho, J. W. K., Liu, H., & Xie, X. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-24 · Cited by 159
Software Product Quality Control
Wagner, S. (2013) · Springer eBooks · doi:10.1007/978-3-642-38571-1 · Cited by 133
Search-based inference of polynomial metamorphic relations
Zhang, J. M., Chen, J., Hao, D., Xiong, Y., Xie, B., Zhang, L., & Mei, H. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642994 · Cited by 116
Metamorphic slice: An application in spectrum-based fault localization
Xie, X., Wong, W. E., Chen, T. Y., & Xu, B. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.08.008 · Cited by 112
Case studies on the selection of useful relations in metamorphic testing
Chen, T. Y., Huang, D., Tse, T.-H., & Zhou, Z. Q. (2004) · Swinburne figshare (Swinburne University of Technology) · Cited by 108
Fundamentals of Software Engineering
O’Regan, G. (2019) · Undergraduate topics in computer science · doi:10.1007/978-3-030-28494-7_2 · Cited by 101
An upper bound on software testing effectiveness
Chen, T. Y., & Merkel, R. (2008) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1363102.1363107 · Cited by 99
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/566171.566202 · Cited by 86
Automatic system testing of programs without test oracles
Murphy, C., Shen, K., & Kaiser, G. E. (2009) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/1572272.1572295 · Cited by 85
Automated functional testing of online search services
Zhou, Z. Q., Zhang, S., Hagenbuchner, M., Tse, T. H., Kuo, F., & Chen, T. Y. (2010) · Software Testing Verification and Reliability · doi:10.1002/stvr.437 · Cited by 85
Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.23 · Cited by 83
Ieong, S., Mishra, N., Sadikov, E., & Zhang, L. (2012) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2124295.2124345 · Cited by 55
Cao, Y., Zhou, Z. Q., & Chen, T. Y. (2013) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2013.43 · Cited by 50
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/566172.566202 · Cited by 48
Metamorphic model-based testing applied on NASA DAT: an experience report
Lindvall, M., Ganesan, D., Ardal, R., & Wiegand, R. E. (2015) · doi:10.5555/2819009.2819030 · Cited by 47
Metamorphic Model-Based Testing Applied on NASA DAT -- An Experience Report
Lindvall, M., Ganesan, D., Ardal, R., & Wiegand, R. E. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.348 · Cited by 45
Oracles for Distributed Testing
Hierons, R. M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.45 · Cited by 43
Automated functional testing of web search engines in the absence of an oracle
Zho, Z. Q. Z., Tse, T. H., Kuo, F., & Chen, T. Y. (2007) · Swinburne figshare (Swinburne University of Technology) · Cited by 33
An effective testing method for end-user programmers
Chen, T. Y., Kuo, F., & Zhou, Z. Q. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083236 · Cited by 32
Soboroff, I. M. (2006) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1148170.1148220 · Cited by 30
A methodology for validating cloud models using metamorphic testing
Núñez, A., & Hierons, R. M. (2014) · Annals of Telecommunications · doi:10.1007/s12243-014-0442-7 · Cited by 25
Automated metamorphic testing of variability analysis tools
Segura, S., Durán, A., Sánchez, A. B., Berre, D. L., Lonca, E., & Ruiz–Cortés, A. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1566 · Cited by 21
An effective testing method for end-user programmers
Chen, T. Y., Kuo, F., & Zhou, Z. Q. (2005) · doi:10.1145/1083231.1083236 · Cited by 15
Mining Iterative Generators and Representative Rules for Software Specification Discovery
Lo, D., Li, J., wong, I., & Khoo, S. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2010.24 · Cited by 13
If You Ask Nicely, I will Answer: Semantic Search and Today's Search Engines
Signorini, A., & Imieliński, T. (2009) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2009.31 · Cited by 12
T-DOC: A Tool for the Automatic Generation of Testing Documentation for OSS Products
Morasca, S., Taibi, D., & Tosi, D. (2010) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-3-642-13244-5_16 · Cited by 9
Extraction and Evaluation of Candidate Named Entities in Search Engine Queries
Alasiry, A., Levene, M., & Poulovassilis, A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35063-4_35 · Cited by 8
Structured web search in small domain
Qader, M. A., Rahman, S., & Hasan, M. (2014) · doi:10.1109/iccitechn.2014.6997348 · Cited by 1
Research works printed but not linked: 6
- denyer, 2014, Microsoft denies censoring results in the ‘uncensored’ version of Chinese-language Bing, The Washington Post
- kristof, 2009, Boycott Microsoft Bing, The New York Times
- chen, 0, Metamorphic testing: Applications and integration with other methods, Proc 12th Int Conf Quality Softw, 285
- 0, Standard—Adoption of ISO/IEC 25010-2010 Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models
- 0, IEEE Standards Association IEEE Standard—Adoption of ISO/IEC 15026-2 2011 Systems and Software Engineering—Systems and Software Assurance—Part 2 Assurance Case
- 2011, “Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models ’
Cited by
Sorted by citations
Chen, T. Y., Kuo, F., Liu, H., Poon, P., Towey, D., Tse, T. H., & Zhou, Z. Q. (2018) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 691
A Survey on Metamorphic Testing
Segura, S., Fraser, G., Sánchez, A. B., & Ruiz–Cortés, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 571
Metamorphic testing of driverless cars
Zhou, Z. Q., & Sun, L. (2019) · Communications of the ACM · doi:10.1145/3241979 · Cited by 200
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Metamorphic Relations for Enhancing System Understanding and Use
Zhou, Z. Q., Sun, L., Chen, T. Y., & Towey, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876433 · Cited by 127
Metadata
- Authors
- Zhou, Z. Q., Xiang, S., & Chen, T. Y.
- Affiliations
- University of Wollongong (AU)
- Swinburne University of Technology (AU)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2016-03-01
- Volume
- 42
- Issue
- 3
- Pages
- 264-284
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2293505944
- Cited by
- 163
- References
- 48
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 17 · 2023: 9 · 2022: 25 · 2021: 25 · 2020: 15 · 2019: 18 · 2018: 12 · 2017: 17 · 2016: 10
- FWCI
- 54.68
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Research Online (University of Wollongong) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Swinburne figshare (Swinburne University of Technology) (submittedVersion)
- Funders
- Australian Research Council (LP100200208)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- metamorphic testing, search engines, oracle problem, software quality assessment, software verification, software validation, functional correctness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
