The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
Metamorphic Testing for Software Quality Assessment: A Study of Search Engines
Open accessAbstract
Metamorphic testing is a testing technique that can be used to verify the functional correctness of software in the absence of an ideal oracle. This paper extends metamorphic testing into a user-oriented approach to software verification, validation, and quality assessment, and conducts large scale empirical studies with four major web search engines: Google, Bing, Chinese Bing, and Baidu. These search engines are very difficult to test and assess using conventional approaches owing to the lack of an objective and generally recognized oracle. The results are useful for both search engine developers and users, and demonstrate that our approach can effectively alleviate the oracle problem and challenges surrounding a lack of specifications when verifying, validating, and evaluating large and complex software systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Violation of a metamorphic relation points to a defect (Zhou et al., 2015).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Zhou et al., 2015
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,421
(2017) · Cited by 16,319
Cohen, J. (1992) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep10768783 · Cited by 5,870
Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research
Levine, T. R., & Hullett, C. R. (2002) · Human Communication Research · doi:10.1111/j.1468-2958.2002.tb00828.x · Cited by 998
An Efficient Boosting Algorithm for Combining Preferences
Freund, Y., Iyer, R. D., Schapire, R. E., & Singer, Y. (1998) · Cited by 806
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
Metamorphic Testing: A New Approach for Generating Next Test CasesPreprint
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 603
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G. E., Xu, B., & Chen, T. Y. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 394
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388
Research works printed but not linked: 6
- denyer, 2014, Microsoft denies censoring results in the ‘uncensored’ version of Chinese-language Bing, The Washington Post
- kristof, 2009, Boycott Microsoft Bing, The New York Times
- chen, 0, Metamorphic testing: Applications and integration with other methods, Proc 12th Int Conf Quality Softw, 285
- 0, Standard—Adoption of ISO/IEC 25010-2010 Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models
- 0, IEEE Standards Association IEEE Standard—Adoption of ISO/IEC 15026-2 2011 Systems and Software Engineering—Systems and Software Assurance—Part 2 Assurance Case
- 2011, “Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—System and Software Quality Models ’
Cited by
Sorted by citations
Chen, T. Y., Kuo, F., Liu, H., Poon, P., Towey, D., Tse, T. H., & Zhou, Z. Q. (2018) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 691
A Survey on Metamorphic Testing
Segura, S., Fraser, G., Sánchez, A. B., & Ruiz–Cortés, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 571
Metamorphic testing of driverless cars
Zhou, Z. Q., & Sun, L. (2019) · Communications of the ACM · doi:10.1145/3241979 · Cited by 200
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Metamorphic Relations for Enhancing System Understanding and Use
Zhou, Z. Q., Sun, L., Chen, T. Y., & Towey, D. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876433 · Cited by 127
Metadata
- Authors
- Zhou, Z. Q., Xiang, S., & Chen, T. Y.
- Affiliations
- University of Wollongong (AU)
- Swinburne University of Technology (AU)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2016-03-01
- Volume
- 42
- Issue
- 3
- Pages
- 264-284
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2293505944
- Cited by
- 163
- References
- 48
- Citations by year
- 2026: 4 · 2025: 11 · 2024: 17 · 2023: 9 · 2022: 25 · 2021: 25 · 2020: 15 · 2019: 18 · 2018: 12 · 2017: 17 · 2016: 10
- FWCI
- 54.68
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Research Online (University of Wollongong) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Swinburne figshare (Swinburne University of Technology) (submittedVersion)
- Funders
- Australian Research Council (LP100200208)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- metamorphic testing, search engines, oracle problem, software quality assessment, software verification, software validation, functional correctness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
