Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 665

The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.

Metamorphic Testing: A Review of Challenges and Opportunities

Open access
Chen, Tsong Yueh, Kuo, Fei-Ching, Liu, Huai, Poon, Pak-Lok, Towey, Dave, Tse, T. H., Zhou, Zhi Quan·2019·ACM Computing Surveys
DOI 10.1145/3143561
Open this paper in CoChat — free account

Abstract

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this article, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3143561

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 72 references

Research works printed but not linked: 27

  1. 10.1145/2479440.2482677 · doi:10.1145/2479440.2482677
  2. Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies . In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing , Vol. 249 . Springer, Berlin, 399--410. Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies. In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing, Vol. 249. Springer, Berlin, 399--410.
  3. 10.1109/COMPSAC.2006.161 · doi:10.1109/compsac.2006.161
  4. 10.1007/978-3-642-02138-1_19 · doi:10.1007/978-3-642-02138-1_19
  5. Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9. Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9. · doi:10.1007/s11432-015-5314-x
  6. 10.1109/TSE.2003.1214323 · doi:10.1109/tse.2003.1214323
  7. 10.1016/S0164-1212(01)00028-0 · doi:10.1016/s0164-1212(01)00028-0
  8. Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2008 . Case studies on testing with compositional metamorphic relations . Journal of Southeast University (English Edition) 24 , 4, 437 -- 443 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2008. Case studies on testing with compositional metamorphic relations. Journal of Southeast University (English Edition) 24, 4, 437--443.
  9. Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2009 . Survey of metamorphic testing . Journal of Frontiers of Computer Science and Technology 3 , 2, 130 -- 143 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2009. Survey of metamorphic testing. Journal of Frontiers of Computer Science and Technology 3, 2, 130--143.
  10. 10.1145/359488.359495 · doi:10.1145/359488.359495
  11. Mingyue Jiang , Tsong Yueh Chen , Fei-Ching Kuo , and Zuohua Ding . 2013 . Testing central processing unit scheduling algorithms using metamorphic testing . In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13) . IEEE Computer Society, Los Alamitos, CA, 530--536. Mingyue Jiang, Tsong Yueh Chen, Fei-Ching Kuo, and Zuohua Ding. 2013. Testing central processing unit scheduling algorithms using metamorphic testing. In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13). IEEE Computer Society, Los Alamitos, CA, 530--536.
  12. 10.1109/LCN.2011.6115306 · doi:10.1109/lcn.2011.6115306
  13. Mikael Lindvall , Dharmalingam Ganesan , Sigurthor Bjorgvinsson , Kristjan Jonsson , Haukur Steinn Logason , Frederik Dietrich , and Robert E. Wiegand . 2016. Agile metamorphic model-based testing . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16) . ACM, New York, NY, 26--32. Mikael Lindvall, Dharmalingam Ganesan, Sigurthor Bjorgvinsson, Kristjan Jonsson, Haukur Steinn Logason, Frederik Dietrich, and Robert E. Wiegand. 2016. Agile metamorphic model-based testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 26--32.
  14. 10.1109/CSEET.2010.28 · doi:10.1109/cseet.2010.28
  15. 10.1109/QSIC.2012.10 · doi:10.1109/qsic.2012.10
  16. 10.1145/2591062.2591109 · doi:10.1145/2591062.2591109
  17. Heng Lu , Wing Kwong Chan , and T. H. Tse . 2006. Testing context-aware middleware-centric programs: A data flow approach and an RFID-based experimentation . In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14) . ACM, New York, NY, 242--252. Heng Lu, Wing Kwong Chan, and T. H. Tse. 2006. Testing context-aware middleware-centric programs: A data flow approach and an RFID-based experimentation. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 242--252.
  18. 10.1109/UIC-ATC.2010.75 · doi:10.1109/uic-atc.2010.75
  19. Ieng Kei Mak . 1997. On the Effectiveness of Random Testing. Master’s Thesis. Department of Computer Science , University of Melbourne , Melbourne, Australia . Ieng Kei Mak. 1997. On the Effectiveness of Random Testing. Master’s Thesis. Department of Computer Science, University of Melbourne, Melbourne, Australia.
  20. 10.1108/ITP-01-2013-0004 · doi:10.1108/itp-01-2013-0004
  21. 10.4018/JOEUC.2017040102 · doi:10.4018/joeuc.2017040102
  22. Laura L. Pullum and Ozgur Ozmen. 2012. Early results from metamorphic testing of epidemiological models . In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12) . IEEE Computer Society, Los Alamitos, CA, 62--67. Laura L. Pullum and Ozgur Ozmen. 2012. Early results from metamorphic testing of epidemiological models. In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12). IEEE Computer Society, Los Alamitos, CA, 62--67.
  23. Arvind Ramanathan , Chad A. Steed , and Laura L. Pullum . 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics . In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12) . IEEE Computer Society, Los Alamitos, CA, 68--73. Arvind Ramanathan, Chad A. Steed, and Laura L. Pullum. 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics. In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12). IEEE Computer Society, Los Alamitos, CA, 68--73.
  24. 10.1109/APSEC.2010.39 · doi:10.1109/apsec.2010.39
  25. 10.1109/TALE.2015.7386036 · doi:10.1109/tale.2015.7386036
  26. Michael Troup , Andrian Yang , Amir Hossein Kamali , Eleni Giannoulatou , Tsong Yueh Chen , and Joshua W. K. Ho . 2016. A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16) . ACM, New York, NY, 33--36. Michael Troup, Andrian Yang, Amir Hossein Kamali, Eleni Giannoulatou, Tsong Yueh Chen, and Joshua W. K. Ho. 2016. A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 33--36.
  27. 10.1109/TSA.2015.13 · doi:10.1109/tsa.2015.13

Cited by

See all 665

Metadata

Authors
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q.
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2018-01-04
Volume
51
Issue
1
Pages
1-27
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3143561
OpenAlex
W2782311202
Cited by
665
References
72
FWCI
198.47
Type
journal article
Language
en
Open access
green
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Funders
  • General Research Fund of the Research Grants Council of Hong Kong (716612)
  • International Doctoral Innovation Centre
  • University of Nottingham
  • Ningbo Science and Technology Bureau
  • Australian Research Council (LP160101691)
  • Artificial Intelligence and Optimisation Research Group of University of Nottingham Ningbo China
  • Ningbo Education Bureau

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Metamorphic Testing: A Review of Challenges and Opportunities. https://app.cochat.ai/papers/10.1145/3143561
The paper
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019). Metamorphic Testing: A Review of Challenges and Opportunities. ACM Computing Surveys. https://doi.org/10.1145/3143561

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.