The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Metamorphic Testing: A Review of Challenges and Opportunities
Open accessAbstract
Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this article, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Buyya, R., Yeo, C. S., Venugopal, S., Broberg, J., & Brandic, I. (2009) · Future Generation Computer Systems · doi:10.1016/j.future.2008.12.001 · Cited by 5,937
Software unit test coverage and adequacy
Zhu, H., Hall, P. A. V., & May, J. H. R. (1997) · ACM Computing Surveys · doi:10.1145/267580.267590 · Cited by 1,247
Encyclopedia of Software Engineering
(2002) · Wiley eBooks · doi:10.1002/0471028959 · Cited by 1,161
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
Symbolic execution for software testing: three decades later
Cadar, C., & Sen, K. (2013) · Communications of the ACM · doi:10.1145/2408776.2408795 · Cited by 776
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Dijkstra, E. W. (1972) · Communications of the ACM · doi:10.1145/355604.361591 · Cited by 705
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
A Survey on Metamorphic Testing
Segura, S., Fraser, G., Sanchez, A. B., & Ruiz-Cortes, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 563
Verifying and Validating Software Requirements and Design Specifications
Boehm, B. W. (1984) · IEEE Software · doi:10.1109/ms.1984.233702 · Cited by 513
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G., Xu, B., & Chen, T. Y. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 389
Self-citationCompiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388
Data diversity: an approach to software fault tolerance
Ammann, P., & KNIGHT, J. C. (1988) · IEEE Transactions on Computers · doi:10.1109/12.2185 · Cited by 384
Adaptive Random Testing: The ART of test case diversity
Chen, T. Y., Kuo, F.-C., Merkel, R. G., & Tse, T. H. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2009.02.022 · Cited by 353
Self-citationA practical approach to programming with assertions
Rosenblum, D. S. (1995) · IEEE Transactions on Software Engineering · doi:10.1109/32.341844 · Cited by 334
A brief survey of program slicing
Xu, B., Qian, J., Zhang, X., Wu, Z., & Chen, L. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1050849.1050865 · Cited by 313
Sabharwal, N., Rathore, R., & Agrawal, U. (2022) · Hands-On Guide to AgileOps · doi:10.1007/978-1-4842-7505-4_5 · Cited by 254
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594334 · Cited by 218
Software testing: a research travelogue (2000–2014)
Orso, A., & Rothermel, G. (2014) · Future of Software Engineering Proceedings · doi:10.1145/2593882.2593885 · Cited by 211
Achievements, Open Problems and Challenges for Search Based Software Testing
Harman, M., Jia, Y., & Zhang, Y. (2015) · 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2015.7102580 · Cited by 209
How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?
Liu, H., Kuo, F.-C., Towey, D., & Chen, T. Y. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.46 · Cited by 206
Self-citationFault-based testing without the need of oracles
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2003) · Information and Software Technology · doi:10.1016/s0950-5849(02)00129-5 · Cited by 179
Self-citationProperties of Machine Learning Applications for Use in Metamorphic Testing
Murphy, C., Kaiser, G. E., & Hu, L. (2008) · doi:10.7916/d8xk8pfd · Cited by 164
Metamorphic testing for software quality assessment: A study of search engines
Zhou, Z. Q., Xiang, S., & Chen, T. Y. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2478001 · Cited by 163
An innovative approach for testing bioinformatics programs using metamorphic testing
Chen, T. Y., Ho, J. W., Liu, H., & Xie, X. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-24 · Cited by 159
Self-citationLidbury, C., Lascu, A., Chong, N., & Donaldson, A. F. (2015) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2737924.2737986 · Cited by 157
Andalia, F. (2017) · IJNMT (International Journal of New Media Technology) · doi:10.31937/ijnmt.v4i2.660 · Cited by 134
Search-based inference of polynomial metamorphic relations
Zhang, J. M., Chen, J., Hao, D., Xiong, Y., Xie, B., Zhang, L., & Mei, H. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642994 · Cited by 121
A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications
Chan, W. K., Cheung, S.-C., & Leung, K. R. P. H. (2007) · International Journal of Web Services Research · doi:10.4018/jwsr.2007040103 · Cited by 118
Metamorphic slice: An application in spectrum-based fault localization
Xie, X., Wong, W. E., Chen, T. Y., & Xu, B. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.08.008 · Cited by 112
Self-citationCase studies on the selection of useful relations in metamorphic testing
Chen, T. Y., Huang, D., Tse, T.-H., & Zhou, Z. Q. (2004) · Swinburne figshare (Swinburne University of Technology) · Cited by 108
Self-citationKanewala, U., Bieman, J. M., & Ben‐Hur, A. (2015) · Software Testing Verification and Reliability · doi:10.1002/stvr.1594 · Cited by 107
Arnaud Gotlieb, B., & Botella, B. (2004) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.2003.1245319 · Cited by 96
Carving differential unit test cases from system test cases
Elbaum, S. G., Chin, H. N., Dwyer, M. B., & Dokulil, J. (2006) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1181775.1181806 · Cited by 92
Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/566171.566202 · Cited by 87
Self-citationMETRIC: METamorphic Relation Identification based on the Category-choice framework
Chen, T. Y., Poon, P., & Xie, X. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.07.037 · Cited by 85
Self-citationAutomated functional testing of online search services
Zhou, Z. Q., Zhang, S., Hagenbuchner, M., Tse, T. H., Kuo, F., & Chen, T. Y. (2010) · Software Testing Verification and Reliability · doi:10.1002/stvr.437 · Cited by 85
Self-citationMetamorphic Testing for Cybersecurity
Chen, T. Y., Kuo, F., Ma, W., Susilo, W., Towey, D., Voas, J. M., & Zhou, Z. Q. (2016) · Computer · doi:10.1109/mc.2016.176 · Cited by 85
Self-citationSemi-Proving: An Integrated Method for Program Proving, Testing, and Debugging
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.23 · Cited by 83
Self-citationTesting context-sensitive middleware-based software applications
Tse, T. H., Yau, S. S., Chan, W. K., Lu, H. Z., & Chen, T. Y. (2004) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.2004.1342879 · Cited by 82
Self-citationAn Empirical Study on the Selection of Good Metamorphic Relations
Mayer, J., & Guderlei, R. (2006) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2006.24 · Cited by 79
Automated metamorphic testing on the analyses of feature models
Segura, S., Hierons, R. M., Benavides, D., & Ruiz–Cortés, A. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.11.002 · Cited by 71
Research Directions for Engineering Big Data Analytics Software
Otero, C. E., & Peter, A. M. (2014) · IEEE Intelligent Systems · doi:10.1109/mis.2014.76 · Cited by 70
Guderlei, R., & Mayer, J. (2007) · Proceedings of the ... International Conference on Quality Software/Proceedings · doi:10.1109/qsic.2007.4385527 · Cited by 66
INTEGRATION TESTING OF CONTEXT-SENSITIVE MIDDLEWARE-BASED APPLICATIONS: A METAMORPHIC APPROACH
Chan, W. K., Chen, T. Y., Lu, H. Z., Tse, T. H., & Yau, S. S. (2006) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194006002951 · Cited by 61
Self-citationSpectrum-Based Fault Localization: Testing Oracles are No Longer Mandatory
Xie, X., Wong, W. E., Chen, T. Y., & Xu, B. (2011) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2011.20 · Cited by 54
Self-citationMetamorphic testing and testing with special values
Chen, T. Y., Kuo, F., Liu, Y., & Tang, A. (2024) · Swinburne figshare (Swinburne University of Technology) · doi:10.25916/sut.26291794 · Cited by 51
Self-citationCao, Y., Zhou, Z. Q., & Chen, T. Y. (2013) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2013.43 · Cited by 50
Self-citationPezzè, M., & Zhang, C. (2014) · Advances in computers · doi:10.1016/b978-0-12-800160-8.00001-2 · Cited by 48
Software testing using model programs
Manolache, L.-I., & Kourie, D. G. (2001) · Software Practice and Experience · doi:10.1002/spe.409 · Cited by 48
Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing
Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/566172.566202 · Cited by 48
Self-citationMetamorphic model-based testing applied on NASA DAT: an experience report
Lindvall, M., Ganesan, D., Ardal, R., & Wiegand, R. E. (2015) · doi:10.5555/2819009.2819030 · Cited by 47
Automated Test Data Generation on the Analyses of Feature Models: A Metamorphic Testing Approach
Segura, S., Hierons, R. M., Benavides, D., & Ruiz–Cortés, A. (2010) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2010.20 · Cited by 47
A metamorphic testing approach for supporting program repair without the need for a test oracle
Jiang, M., Chen, T. Y., Kuo, F., Towey, D., & Ding, Z. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.04.002 · Cited by 46
Self-citationA Machine Learning Based Framework for Verification and Validation of Massive Scale Image Data
Ding, J., Hu, X., & Gudivada, V. N. (2017) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2017.2680460 · Cited by 46
Metamorphic Testing for Web Services: Framework and a Case Study
Sun, C., Wang, G., Mu, B., Liu, H., Wang, Z., & Chen, T. Y. (2011) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2011.65 · Cited by 44
Self-citationA Cost-Effective Random Testing Method for Programs with Non-Numeric Inputs
Barus, A., Chen, T. Y., Kuo, F., Liu, H., Merkel, R., & Rothermel, G. (2016) · IEEE Transactions on Computers · doi:10.1109/tc.2016.2547380 · Cited by 42
Self-citationAn empirical comparison between direct and indirect test result checking approaches
Hu, P., Zhang, Z., Chan, W. K., & Tse, T. H. (2006) · doi:10.1145/1188895.1188901 · Cited by 37
Self-citationOn effective testing of health care simulation software
Murphy, C., Raunak, M. S., King, A., Chen, S., Imbriano, C., Kaiser, G. E., Lee, I., Sokolsky, O. V., Clarke, L. A., & Osterweil, L. J. (2011) · doi:10.1145/1987993.1988003 · Cited by 37
Dataset Coverage for Testing Machine Learning Computer Programs
Nakajima, S., & Bui, H. N. (2016) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2016.049 · Cited by 35
An effective testing method for end-user programmers
Chen, T. Y., Kuo, F., & Zhou, Z. Q. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083236 · Cited by 33
Self-citationde Oliveira, R. A. P., Kanewala, U., & Nardi, P. A. (2014) · Advances in computers · doi:10.1016/b978-0-12-800160-8.00003-6 · Cited by 32
Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking
Zhang, Z., Chan, W. K., Tse, T. H., & Pei-Feng, H. U. (2009) · Journal of Software · doi:10.3724/sp.j.1001.2009.00578 · Cited by 26
Self-citationImpacts of Test Suite's Class Imbalance on Spectrum-Based Fault Localization Techniques
Rao, P. C., Zheng, Z., Chen, T. Y., Wang, N., & Cai, K. (2013) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2013.18 · Cited by 25
Self-citationThe impact of source test case selection on the effectiveness of metamorphic testing
Barus, A., Chen, T. Y., Kuo, F., Liu, H., & Schmidt, H. W. (2016) · IEEE/ACM International Workshop on Metamorphic Testing (MET) · doi:10.1145/2896971.2896977 · Cited by 21
Self-citationConcolic Metamorphic Debugging
Jin, H., Jiang, Y., Liu, N., Xu, C., Ma, X., & Lu, J. J. (2015) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2015.79 · Cited by 15
An effective testing method for end-user programmers
Chen, T. Y., Kuo, F., & Zhou, Z. Q. (2005) · doi:10.1145/1083231.1083236 · Cited by 15
Self-citationVERIFICATION OF PHYLOGENETIC INFERENCE PROGRAMS USING METAMORPHIC TESTING
SADI, M. S., Kuo, F., Ho, J. W. K., Charleston, M., & Chen, T. Y. (2011) · Journal of Bioinformatics and Computational Biology · doi:10.1142/s021972001100563x · Cited by 14
Self-citationAchievements and Challenges of Metamorphic Testing
Hui, Z., & Huang, S. (2013) · World Congress on Software Engineering (WCSE) · doi:10.1109/wcse.2013.16 · Cited by 11
Metamorphic testing: A new student engagement approach for a new software testing paradigm
Towey, D., Chen, T. Y., Kuo, F., Liu, H., & Zhou, Z. Q. (2016) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale.2016.7851797 · Cited by 11
Self-citationEffectiveness of Teaching Metamorphic Testing, Part II
Mishra, K. S., Kaiser, G. E., & Sheth, S. (2013) · doi:10.7916/d8902c8q · Cited by 5
Research works printed but not linked: 27
- 10.1145/2479440.2482677 · doi:10.1145/2479440.2482677
- Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies . In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing , Vol. 249 . Springer, Berlin, 399--410. Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies. In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing, Vol. 249. Springer, Berlin, 399--410.
- 10.1109/COMPSAC.2006.161 · doi:10.1109/compsac.2006.161
- 10.1007/978-3-642-02138-1_19 · doi:10.1007/978-3-642-02138-1_19
- Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9. Tsong Yueh Chen Fei-Ching Kuo Dave Towey and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58 5 052104:1--052104:9. · doi:10.1007/s11432-015-5314-x
- 10.1109/TSE.2003.1214323 · doi:10.1109/tse.2003.1214323
- 10.1016/S0164-1212(01)00028-0 · doi:10.1016/s0164-1212(01)00028-0
- Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2008 . Case studies on testing with compositional metamorphic relations . Journal of Southeast University (English Edition) 24 , 4, 437 -- 443 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2008. Case studies on testing with compositional metamorphic relations. Journal of Southeast University (English Edition) 24, 4, 437--443.
- Guowei Dong , Baowen Xu , Lin Chen , Changhai Nie , and Lulu Wang . 2009 . Survey of metamorphic testing . Journal of Frontiers of Computer Science and Technology 3 , 2, 130 -- 143 . Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2009. Survey of metamorphic testing. Journal of Frontiers of Computer Science and Technology 3, 2, 130--143.
- 10.1145/359488.359495 · doi:10.1145/359488.359495
- Mingyue Jiang , Tsong Yueh Chen , Fei-Ching Kuo , and Zuohua Ding . 2013 . Testing central processing unit scheduling algorithms using metamorphic testing . In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13) . IEEE Computer Society, Los Alamitos, CA, 530--536. Mingyue Jiang, Tsong Yueh Chen, Fei-Ching Kuo, and Zuohua Ding. 2013. Testing central processing unit scheduling algorithms using metamorphic testing. In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13). IEEE Computer Society, Los Alamitos, CA, 530--536.
- 10.1109/LCN.2011.6115306 · doi:10.1109/lcn.2011.6115306
- Mikael Lindvall , Dharmalingam Ganesan , Sigurthor Bjorgvinsson , Kristjan Jonsson , Haukur Steinn Logason , Frederik Dietrich , and Robert E. Wiegand . 2016. Agile metamorphic model-based testing . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16) . ACM, New York, NY, 26--32. Mikael Lindvall, Dharmalingam Ganesan, Sigurthor Bjorgvinsson, Kristjan Jonsson, Haukur Steinn Logason, Frederik Dietrich, and Robert E. Wiegand. 2016. Agile metamorphic model-based testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 26--32.
- 10.1109/CSEET.2010.28 · doi:10.1109/cseet.2010.28
- 10.1109/QSIC.2012.10 · doi:10.1109/qsic.2012.10
- 10.1145/2591062.2591109 · doi:10.1145/2591062.2591109
- Heng Lu , Wing Kwong Chan , and T. H. Tse . 2006. Testing context-aware middleware-centric programs: A data flow approach and an RFID-based experimentation . In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14) . ACM, New York, NY, 242--252. Heng Lu, Wing Kwong Chan, and T. H. Tse. 2006. Testing context-aware middleware-centric programs: A data flow approach and an RFID-based experimentation. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 242--252.
- 10.1109/UIC-ATC.2010.75 · doi:10.1109/uic-atc.2010.75
- Ieng Kei Mak . 1997. On the Effectiveness of Random Testing. Master’s Thesis. Department of Computer Science , University of Melbourne , Melbourne, Australia . Ieng Kei Mak. 1997. On the Effectiveness of Random Testing. Master’s Thesis. Department of Computer Science, University of Melbourne, Melbourne, Australia.
- 10.1108/ITP-01-2013-0004 · doi:10.1108/itp-01-2013-0004
- 10.4018/JOEUC.2017040102 · doi:10.4018/joeuc.2017040102
- Laura L. Pullum and Ozgur Ozmen. 2012. Early results from metamorphic testing of epidemiological models . In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12) . IEEE Computer Society, Los Alamitos, CA, 62--67. Laura L. Pullum and Ozgur Ozmen. 2012. Early results from metamorphic testing of epidemiological models. In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12). IEEE Computer Society, Los Alamitos, CA, 62--67.
- Arvind Ramanathan , Chad A. Steed , and Laura L. Pullum . 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics . In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12) . IEEE Computer Society, Los Alamitos, CA, 68--73. Arvind Ramanathan, Chad A. Steed, and Laura L. Pullum. 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics. In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12). IEEE Computer Society, Los Alamitos, CA, 68--73.
- 10.1109/APSEC.2010.39 · doi:10.1109/apsec.2010.39
- 10.1109/TALE.2015.7386036 · doi:10.1109/tale.2015.7386036
- Michael Troup , Andrian Yang , Amir Hossein Kamali , Eleni Giannoulatou , Tsong Yueh Chen , and Joshua W. K. Ho . 2016. A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline . In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16) . ACM, New York, NY, 33--36. Michael Troup, Andrian Yang, Amir Hossein Kamali, Eleni Giannoulatou, Tsong Yueh Chen, and Joshua W. K. Ho. 2016. A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 33--36.
- 10.1109/TSA.2015.13 · doi:10.1109/tsa.2015.13
Cited by
Bond, M., Buntins, K., Bedenlier, S., Zawacki-Richter, O., & Kerres, M. (2020) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0176-8 · Cited by 1,097
Trustworthy Artificial Intelligence: A Review
Kaur, D., Uslu, S., Rittichier, K. J., & Durresi, A. (2023) · ACM Computing Surveys · doi:10.1145/3491209 · Cited by 709
Privacy preservation in federated learning: An insightful survey from the GDPR perspective
Truong, N., Sun, K., Wang, S., Guitton, F., & Guo, Y. (2021) · Computers & Security · doi:10.1016/j.cose.2021.102402 · Cited by 399
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267
Deep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167
Metadata
- Authors
- Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q.
- Venue
- ACM Computing Surveys
- Published
- 2018-01-04
- Volume
- 51
- Issue
- 1
- Pages
- 1-27
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2782311202
- Cited by
- 665
- References
- 72
- FWCI
- 198.47
- Type
- journal article
- Language
- en
- Open access
- green
- Funders
- General Research Fund of the Research Grants Council of Hong Kong (716612)
- International Doctoral Innovation Centre
- University of Nottingham
- Ningbo Science and Technology Bureau
- Australian Research Council (LP160101691)
- Artificial Intelligence and Optimisation Research Group of University of Nottingham Ningbo China
- Ningbo Education Bureau
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.