The DOI matches its Crossref record; 29 of the 30 references it cites were found in a registry or an index, and 1 are not checked yet.
Fooling MOSS Detection with Pretrained Language Models
Open accessAbstract
As artificial intelligence (AI) technologies become increasingly powerful and prominent in society, their misuse is a growing concern. In educational settings, AI technologies could be used by students to cheat on assignments and exams. In this paper we explore whether transformers can be used to solve introductory level programming assignments while bypassing commonly used AI tools to detect similarities between pieces of software. We find that a student using GPT-J [60] can complete introductory level programming assignments without triggering suspicion from MOSS [2], a widely used software similarity and plagiarism detection tool. This holds despite the fact that GPT-J was not trained on the problems in question and is not provided with any examples to work from. We further find that the code written by GPT-J is diverse in structure, lacking any particular tells that future plagiarism detection techniques may use to try to identify algorithmically generated code. We conclude with a discussion of the ethical and educational implications of large language models and directions for future research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Moreover, AI-generated code is diverse in structure and resilient to standard plagiarism detection tools.”
From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [13]
In the citing paper
“Biderman and Raff show that introductory level programming assignments can be completed by AI code generators without triggering suspicion from MOSS, a state of the art code plagiarism detection tool [13].”
From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [13]
In the citing paper
“These generators enable effortless obfuscation attacks, exploiting vulnerabilities in existing detectors by inserting statements to disrupt the matching of related programs.”
From Detecting Automatic Software Plagiarism via Token Sequence Normalization (2024) · cites this paper as [6]
In the citing paper
“With the free and accessible GenAI tools, students now have an easy way to produce solutions that cannot be detected using common approaches [41].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [41]
In the citing paper
“Biderman and Raff [2] explored the performance of Java code generated by GPT-J on MOSS (measure of software similarity tool) and discovered that plagiarism was not detected, although they do not have evidence that this would be the case for tasks in more advanced programming courses.”
From Incorporating Generative AI into Software Development Education (2024) · cites this paper as [2]
Citations
Cites
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
Text Summarization with Pretrained Encoders
Liu, Y., & Lapata, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1387 · Cited by 1,553
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1×
“There is a substantial literature on “prompt programming,” or crafting inputs to language models in ways that help the model perform better at downstream tasks [12, 42, 51] (for a survey, see [43]).”
BackgroundRanking Sentences for Extractive Summarization with Reinforcement Learning
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1158 · Cited by 684
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
Self-citationGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 404
Self-citationKnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
Chen, X., Zhang, N., Xie, X., Deng, S., Yao, Y., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511998 · Cited by 389
Designing Quality E-Learning Environments for Emergency Remote Teaching in Coronavirus Crisis
Affouneh, S. J., Salha, S. H., & Khlaif, Z. N. (2020) · DOAJ (DOAJ: Directory of Open Access Journals) · doi:10.30476/ijvlms.2020.86120.1033 · Cited by 321
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2021) · arXiv · doi:10.48550/arxiv.2106.15590 · Cited by 317
Research works printed but not linked: 35
- Alex Aiken . 2000. Moss (measure of software similarity) plagiarism detection system. http://www. cs. berkeley. edu/moss/ ( 2000 ). Alex Aiken. 2000. Moss (measure of software similarity) plagiarism detection system. http://www. cs. berkeley. edu/moss/ (2000).
- Jacob Austin , Augustus Odena , Maxwell Nye , Maarten Bosma , Henryk Michalewski , David Dohan , Ellen Jiang , Carrie Cai , Michael Terry , Quoc V. Le , and Charles Sutton . 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 ( 2021 ). Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). · arXiv
- Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel Ziegler , Jeffrey Wu , Clemens Winter , Chris Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020 . Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H . Lin (Eds.) , Vol. 33 . Curran Associates, Inc. , 1877--1901. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877--1901.
- Nicholas Carlini , Chang Liu , Úlfar Erlingsson , Jernej Kos , and Dawn Song . 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks . In 28th $$USENIX$$ Security Symposium ($$USENIX$$ Security 19). 267--284. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th $$USENIX$$ Security Symposium ($$USENIX$$ Security 19). 267--284.
- Nicholas Carlini , Florian Tramer , Eric Wallace , Matthew Jagielski , Ariel Herbert-Voss , Katherine Lee , Adam Roberts , Tom Brown , Dawn Song , Ulfar Erlingsson , Alina Oprea , and Colin Raffel . 2021. Extracting training data from large language models . In 30th $$USENIX$$ Security Symposium ($$USENIX$$ Security 21). 2633--2650. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th $$USENIX$$ Security Symposium ($$USENIX$$ Security 21). 2633--2650.
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harrison Edwards Yura Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea. Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such David W. Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William H. Guss Alex Nichol Igor Babuschkin S. Arun Balaji Shantanu Jain Andrew Carr Jan Leike Joshua Achiam Vedant Misra Evan Morikawa Alec Radford Matthew M. Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021a. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde Jared Kaplan Harrison Edwards Yura Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea. Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such David W. Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William H. Guss Alex Nichol Igor Babuschkin S. Arun Balaji Shantanu Jain Andrew Carr Jan Leike Joshua Achiam Vedant Misra Evan Morikawa Alec Radford Matthew M. Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021a. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
- Xiang Chen , Xin Xie , Ningyu Zhang , Jiahuan Yan , Shumin Deng , Chuanqi Tan , Fei Huang , Luo Si , and Huajun Chen . 2021 b. Adaprompt: Adaptive prompt-based finetuning for relation extraction. arXiv preprint arXiv:2104.07650 (2021). Xiang Chen, Xin Xie, Ningyu Zhang, Jiahuan Yan, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2021b. Adaprompt: Adaptive prompt-based finetuning for relation extraction. arXiv preprint arXiv:2104.07650 (2021). · arXiv
- Miranda Chong , Lucia Specia , and Ruslan Mitkov . 2010 . Using natural language processing for automatic detection of plagiarism . In Proceedings of the 4th International Plagiarism Conference (IPC-2010) . Miranda Chong, Lucia Specia, and Ruslan Mitkov. 2010. Using natural language processing for automatic detection of plagiarism. In Proceedings of the 4th International Plagiarism Conference (IPC-2010).
- Katherine Crowson , Stella Biderman , Daniel Kornis , Dashiell Stander , Eric Hallahan , Louis Castricato , and Edward Raff . 2022. VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance. arXiv preprint arXiv:2204.08583 ( 2022 ). Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. 2022. VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance. arXiv preprint arXiv:2204.08583 (2022). · arXiv
- Phillip Dawson , Wendy Sutherland-Smith , and Mark Ricksen . 2020. Can software improve marker accuracy at detecting contract cheating? A pilot study of the Turnitin authorship investigate alpha. Assessment & Evaluation in higher education , Vol. 45 , 4 ( 2020 ), 473--482. Phillip Dawson, Wendy Sutherland-Smith, and Mark Ricksen. 2020. Can software improve marker accuracy at detecting contract cheating? A pilot study of the Turnitin authorship investigate alpha. Assessment & Evaluation in higher education, Vol. 45, 4 (2020), 473--482.
- Iddo Drori , Sunny Tran , Roman Wang , Newman Cheng , Kevin Liu , Leonard Tang , Elizabeth Ke , Nikhil Singh , Taylor L Patti , Jayson Lynch , 2021 . A Neural Network Solves and Generates Mathematics Problems by Program Synthesis: Calculus, Differential Equations, Linear Algebra, and More. arXiv preprint arXiv:2112.15594 (2021). Iddo Drori, Sunny Tran, Roman Wang, Newman Cheng, Kevin Liu, Leonard Tang, Elizabeth Ke, Nikhil Singh, Taylor L Patti, Jayson Lynch, et al. 2021. A Neural Network Solves and Generates Mathematics Problems by Program Synthesis: Calculus, Differential Equations, Linear Algebra, and More. arXiv preprint arXiv:2112.15594 (2021). · arXiv
- Iddo Drori and Nakul Verma . 2021. Solving Linear Algebra by Program Synthesis. arXiv preprint arXiv:2111.08171 ( 2021 ). Iddo Drori and Nakul Verma. 2021. Solving Linear Algebra by Program Synthesis. arXiv preprint arXiv:2111.08171 (2021). · arXiv
- Daniel Fried , Armen Aghajanyan , Jessy Lin , Sida Wang , Eric Wallace , Freda Shi , Ruiqi Zhong , Wen-tau Yih, Luke Zettlemoyer , and Mike Lewis . 2022. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv preprint arXiv:2204.05999 ( 2022 ). Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv preprint arXiv:2204.05999 (2022). · arXiv
- Leo Gao , Stella Biderman , Sid Black , Laurence Golding , Travis Hoppe , Charles Foster , Jason Phang , Horace He , Anish Thite , Noa Nabeshima , Shawn Presser , and Connor Leahy . 2020 . The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027 (2020). Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027 (2020). · arXiv
- Seungju Han , Beomsu Kim , Jin Yong Yoo , Seokjun Seo , Sangbum Kim , Enkhbayar Erdenee , and Buru Chang . 2022 . Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances. arXiv preprint arXiv:2204 .10825 (2022). Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjun Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru Chang. 2022. Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances. arXiv preprint arXiv:2204.10825 (2022). · arXiv
- Dan Hendrycks , Steven Basart , Saurav Kadavath , Mantas Mazeika , Akul Arora , Ethan Guo , Collin Burns , Samir Puranik , Horace He , Dawn Song , and Jacob Steinhardt . 2021 . Measuring Coding Challenge Competence With APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Ari Holtzman , Jan Buys , Li Du , Maxwell Forbes , and Yejin Choi . 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 ( 2019 ). Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019). · arXiv
- Zhiqiang Hu , Roy Ka-Wei Lee , Charu C Aggarwal, and Aston Zhang. 2020 . Text Style Transfer: A Review and Experimental Evaluation . arXiv preprint arXiv:2010.12742 (2020). Zhiqiang Hu, Roy Ka-Wei Lee, Charu C Aggarwal, and Aston Zhang. 2020. Text Style Transfer: A Review and Experimental Evaluation. arXiv preprint arXiv:2010.12742 (2020). · arXiv
- Gabbrielle Johnson . 2022. Are Algorithms Value-Free? Feminist Theoretical Virtues in Machine Learning. Journal of Moral Philosophy, special issue on "Justice, Power, and the Ethics of Algorithmic Decision-Making " ( 2022 ). Gabbrielle Johnson. 2022. Are Algorithms Value-Free? Feminist Theoretical Virtues in Machine Learning. Journal of Moral Philosophy, special issue on "Justice, Power, and the Ethics of Algorithmic Decision-Making" (2022).
- Kalpesh Krishna , Gaurav Singh Tomar , Ankur P. Parikh , Nicolas Papernot , and Mohit Iyyer . 2020 . Thieves on Sesame Street! Model Extraction of BERT-based APIs . In International Conference on Learning Representations. Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit Iyyer. 2020. Thieves on Sesame Street! Model Extraction of BERT-based APIs. In International Conference on Learning Representations.
- Connor Leahy and Stella Biderman . 2021. The Hard Problem of Aligning AI to Human Values . In The State of AI Ethics Report ( Volume 4 ). Connor Leahy and Stella Biderman. 2021. The Hard Problem of Aligning AI to Human Values. In The State of AI Ethics Report (Volume 4).
- Mina Lee , Percy Liang , and Qian Yang . 2022b. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. arXiv preprint arXiv:2201.06796 ( 2022 ). Mina Lee, Percy Liang, and Qian Yang. 2022b. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. arXiv preprint arXiv:2201.06796 (2022). · arXiv
- Pengfei Liu , Weizhe Yuan , Jinlan Fu , Zhengbao Jiang , Hiroaki Hayashi , and Graham Neubig . 2021. Pre-train , prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 ( 2021 ). Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021). · arXiv
- Divya Luke , PS Divya , Sony L Johnson , S Sreeprabha , and Elizabeth B Varghese . 2014 . Software plagiarism detection techniques: A comparative study . In 2018 5th International Symposium on Emerging Trends and Technologies in Libraries and Information Services (ETTLIS). Citeseer. Divya Luke, PS Divya, Sony L Johnson, S Sreeprabha, and Elizabeth B Varghese. 2014. Software plagiarism detection techniques: A comparative study. In 2018 5th International Symposium on Emerging Trends and Technologies in Libraries and Information Services (ETTLIS). Citeseer.
- Rohan Mukherjee , Yeming Wen , Dipak Chaudhari , Thomas Reps , Swarat Chaudhuri , and Chris Jermaine . 2021 . Neural Program Generation Modulo Static Analysis. In Thirty-Fifth Conference on Neural Information Processing Systems. Rohan Mukherjee, Yeming Wen, Dipak Chaudhari, Thomas Reps, Swarat Chaudhuri, and Chris Jermaine. 2021. Neural Program Generation Modulo Static Analysis. In Thirty-Fifth Conference on Neural Information Processing Systems.
- Jonas Oppenlaender . 2022. Prompt Engineering for Text-Based Generative Art. arXiv preprint arXiv:2204.13988 ( 2022 ). Jonas Oppenlaender. 2022. Prompt Engineering for Text-Based Generative Art. arXiv preprint arXiv:2204.13988 (2022). · arXiv
- Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J Liu . 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 ( 2019 ). Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019). · arXiv
- Waseem Sheikh . 2022. Teaching C programming using automated unit testing and test-driven development-Design and efficacy study. Computer Applications in Engineering Education ( 2022 ). Waseem Sheikh. 2022. Teaching C programming using automated unit testing and test-driven development-Design and efficacy study. Computer Applications in Engineering Education (2022).
- Avi Shporer , Sunny Tran , Nikhil Singh , Brandon Kates , Jayson Lynch , and Iddo Drori . 2021. Learning Methods for Solving Astronomy Course Problems. preprint ( 2021 ). https://www.cs.columbia.edu/ idrori/Learning_Methods_for_Solving_Astronomy_Course_Problems.pdf Avi Shporer, Sunny Tran, Nikhil Singh, Brandon Kates, Jayson Lynch, and Iddo Drori. 2021. Learning Methods for Solving Astronomy Course Problems. preprint (2021). https://www.cs.columbia.edu/ idrori/Learning_Methods_for_Solving_Astronomy_Course_Problems.pdf
- Sunny Tran , Pranav Krishna , Ishan Pakuwal , Prabhakar Kafle , Nikhil Singh , Jayson Lynch , and Iddo Drori . 2021 . Solving machine learning problems . In Asian Conference on Machine Learning. PMLR, 470--485 . Sunny Tran, Pranav Krishna, Ishan Pakuwal, Prabhakar Kafle, Nikhil Singh, Jayson Lynch, and Iddo Drori. 2021. Solving machine learning problems. In Asian Conference on Machine Learning. PMLR, 470--485.
- Zhuohao Wu , Danwen Ji , Kaiwen Yu , Xianxu Zeng , Dingming Wu , and Mohammad Shidujaman . 2021 . AI Creativity and the Human-AI Co-creation Model. In International Conference on Human-Computer Interaction. Springer, 171--190 . Zhuohao Wu, Danwen Ji, Kaiwen Yu, Xianxu Zeng, Dingming Wu, and Mohammad Shidujaman. 2021. AI Creativity and the Human-AI Co-creation Model. In International Conference on Human-Computer Interaction. Springer, 171--190.
- Frank F Xu , Uri Alon , Graham Neubig , and Vincent J Hellendoorn . 2022. A systematic evaluation of large language models of code. arXiv preprint arXiv:2202.13169 ( 2022 ). Frank F Xu, Uri Alon, Graham Neubig, and Vincent J Hellendoorn. 2022. A systematic evaluation of large language models of code. arXiv preprint arXiv:2202.13169 (2022). · arXiv
- Muhammad Yasid , Gomgom T.P. Siregar , and Muhammad Ansori Lubis . 2020. The Policy of Credit Payment Relaxation in Overcoming the Impact of Covid-19 Spread to the Economic Society in Indonesia. Journal of Advanced Research in Dynamical and Control Systems ( 2020 ). Muhammad Yasid, Gomgom T.P. Siregar, and Muhammad Ansori Lubis. 2020. The Policy of Credit Payment Relaxation in Overcoming the Impact of Covid-19 Spread to the Economic Society in Indonesia. Journal of Advanced Research in Dynamical and Control Systems (2020).
- Ann Yuan , Andy Coenen , Emily Reif , and Daphne Ippolito . 2022 . Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces. 841--852 . Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces. 841--852.
- Susan Zhang , Stephen Roller , Naman Goyal , Mikel Artetxe , Moya Chen , Shuohui Chen , Christopher Dewan , Mona Diab , Xian Li , Xi Victoria Lin, et al . 2022 . OPT : Open Pre-trained Transformer Language Models . arXiv preprint arXiv:2205.01068 (2022). Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. OPT: Open Pre-trained Transformer Language Models. arXiv preprint arXiv:2205.01068 (2022). · arXiv
Cited by
Perkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“However, recent work has shown that common plagiarism detection tools are often ineffective against AI-generated solutions [6].”
BackgroundPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“Biderman and Raff show that introductory level programming assignments can be completed by AI code generators without triggering suspicion from MOSS, a state of the art code plagiarism detection tool [13].”
BackgroundPerkins, M., Roe, J., Postma, D., McGaughran, J., & Hickerson, D. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-023-09492-6 · Cited by 234
Metadata
- Authors
- Biderman, S., & Raff, E.
- Affiliations
- Booz Allen Hamilton (United States) (US)
- Venue
- Proceedings of the 31st ACM International Conference on Information & Knowledge Management
- Publisher
- ACM
- Published
- 2022-10-17
- Pages
- 2933-2943
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W4306317681
- Cited by
- 37
- References
- 26
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 17 · 2023: 14 · 2022: 1
- FWCI
- 8.70
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion, cc-by-sa)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- bryophytes, GPT-J, large language models, code plagiarism detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fooling MOSS Detection with Pretrained Language Models. https://app.cochat.ai/papers/10.1145/3511808.3557079
- The paper
- Biderman, S., & Raff, E. (2022). Fooling MOSS Detection with Pretrained Language Models. Proceedings of the 31st ACM International Conference on Information & Knowledge Management. https://doi.org/10.1145/3511808.3557079
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.