Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3
Open accessAbstract
In this paper we study the problem-solving ability of the Large Language Model known as GPT-3 (codename DaVinci), by considering its performance in solving tasks proposed in the “Bebras International Challenge on Informatics and Computational Thinking”. In our experiment, GPT-3 was able to answer with a majority of correct answers about one third of the Bebras tasks we submitted to it. The linguistic fluency of GPT-3 is impressive and, at a first reading, its explanations sound coherent, on-topic and authoritative; however the answers it produced are in fact erratic and the explanations often questionable or plainly wrong. The tasks in which the system performs better are those that describe a procedure, asking to execute it on a specific instance of the problem. Tasks solvable with simple, one-step deductive reasoning are more likely to obtain better answers and explanations. Synthesis tasks, or tasks that require a more complex logical consistency get the most incorrect answers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Among participants who relied on simple copy-paste strategies, Participants 1 and 4 encountered multiple challenges.
“Consistent with prior research underscoring the importance of critical engagement with AI outputs [37], among participants who relied on simple copy-paste strategies, Participants 1 and 4 (i.e., P_01 and P_04 in Appendix B) encountered multiple challenges.”
From Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis (2025) · cites this paper as [37]
For other tasks, such as answering multiple-choice questions, Parsons problems, and computational thinking tasks, the performance of LLMs is currently not as great.
“For other tasks, such as answering MCQs [176], Parsons problems [167] and computational thinking tasks [39], the performance of LLMs is currently not as great.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [39]
Evidence shows that LLMs do not perform as effectively on computational thinking problems that do not involve code writing.
“However, evidence also shows that LLMs do not perform as effectively on computational thinking problems that do not involve code writing [39].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 49,574 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
(2001) · Cited by 13,930 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Gelman, A., Hill, J., & Vehtari, A. (2020) · doi:10.1017/9781139161879 · Cited by 613 · see citing works
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · see citing works
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · see citing works
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · see citing works
It’s Computational Thinking! Bebras Tasks in the Curriculum
Dagienė, V., & Sentance, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46747-4_3 · Cited by 105 · see citing works
Sustaining Informatics Education by Contests
Dagienė, V. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-11376-5_1 · Cited by 51 · see citing works
Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
Saparov, A., & He, H. (2022) · arXiv · doi:10.48550/arxiv.2210.01240 · Cited by 46 · see citing works
Chiazzese, G., Arrigo, M., Chifari, A., Lonati, V., & Tosto, C. (2018) · Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21) · doi:10.1145/3284179.3284186 · Cited by 32 · see citing works
Self-citationProgramming Pedagogy and Assessment in the Era of AI/ML: A Position Paper
Raman, A., & Kumar, V. (2022) · Bangalore Annual Compute Conference · doi:10.1145/3561833.3561843 · Cited by 21 · see citing works
Promoting Computational Thinking Skills: Would You Use this Bebras Task?
Calcagni, A., Lonati, V., Malchiodi, D., Monga, M., & Morpurgo, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71483-7_9 · Cited by 20 · see citing works
Self-citationRandomised controlled trial and process evaluation of Code Clubs
Straw, S. (2017) · Cited by 11 · see citing works
A Platform for the Italian Bebras
Bellettini, C., Carimati, F., Lonati, V., Macoratti, R., Malchiodi, D., Monga, M., & Morpurgo, A. (2018) · International Conference on Computer Supported Education · doi:10.5220/0006775103500357 · Cited by 8 · see citing works
Self-citationSituated Learning with Bebras Tasklets
Bellettini, C., Lonati, V., Monga, M., Morpurgo, A., & Palazzolo, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33759-9_18 · Cited by 7 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
- Haberman, B., Cohen, A., and Dagien˙ e, V. (2011). The beaver contest: Attracting youngsters to study computing. In Proceedings of ITiCSE 2011 , pages 378378, Darmstadt, Germany. ACM.
Cited by
Sorted by citations
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54 · see citing works
Liu, S., Yu, Z., Huang, F., Bulbulia, Y., Bergen, A., & Liut, M. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653554 · Cited by 48 · see citing works
Fock, A., & Siller, H.-S. (2025) · International Journal of STEM Education · doi:10.1186/s40594-025-00589-5 · Cited by 21 · see citing works
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 13 · see citing works
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9 · see citing works
Utilization of Artificial Intelligence in Education: A Perspective on Learning Strategies
Yalçınalp, S., Türkoğlu, H., Koç, S. E., & Ersoy, H. (2024) · Artificial intelligence · doi:10.5772/intechopen.1005086 · Cited by 12 · see citing works
Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis
Musazade, N., Mezei, J., & Wang, X. (2025) · Algorithms · doi:10.3390/a18100631 · Cited by 4 · see citing works
“Bellettini et al. [37] tested the capacity of the GPT-3 model in solving problems designed for school students participating in an international competition.”
Teaching Programming in the Age of Generative AI
Martini, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653527 · Cited by 2 · see citing works
Benedetti, E. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724389.3731294
- Authors
- Bellettini, C., Lodi, M., Lonati, V., Monga, M., & Morpurgo, A.
- Affiliations
- University of Milan (IT)
- Venue
- Proceedings of the 15th International Conference on Computer Supported Education
- Published
- 2023
- Volume
- 2
- Pages
- 59-69
- OpenAlex
- W4366808792
- Cited by
- 10
- References
- 18
- Citations by year
- 2025: 4 · 2024: 4 · 2023: 2
- FWCI
- 3.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- International Conference on Computer Supported Education (publishedVersion, cc-by-nc-nd)
- HAL (submittedVersion, cc-by-nc-nd)
- Archivio Istituzionale della Ricerca (Universita Degli Studi Di Milano) (submittedVersion, other-oa)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, cc-by-nc-nd)
- Funders
- NextGenerationEU
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- GPT-3, large language models, problem solving, computational thinking, deductive reasoning, logical consistency, AI-generated explanations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
