CoChat
Sign Up
PreprintCited by 21Artificial Intelligence in EducationTeaching and Learning ProgrammingOnline Learning and Analytics

Robosourcing Educational Resources -- Leveraging Large Language Models for Learnersourcing

Open access
Denny, Paul, Sarsa, Sami, Hellas, Arto, Leinonen, Juho·2022·arXiv
DOI 10.48550/arxiv.2211.04715arXiv 2211.04715
Open this paper in CoChat — free accountOpen PDF

Abstract

In this article, we introduce and evaluate the concept of robosourcing for creating educational content. Robosourcing lies in the intersection of crowdsourcing and large language models, where instead of a crowd of humans, requests to large language models replace some of the work traditionally performed by the crowd. Robosourcing includes a human-in-the-loop to provide priming (input) as well as to evaluate and potentially adjust the generated artefacts; these evaluations could also be used to improve the large language models. We propose a system to outline the robosourcing process. We further study the feasibility of robosourcing in the context of education by conducting an evaluation of robosourced and programming exercises, generated using OpenAI Codex. Our results suggest that robosourcing could significantly reduce human effort in creating diverse educational content while maintaining quality similar to human-created content.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.04715

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Instructors can benefit as these models can rapidly generate new and personalised teaching materials and programming assignments.

    “Instructors can also benefit as these models can rapidly generate new and personalised teaching materials and programming assignments [75,174].”

    From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [75]

  • LLMs can be coaxed into generating exercises with prescribed themes and programming concepts.

    “One of the main findings in both papers was that LLMs can be coaxed into generating exercises with prescribed themes (such as basketball or cooking) and programming concepts (such as loops or conditionals).”

    From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [75]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 16

  1. [2][2] R. J. Crutcher, A. F. Healy, Cognitive operations and the generation effect., Journal of Experimental Psychology: Learning, Memory, and Cognition 15 (1989) 669. -
  2. [4][4] K. VanLehn, R. M. Jones, M. T. Chi, A model of the self-explanation effect, The journal of the learning sciences 2 (1992) 1-59. -
  3. [7][7] N. T. Heffernan, K. S. Ostrow, K. Kelly, D. Selent, E. G. Van Inwegen, X. Xiong, J. J. Williams, The future of adaptive learning: does the crowd hold the key?, International Journal of Artificial Intelligence in Education 26 (2016) 615-644. -
  4. [14][14] H. Purchase, J. Hamer, P. Denny, A. Luxton-Reilly, The quality of a peerwise mcq repository, in: Proceedings of the Twelfth Australasian Conference on Computing Education - Volume 15. 103, ACE '10, Australian Computer Society, Inc., AUS, 2010, p. 137-146. -
  5. [18][18] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, et al., Competition-level code generation with alphacode, arXiv preprint arXiv:2203.07814 (2022). - · arXiv
  6. [24][24] H. L. Roediger, A. C. Butler, Retrieval practice (testing) effect, In H. L. Pashler (Ed.), Encyclopedia of the Mind, Sage Publishing Co. (2013) 660-661. -
  7. [25][25] R. L. Bangert-Drowns, J. A. Kulik, C.-L. C. Kulik, Effects of frequent classroom testing, The Journal of Educational Research 85 (1991) 89-99. -
  8. [26][26] K. A. Ericsson, R. R. Hoffman, A. Kozbelt, A. M. Williams, The Cambridge handbook of expertise and expert performance, Cambridge University Press, 2018. doi: 10.1017/ 9781316480748 . -
  9. [27][27] J. Hattie, H. Timperley, The power of feedback, Review of educational research 77 (2007) 81-112. -
  10. [29][29] V. J. Shute, Focus on formative feedback, Review of educational research 78 (2008) 153-189. -
  11. [33][33] T. Liu, B. Wei, B. Chang, Z. Sui, Large-scale simple question generation by template-based seq2seq learning, in: X. Huang, J. Jiang, D. Zhao, Y. Feng, Y. Hong (Eds.), Natural Language Processing and Chinese Computing, Springer International Publishing, Cham, 2018, pp. 75-87. -
  12. [34][34] I.-H. Hsiao, S. Sosnovsky, P. Brusilovsky, Adaptive navigation support for parameterized questions in object-oriented programming, in: U. Cress, V. Dimitrova, M. Specht (Eds.), Learning in the Synergy of Multiple Disciplines, Springer Berlin Heidelberg, Berlin, Heidelberg, 2009, pp. 88-98. -
  13. [39][39] O. Polozov, E. O'Rourke, A. M. Smith, L. Zettlemoyer, S. Gulwani, Z. Popović, Personalized mathematical word problem generation, in: IJCAI 2015, 2015. URL: https://www.microsoft. com/en-us/research/publication/personalized-mathematical-word-problem-generation/. -
  14. [42][42] J. Finnie-Ansley, P. Denny, B. A. Becker, A. Luxton-Reilly, J. Prather, The robots are coming: Exploring the implications of openai codex on introductory programming, in: Australasian Computing Education Conference, 2022, pp. 10-19. -
  15. [46][46] I. Solaiman, C. Dennison, Process for adapting language models to society (palms) with values-targeted datasets, Advances in Neural Information Processing Systems 34 (2021). -
  16. [48][48] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al., Extracting training data from large language models, in: 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 2633-2650. -

Cited by

Sorted by citations

Related

Authors
Denny, P., Sarsa, S., Hellas, A., & Leinonen, J.
Affiliations
  • University of Auckland (NZ)
  • Aalto University (FI)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2211.04715
arXiv
2211.04715
OpenAlex
W4308757526
Cited by
21
References
0
Citations by year
2025: 5 · 2024: 11 · 2023: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Keywords
large language models, crowdsourcing, educational content development, human-in-the-loop, OpenAI Codex, programming exercises, human effort reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review