A Survey of Automated Programming Hint Generation: The HINTS Framework
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Automated tutoring systems offer the flexibility and scalability necessary to facilitate the provision of high-quality and universally accessible programming education. To realise the potential of these systems, recent work has proposed a diverse range of techniques for automatically generating feedback in the form of hints to assist students with programming exercises. This article integrates these apparently disparate approaches into a coherent whole. Specifically, it emphasises that all hint techniques can be understood as a series of simpler components with similar properties. Using this insight, it presents a simple framework for describing such techniques, the Hint Iteration by Narrow-down and Transformation Steps framework, and surveys recent work in the context of this framework. Findings from this survey include that (1) hint techniques share similar properties, which can be used to visualise them together, (2) the individual steps of hint techniques should be considered when designing and evaluating hint systems, (3) more work is required to develop and improve evaluation methods, and (4) interesting relationships, such as the link between automated hints and data-driven evaluation, should be further investigated. Ultimately, this article aims to facilitate the development, extension, and comparison of automated programming hint techniques to maximise their educational potential.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Proceedings of the 10th Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 554 · cited 1×
“ture of programming hints. Our work is situated more broadly in the area of automated programming tutors. Other reviews in this area have considered techniques for automated assessment (e.g. [22] and [23]), or approaches to 2 tutoring that can be supported by AI techniques [24], such as \example", \simulation&quo…”
MethodA Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. M. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 546
A systematic literature review of automated feedback generation for programming exercises
Keuning, H., Jeuring, J., & Heeren, B. (2019) · ACM Transactions on Computing Education · doi:10.1145/3231711 · Cited by 370
Automatic Software Repair: A Bibliography
Monperrus, M. (2019) · ACM Computing Surveys · doi:10.1145/3105906 · Cited by 321
Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor
Rivers, K., & Koedinger, K. R. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0070-z · Cited by 258 · cited 13×
“The set of features can also be searched until some correctness test is passed as in [48, 6, 1, 11], or a model of correct features can be used to narrow them down as in [46, 4, 45, 47]”
BackgroundMethodInfluentialLearning Program Embeddings to Propagate Feedback on Student CodePreprint
Piech, C., Huang, J., Nguyen, A., Phulsuksombati, M., Sahami, M., & Guibas, L. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05969 · Cited by 205
Intelligent tutoring systems for programming education: a systematic review
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018) · Proceedings of the 20th Australasian Computing Education Conference · doi:10.1145/3160489.3160492 · Cited by 166 · cited 1×
“or \dialogue"- based approaches. In addition, some work has focused on the general features of programming tutors, such as the programming languages taught, or primary and supplementary features [25]. While some of these reviews also reference automated programming feedback, it is generally brief or not the main…”
BackgroundFifteen years of constraint-based tutors: what we have achieved and where we are going
Mitrović, A. (2011) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9105-9 · Cited by 151
Codewebs: scalable homework search for massive open online programming courses
Nguyen, A., Piech, C., Huang, J., & Guibas, L. J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568023 · Cited by 150 · cited 5×
“values of some expressions in the program. When a student submits an incorrect program following a particular strategy, the teacher feedback associated with that strategy is given. Patterns Codewebs [38] - teacher feedback is written to address particular bugs in peer data. Then, when these bugs are identied in a new…”
BackgroundMethodInfluentialTowards a Systematic Review of Automated Feedback Generation for Programming Exercises
Keuning, H., Jeuring, J., & Heeren, B. (2016) · Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/2899415.2899422 · Cited by 142 · cited 2×
“Single File vs. Multi File Analysis") and the program representations they require as input (e.g. \Trees vs. Graphs", "Source Code vs. Byte Code Analysis"). In [20] (extended from [21]), all three types are used separately to label programming tools. The tools are labelled based on the technique us…”
BackgroundMethodAutonomously Generating Hints by Inferring Problem Solving Policies
Piech, C., Sahami, M., Huang, J., & Guibas, L. J. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2724668 · Cited by 139 · cited 2×
“ilar to (2), except the step is applied to the help-seeking student’s program instead of peer programs. 4. general features ! steps. Program states can be collected together into a state space, as in [12, 16, 32, 13, 33, 34]. Note that if a technique involves steps (2) and (4), they can be treated as a single transfor…”
BackgroundInfluentialWriting Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis
Head, A., Glassman, E. L., Soares, G., Suzuki, R., Figueredo, L., D’Antoni, L., & Hartmann, B. (2017) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3051457.3051467 · Cited by 134 · cited 5×
“are illustrative of their diversity, and the similarities they exhibit will act as a basis for the general framework discussed in the next section. The examples are as follows: 1. (MB) MistakeBrowser [1]. Using a database of program transformations learned from peer data, the system searches for a subset of these tran…”
BackgroundMethodInfluentialAutomated Correction for Syntax Errors in Programming Assignments using Recurrent Neural NetworksPreprint
Bhatia, S., & Singh, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06129 · Cited by 125 · cited 5×
“In [45], the authors present a method for automatically correcting syntactic errors in student programs using machine learning and peer data.”
BackgroundMethodInfluentialAsk-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback
Gerdes, A., Heeren, B. J., Jeuring, J. T., & van Binsbergen, L. T. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0080-x · Cited by 102
Generating Data-Driven Hints for Open-Ended Programming.
Price, T., Dong, Y., & Barnes, T. M. (2016) · Educational Data Mining · Cited by 76
Automated Personalized Feedback in Introductory Java Programming MOOCs
Marin, V. J., Pereira, T., Sridharan, S., & Rivero, C. R. (2017) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2017.169 · Cited by 75
Feedback generation for performance problems in introductory programming assignments
Gulwani, S., Radiček, I., & Zuleger, F. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635912 · Cited by 71 · cited 2×
“relevant teacher hints. Teacher feedback can be narrowed down to just the feedback relevant to the student’s features. These can include output [2, 3], edits that correct the program [1] or strategy [42] (See Sec. 5.3). 12. external resources ! relevant external resources. External resources, such as web links [9], ca…”
BackgroundInfluentialAutomating Hint Generation with Solution Space Path Construction
Rivers, K., & Koedinger, K. R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_41 · Cited by 60 · cited 3×
“st common next-state of peers in a similar position [16] (\Codebased" technique), using a scoring function that accounts for popularity, correctness and distance from the student’s current state [33], nding the path with the shortest expected time to a solution [12] (\Poisson Path"), nding the most likely pa…”
BackgroundMethodInfluentialThe Continuous Hint Factory - Providing Hints in Vast and Sparsely Populated Edit Distance Spaces
Paaßen, B., Hammer, B., Price, T., Barnes, T. M., Groß, S., & Niels Pinkwart, C. (2017) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554697 · Cited by 56
Hint Generation Under Uncertainty: The Effect of Hint Quality on Help-Seeking Behavior
Price, T., Zhi, R., & Barnes, T. M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-61425-0_26 · Cited by 53 · cited 1×
“However, some recent research has suggested that poor quality hints can discourage students from seeking further help [37], so the balance between hint availability and quality in these systems is an important consideration that requires further investigation.”
BackgroundA Review of AI-Supported Tutoring Approaches for Learning Programming
Le, N.-T., Strickroth, S., Groß, S., & Niels Pinkwart, C. (2013) · Studies in computational intelligence · doi:10.1007/978-3-319-00293-4_20 · Cited by 52
Providing Meaningful Feedback for Autograding of Programming Assignments
Haldeman, G., Tjang, A., Babeş-Vroman, M., Bartos, S., Shah, J., Yucht, D., & Nguyen, T. D. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159502 · Cited by 52 · cited 4×
“t hints, as in [44]. (See Sec. 5.3) 11. teacher hints ! relevant teacher hints. Teacher feedback can be narrowed down to just the feedback relevant to the student’s features. These can include output [2, 3], edits that correct the program [1] or strategy [42] (See Sec. 5.3). 12. external resources ! relevant external…”
BackgroundMethodInfluentialA Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms
Price, T., Dong, Y., Zhi, R., Paaßen, B., Lytle, N., Cateté, V., & Barnes, T. M. (2019) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-019-00177-z · Cited by 51
A survey on algorithmic debugging strategies
Silva, J. (2011) · Advances in Engineering Software · doi:10.1016/j.advengsoft.2011.05.024 · Cited by 49
Evaluation of a Data-Driven Feedback Algorithm for Open-Ended Programming.
Price, T., Zhi, R., & Barnes, T. M. (2017) · Educational Data Mining · Cited by 49
A Classification of Adaptive Feedback in Educational Systems for Programming
Le, N.-T. (2016) · Systems · doi:10.3390/systems4020022 · Cited by 48
Adaptive Feedback Based on Student Emotion in a System for Programming Practice
Tiam-Lee, T. J., & Sumi, K. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-91464-0_24 · Cited by 41
Building Games to Learn from Their Players: Generating Hints in a Serious Game
Hicks, A. G., Peddycord, B., & Barnes, T. M. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_39 · Cited by 39 · cited 4×
“ed, and these further steps will be discussed in later sections. 5.1.1 Example - Hint Factory. Originally presented in the context of a logic tutor [31] and later adapted to the domain of programming [5, 32, 13], the Hint Factory is a technique that uses peer data and a Markov Decision Process (MDP) to produce next-st…”
BackgroundMethodInfluentialA Review of Static Analysis Approaches for Programming Exercises
Striewe, M., & Goedicke, M. (2014) · Communications in computer and information science · doi:10.1007/978-3-319-08657-6_10 · Cited by 38 · cited 1×
“where\yes/no" feedback reveals whether work is correct, and the other classes reveal information about syntactic, semantic, layout or quality (e.g. eciency) issues respectively. In contrast, in [19], which reviews static analysis approaches to producing feedback for Java programs, tools are classied based on the…”
MethodDeep Reinforcement Learning for Programming Language CorrectionPreprint
Gupta, R., Kanade, A., & Shevade, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10467 · Cited by 30 · cited 4×
“These examples are similar, but the comparison is even more indirect, because the expected features are now abstracted to a model [46] Here, the program features are sequences of actions that correct the program.”
BackgroundInfluentialViDA: A virtual debugging advisor for supporting learning in computer programming courses
Lee, V. C. S., Yu, Y. T., Tang, C. M., Wong, T.-L., & Poon, C. K. (2018) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12238 · Cited by 30
Generating Hints for Programming Problems Using Intermediate Output.
Peddycord, B., Hicks, A. G., & Barnes, T. M. (2014) · Cited by 29
An Incremental Hint System For Automated Programming Assignments
Antonucci, P., Estler, C., Nikolić, Đ., Piccioni, M., & Meyer, B. (2015) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2729094.2742607 · Cited by 28 · cited 3×
“Similar approaches to this that use model solutions include [8], which also uses program strategies, and AutoTeach [14], where a solution is automatically converted into steps of increasing detail, called hint levels.”
BackgroundMethodAutomated Data-Driven Hints for Computer Programming Students
Chow, S., Yacef, K., Koprinska, I., & Curran, J. (2017) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3099023.3099065 · Cited by 25 · cited 5×
“are given in Table 3. Table 3: Examples of how features of a student program can be directly compared to a set of expected features in order to produce hints Feature Comparison Example(s) Concepts In [16], the \concepts" present in a student’s program (e.g. if statements, for loops) are compared to expected conce…”
BackgroundMethodInfluentialSelf-citationStrategy-based feedback in a programming tutor
Keuning, H., Heeren, B. J., & Jeuring, J. T. (2014) · Computer Science Education Research Conference · doi:10.1145/2691352.2691356 · Cited by 22 · cited 5×
“r feedback [1, 2, 3], extracting patterns from peer data [4, 5, 6], identifying particular dynamic or static issues with student programs [7], automatically generating personalised paths to solutions [8, 9, 10, 11], or combinations of these [12, 13]. While this diversity of approaches oers a great range of potential o…”
BackgroundMethodInfluentialSuzuki, R., Soares, G., Glassman, E. L., Head, A., D’Antoni, L., & Hartmann, B. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3053187 · Cited by 22 · cited 1×
“An interesting exploration of many different possible uses can be found in [49] where, for example, the authors suggest running the corrected code, then highlighting differences in the behaviour”
BackgroundThe Impact of Data Quantity and Source on the Quality of Data-Driven Hints for Programming
Price, T., Zhi, R., Dong, Y., Lytle, N., & Barnes, T. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93843-1_35 · Cited by 22 · cited 1×
“de-phrases (Codewebs [38]) use a templating language to express model solutions in many dierent forms to account for dierent programming strategies when evaluating student solutions against experts.[51] use a templating language (\strategy language") to express paths to model solutions in many dierent forms to ac…”
BackgroundNguyen, T. B., Do, T. V., Thi, H. A. L., & Nguyên, N. T. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-38884-7 · Cited by 21
A Data-Driven Method for Helping Teachers Improve Feedback in Computer Programming Automated Tutors
McBroom, J., Yacef, K., Koprinska, I., & Curran, J. R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93843-1_24 · Cited by 21 · cited 2×
“transformations to just the ones that lead to a solution [1] transform student programs into a standardised form and create an state space so teachers can visualise how students complete an exercise [50] transform student programs into a standardised form and create an state space so the next state can be selected [5]…”
BackgroundMethodInfluentialSelf-citationAutomatic Extraction of AST Patterns for Debugging Student Programs
Lazar, T., Možina, M., & Ivan Bratko, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-61425-0_14 · Cited by 20 · cited 5×
“it to a standard form. For example, a model solution to a programming task could be divided into a set of functions, represented with a dierent data structure, such as an abstract syntax tree (AST) [4], or converted to a canonical (standard) form through semanticspreserving transformations [9]. In addition, a set of s…”
BackgroundMethodInfluentialPerelman, D., Gulwani, S., Grossman, D., & Provost, P. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594297 · Cited by 19
Jeuring, J. T., Gerdes, A., & Heeren, B. J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33263-0_42 · Cited by 17
Rewrite Rules for Debugging Student Programs in Programming Tutors
Lazar, T., Sadikov, A., & Ivan Bratko, A. (2017) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2017.2743701 · Cited by 17
TipsC: Tips and Corrections for programming MOOCs
Sharma, S., Agarwal, P., Mor, P., & Karkare, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93846-2_60 · Cited by 15
An Affective and Cognitive Tutoring System for Learning Programming
Estrada, M. L. B., Cabada, R. Z., González-Hernández, F., Bustillos, R. O., & Reyes-Garcia, C. A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-27101-9_12 · Cited by 12 · cited 1×
“based on student emotion [28, 29]), restricting hint availability (e.”
BackgroundIdentifying typical approaches and errors in Prolog programming with argument-based machine learning
Možina, M., Lazar, T., & Bratko, I. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.06.029 · Cited by 12
Model solutions and properties for diagnosing student programs in Ask-Elle
Jeuring, J. T., van Binsbergen, L. T., Gerdes, A., & Heeren, B. J. (2014) · Computer Science Education Research Conference · doi:10.1145/2691352.2691355 · Cited by 11 · cited 2×
“Properties In the property-based testing technique in [39, 9] (AskElle) the properties of a student’s program are compared to the properties it is expected to satisfy.”
BackgroundMethodExecution Traces as a Powerful Data Representation for Intelligent Tutoring Systems for Programming
Paaßen, B., Jensen, J., & Hammer, B. (2016) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 8
Applying spectrum-based fault localization to generate debugging suggestions for student programmers
Edmison, B., & Edwards, S. H. (2015) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2015.7392052 · Cited by 7 · cited 1×
“e identied as incorrect, either directly or indirectly. For example, edits for deleting code in [10, 11, 35]. (See Sec. 5.2). These can also be based on buggy patterns [38], constraints [15], output [7] or execution traces [41]. (See Sec. 5.3) 10. teacher hints ! model for correct hints. Teacher hints attached to part…”
BackgroundInfluentialComparing different styles of automated feedback for programming exercises
Annamaa, A., Suviste, R., & Vene, V. (2017) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3141880.3141909 · Cited by 6 · cited 1×
“to prevent hint overuse [30]) or deciding which hint technique to use for a particular student (e.”
BackgroundReal-time programming exercise feedback in MOOCs.
Chen, Z., Nguyễn, A., Schlender, A., & Ngiam, J. (2017) · Educational Data Mining · Cited by 2
Research works printed but not linked: 2
- 10.5555/3045118.3045235 · doi:10.5555/3045118.3045235
- John Stamper , Tiffany Barnes , Lorrie Lehmann , and Marvin Croy . 2008 . The hint factory: Automatic generation of contextualized help for existing computer aided instruction . In Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. 71–78 . John Stamper, Tiffany Barnes, Lorrie Lehmann, and Marvin Croy. 2008. The hint factory: Automatic generation of contextualized help for existing computer aided instruction. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. 71–78.
Cited by
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“More recently, automated hint generation systems have been employed to generate hints for any programming exercises [58].”
MethodCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
“Most of the work has been devoted to various approaches for the generation of effective hints [21, 22] and feedback [18].”
BackgroundKochmar, E., Vu, D. D., Belfer, R., Gupta, V., Serban, I. V., & Pineau, J. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00267-x · Cited by 112
“There is a growing body of research on automated hint generation for programming exercises (McBroom et al., 2019; Price et al., 2019).”
BackgroundYang, A. C. M., Lin, J.-Y., Lin, C.-Y., & Ogata, H. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100309 · Cited by 95
Can Language Models Employ the Socratic Method? Experiments with Code Debugging
Al-Hossami, E., Bunescu, R., Smith, J., & Teehan, R. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630799 · Cited by 60
Metadata
- Authors
- McBroom, J., Koprinska, I., & Yacef, K.
- Affiliations
- The University of Sydney (AU)
- Venue
- ACM Computing Surveys
- Published
- 2021-10-04
- Volume
- 54
- Issue
- 8
- Pages
- 1-27
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2970680470
- Cited by
- 39
- References
- 52
- Citations by year
- 2026: 1 · 2025: 10 · 2024: 9 · 2023: 11 · 2022: 4 · 2021: 3
- FWCI
- 5.71
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- automated feedback, programming education, data-driven evaluation, programming exercises
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.