A Survey on Online Judge Systems and Their Applications
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Online judges are systems designed for the reliable evaluation of algorithm source code submitted by users, which is next compiled and tested in a homogeneous environment. Online judges are becoming popular in various applications. Thus, we would like to review the state of the art for these systems. We classify them according to their principal objectives into systems supporting organization of competitive programming contests, enhancing education and recruitment processes, facilitating the solving of data mining challenges, online compilers and development platforms integrated as components of other custom systems. Moreover, we introduce a formal definition of an online judge system and summarize the common evaluation methodology supported by such systems. Finally, we briefly discuss an Optil.io platform as an example of an online judge system, which has been proposed for the solving of complex optimization problems. We also analyze the competition results conducted using this platform. The competition proved that online judge systems, strengthened by crowdsourcing concepts, can be successfully applied to accurately and efficiently solve complex industrial- and science-driven challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No free lunch theorems for optimization
Wolpert, D. H., & Macready, W. G. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585893 · Cited by 14,237
From game design elements to gamefulness: defining "gamification"
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011) · Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments · doi:10.1145/2181037.2181040 · Cited by 8,402 · cited 1×
“Some of the online judges introduce additional gamification elements (Deterding et al. 2011) to strengthen the commitment and excitation of the students.”
BackgroundSystematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 1×
“Following the guidelines for preparing systematic literature reviews (Keele 2007; Kitchenham et al. 2009), we defined the following acceptance criteria, all of which should be satisfied by the system considered in the review; i.e., the system: (1) supports any subset of evaluation procedure phases,…”
MethodEdmonds, J. (1965) · Canadian Journal of Mathematics · doi:10.4153/cjm-1965-045-4 · Cited by 2,428 · cited 1×
“The aforementioned observation was confirmed for various combinatorial problems and became a cause of the rise of computational complexity theory (Edmonds 1965; Garey and Johnson 1979).”
BackgroundSchleimer, S., S, W. D., & Aiken, A. (2003) · Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data · doi:10.1145/872757.872770 · Cited by 1,099 · cited 1×
“Kaya and ¨Ozel in (Kaya and ¨Ozel 2012, 2014) presented a similar approach, but enriched with a plagiarism detection package called Moss (Schleimer et al. 2003).”
MethodDhar, V. (2013) · Communications of the ACM · doi:10.1145/2500499 · Cited by 856 · cited 1×
“The Kaggle platform focuses mainly on data-mining problems related to the concept of data science, which has recently gained popularity (Dhar 2013).”
BackgroundKaya, M., & Özel, S. A. (2015) · Computer Applications in Engineering Education · doi:10.1002/cae.21606 · Cited by 38
Cited by
Automatic feedback in online learning environments: A systematic literature review
Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100027 · Cited by 440
Escapp: A Web Platform for Conducting Educational Escape Rooms
Lopez-Pernas, S., Gordillo, A., Barra, E., & Quemada, J. (2021) · IEEE Access · doi:10.1109/access.2021.3063711 · Cited by 51
“Authors of [58] reviewed several software tools for conducting programming competitions, such as Mooshak, which has been successfully used for hosting activities of this kind in the classroom [59].”
MethodPereira, F. D., Rodrigues, L., Henklain, M. H. O., Freitas, H., Oliveira, D. F., Cristea, A. I., Carvalho, L., Isotani, S., Benedict, A., Dorodchi, M., et al. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3224121 · Cited by 26
“More-over, to fill the gap of POJs on the lack of problem organization [1], [7], [8], we use a deep learning (DL) architecture and natural language processing (NLP) techniques (details in Section IV-C1).”
BackgroundMethodInfluential(2020) · Durham Research Online (Durham University) · Cited by 21
Metadata
- Authors
- Wasik, S., Antczak, M., Badura, J., Laskowski, A., & Sternal, T.
- Venue
- ACM Computing Surveys
- Published
- 2018-01-04
- Volume
- 51
- Issue
- 1
- Pages
- 1-34
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2767123881
- Cited by
- 189
- References
- 7
- FWCI
- 25.07
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.