Jutge.org: Characteristics and Experiences
Open accessAbstract
Jutge.org is an open educational online programming judge designed for students and instructors, featuring a repository of problems that is well organized by courses, topics, and difficulty. Internally, Jutge.org uses a secure and efficient architecture and integrates modern verification techniques, formal methods, static code analysis, and data mining. Jutge.org has exhaustively been used during the last decade at the Universitat Politècnica de Catalunya to strengthen the learning-by-doing approach in several courses. This paper presents the main characteristics of Jutge.org and shows its use and impact on a wide range of courses covering basic programming, data structures, algorithms, artificial intelligence, functional programming, and circuit design.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A learning website, Jutge.org, was developed with the features of a rich and well-organized problem repository; the website provides instant feedback and helps students to progressively solve problems and learn from their mistakes.
“a learning website, Jutge.org, was developed with the features of a rich and well-organized problem repository.e website provides instant feedback and helps students to progressively solve problems and learn from their mistakes [36].”
From A Review of Artificial Intelligence (AI) in Education from 2010 to 2020 (2021) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059 · see citing works
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,339 · see citing works
Hanus, M. D., & Fox, J. (2015) · Computers & Education · doi:10.1016/j.compedu.2014.08.019 · Cited by 1,918 · cited 1× · see citing works
“Wider implications of gamification in learning can be found, for instance, in [32], [33].”
BackgroundNuSMV 2: An OpenSource Tool for Symbolic Model Checking
Cimatti, A., Clarke, E., Giunchiglia, E., et al. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45657-0_29 · Cited by 1,664 · see citing works
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 561 · see citing works
Automatic test-based assessment of programming
Douce, C., Livingstone, D., & Orwell, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163409 · Cited by 416 · see citing works
Automated feedback generation for introductory programming assignments
Singh, R. K., Gulwani, S., & Solar-Lezama, A. (2013) · Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2491956.2462195 · Cited by 388 · see citing works
The boss online submission and assessment system
Joy, M. S., Griffiths, N., & Boyatt, R. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163407 · Cited by 282 · see citing works
Haskell: The Craft of Functional Programming
Thompson, S. (1996) · Kent Academic Repository (University of Kent) · Cited by 276 · see citing works
The effect of competition on learning in games
Çağıltay, N. E., Özçelik, E., & Ozcelik, N. S. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.04.001 · Cited by 256 · see citing works
Grading student programs using ASSYST
Jackson, D., & Usher, M. (1997) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/268084.268210 · Cited by 254 · see citing works
A critique of cyclomatic complexity as a software metric
Shepperd, M. (1988) · Software Engineering Journal · doi:10.1049/sej.1988.0003 · Cited by 240 · see citing works
On automated grading of programming assignments in an academic institution
Cheang, B., Kurnia, A., Lim, A. E. B., & Oon, W.-C. (2003) · Computers & Education · doi:10.1016/s0360-1315(03)00030-7 · Cited by 227 · see citing works
Automated feedback generation for introductory programming assignments
Singh, R. K., Gulwani, S., & Solar-Lezama, A. (2013) · ACM SIGPLAN Notices · doi:10.1145/2499370.2462195 · Cited by 137 · see citing works
Kurnia, A., Lim, A. E. B., & Cheang, B. (2001) · Computers & Education · doi:10.1016/s0360-1315(01)00018-5 · Cited by 111 · see citing works
Nguyen, A., Piech, C., Huang, J., & Guibas, L. J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568023 · Cited by 108 · see citing works
Grading student programs using ASSYST
Jackson, D., & Usher, M. (1997) · ACM SIGCSE Bulletin · doi:10.1145/268085.268210 · Cited by 107 · see citing works
A theory and implementation of sequential hardware equivalence
Pixley, C. (1992) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/43.180261 · Cited by 103 · see citing works
Ask-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback
Gerdes, A., Heeren, B. J., Jeuring, J. T., & van Binsbergen, L. T. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-015-0080-x · Cited by 96 · see citing works
A Web-Based Intelligent Tutoring System for Computer Programming
Butz, C. J., Song, H., & Maguire, R. B. (2004) · Web Intelligence · doi:10.1109/wi.2004.15 · Cited by 69 · see citing works
A Review of AI-Supported Tutoring Approaches for Learning Programming
Le, N.-T., Strickroth, S., Groß, S., & Niels Pinkwart, C. (2013) · Studies in computational intelligence · doi:10.1007/978-3-319-00293-4_20 · Cited by 52 · see citing works
A Web-Based Intelligent Tutoring System for Computer Programming
Butz, C. J., Hua, S., & Maguire, R. B. (2005) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1109/wi.2004.10104 · Cited by 48 · see citing works
Experiences in Digital Circuit Design Courses: A Self-Study Platform for Learning Support
Bañeres, D., Clarisó, R., Jorba, J., & Serra, M. (2014) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2014.2320919 · Cited by 47 · see citing works
Analysing Student Programs in the PHP Intelligent Tutoring System
Weragama, D., & Reye, J. (2014) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-014-0014-z · Cited by 36 · see citing works
Application of an Online Judge & Contester System in Academic Tuition
Kosowski, A., Małafiejski, M., & Noiński, T. (2008) · Springer eBooks · doi:10.1007/978-3-540-78139-4_31 · Cited by 27 · see citing works
Solving Digital Logic Assignments with Automatic Verification in SCORM Modules
Damm, M., Bauer, F., & Zucker, G. (2009) · Cited by 13 · see citing works
Zimmerman, D. M., Kiniry, J. R., & Fairmichael, F. (2011) · doi:10.1109/cseet.2011.5876114 · Cited by 10 · see citing works
LogiFlash – A Flash-based Logic-Simulator for educational Purposes
Damm, M., Klauer, B., & Waldschmidt, K. (2003) · EdMedia: World Conference on Educational Media and Technology · Cited by 10 · see citing works
Integrating formal verification in an online judge for e-Learning logic circuit design
Pedro, J. D. S., Carmona, J., Cortadella, J., et al. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157268 · Cited by 8 · see citing works
Self-citationThe Use of Cyclomatic Complexity Metrics in Programming Performance's Assessment
Mohamed, N. B., Sulaiman, R. F. R., & Endut, W. R. W. (2013) · Procedia - Social and Behavioral Sciences · doi:10.1016/j.sbspro.2013.07.119 · Cited by 7 · see citing works
Better Feedback for Educational Online Judges
Mani, A., Venkataramani, D., Petit, J., et al. (2014) · International Conference on Computer Supported Education · doi:10.5220/0004842801760183 · Cited by 6 · see citing works
Self-citationUse of HDLs in teaching of computer hardware courses
Vranesic, Z. G., & Brown, S. D. (2003) · doi:10.1145/1275521.1275542 · Cited by 5 · see citing works
A MOOC on Approaches to Machine Translation
Costa‐jussà, M. R., Formiga, L., Torrillas, O., et al. (2015) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v16i6.2145 · Cited by 4 · see citing works
(1972) · Software Practice and Experience · doi:10.1002/spe.4380020110 · Cited by 4 · see citing works
Duch, A., Petit, J., Rodríguez-Carbonell, E., et al. (2013) · International Conference on Computer Supported Education · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- J. L. Bez, N. A. Tonin, and P. R. Rodegheri. URI online judge academic. In 2014 9th Int. Conf. on Computer Science Education, pages 149-152, 2014.
- A. Cimatti et al. NuSMV2: An OpenSource Tool for Symbolic Model Checking. In Proc. Int. Conf. on Computer-Aided Verification (CAV 2002), volume 2404 of LNCS. Springer, 2002.
- A. Duch, J. Petit, E. Rodríguez-Carbonell, and S. Roura. Fun in CS2. In 5th Int. Conf. on Computer Supported Education, pages 437-442. SCITEPRESS, 2013.
- O. Giménez, J. Petit, and S. Roura. Programació 1: A pure problem-oriented approach for a CS1 course. In Informatics Education Europe IV, pages 185-192, 2009.
- A. Nguyen, C. Piech, J. Huang, and L. Guibas. Codewebs: scalable homework search for massive open online programming courses. Procs. of the 23rd int. conf. on WWW, pages 491-502, 2014.
Cited by
Sorted by citations
A Review of Artificial Intelligence (AI) in Education from 2010 to 2020
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., & Li, Y. (2021) · Complexity · doi:10.1155/2021/8812542 · Cited by 1,364 · see citing works
“Besides, a learning website, Jutge.org, was developed with the features of a rich and well-organized problem repository. website provides instant feedback and helps students to progressively solve problems and learn from their mistakes [36].”
BackgroundArtificial Intelligence in Health Professions Education
Shankar, P. R. (2022) · Archives of Medicine and Health Sciences · doi:10.4103/amhs.amhs_234_22 · Cited by 40 · see citing works
- Authors
- Petit, J., Roura, S., Carmona, J., Cortadella, J., Duch, J., Gimenez, O., Mani, A., Mas, J., Rodriguez-Carbonell, E., Rubio, E., Pedro, E. D. S., & Venkataramani, D.
- Affiliations
- Universitat Politècnica de Catalunya (ES)
- SASTRA University (IN)
- Google (United States) (US)
- Venue
- IEEE Transactions on Learning Technologies
- Published
- 2018-07-01
- Volume
- 11
- Issue
- 3
- Pages
- 321-333
- ISSN
- 1939-1382 (ISSN-L)
- 2372-0050
- OpenAlex
- W2727870302
- Cited by
- 33
- References
- 35
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 2 · 2023: 7 · 2022: 5 · 2021: 9 · 2020: 5 · 2019: 1 · 2018: 1
- FWCI
- 2.29
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- IEEE Transactions on Learning Technologies (publishedVersion)
- UPCommons institutional repository (Universitat Politècnica de Catalunya) (submittedVersion, other-oa)
- Funders
- Spanish Ministry for Economy and Competitiveness (MINECO)
- European Union
- FEDER (TIN2013-46181-C2-1-R, TIN2015-69175-C4-3-R, SGR 2014 1034)
- ALBCOM
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Educational Robotics and Engineering
- Keywords
- open educational resources, programming education, hands-on learning, static code analysis, formal methods, data mining, programming course design, Universitat Politècnica de Catalunya
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
