What Do We Know About Hackathon Outcomes and How to Support Them? – A Systematic Literature Review
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Hackathons can facilitate both upskilling and reskilling by providing participants with the opportunity to learn new skills and knowledge and apply them in a real -world setting (Medina Angarita, & Nolte, 2020).”
From The Impact of Artificial Intelligence on Workers' Skills: Upskilling and Reskilling in Organisations (2023) · cites this paper as Medina Angarita, & Nolte, 2020
Citations
Cites
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537 · cited 2×
“In order to answer the aforementioned research questions, we conducted a systematic literature review based on the guidelines described by Kitchenham et al. [27].”
MethodBrereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394 · cited 1×
“We searched for publications from 2010 to 2020 to focus on the most recent work about hackathons We conducted our search using online libraries proposed by Brereton [7]: IEEExplore, ACM Digital library, and Google Scholar, as well as Scopus and Web of Science.”
MethodDigital Innovation: The Hackathon Phenomenon
Briscoe, G. (2014) · Queen Mary Research Online (Queen Mary University of London) · Cited by 293
Hackathons as an Informal Learning Platform
Nandi, A., & Mandernach, M. (2016) · Proceedings of the 47th ACM Technical Symposium on Computing Science Education · doi:10.1145/2839509.2844590 · Cited by 198 · cited 3×
“Intangible hackathon outcomes include participants learning about the main issue of a hackathon [50], new technologies [9, 46], or acquiring industry and in-university skills [33].”
BackgroundMethodHow to Hackathon: Socio-technical Tradeoffs in Brief, Intensive Collocation
Trainer, E. H., Kalyanasundaram, A., Chaihirunkarn, C., & Herbsleb, J. D. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2819946 · Cited by 136 · cited 6×
“Tangible outcomes may also include non-technical artifacts such as visualizations [40, 43, 49], new or improved documentation [46] and publications [49].”
BackgroundInfluentialDesigning Corporate Hackathons With a Purpose: The Future of Software Development
Pe‐Than, E. P. P., Nolte, A., Filippova, A., Bird, C., Scallen, S., & Herbsleb, J. D. (2018) · IEEE Software · doi:10.1109/ms.2018.290110547 · Cited by 98 · cited 4×
“Most research on hackathons, however, currently focuses on studying singular events in specific domains covering aspects such as how to organize a hackathon [37] and how teams self-organize [49].”
BackgroundInfluentialSelf-citationHackathons as Community-Based Learning: a Case Study
Lara, M., & Lockwood, K. (2016) · TechTrends · doi:10.1007/s11528-016-0101-0 · Cited by 86 · cited 4×
“takes place as a competitive event, feedback [24] can also be provided by the jury, who evaluates projects and selects winner teams that receive prizes [29].”
BackgroundInfluentialUnleashing innovation through internal hackathons
Rosell, B., Kumar, S., & Shepherd, J. (2014) · IEEE Innovations in Technology Conference (InnoTek) · doi:10.1109/innotek.2014.6877369 · Cited by 79 · cited 7×
“A hackathon commonly begins with a kickoff, such as a keynote [41].”
BackgroundInfluentialOlson, K. R., Walsh, M., Garg, P., Steel, A., Mehta, S., Santorino, D., Petersen, R., GUARINO, A. J., Bailey, E., & Bangsberg, D. R. (2017) · BMJ Innovations · doi:10.1136/bmjinnov-2016-000147 · Cited by 77 · cited 3×
“…Coaching and mentoring to the winning teams [1, 35], a showcase of technical artifacts developed during an event at a forum [1], post-hackathon prizes [31, 35], the release of the productive version of technical artifacts [1, 20], recruitment of new team members [35], and grant writing [35].”
BackgroundHacking with NPOs: Collaborative Analytics and Broker Roles in Civic Data Hackathons
Hou, Y., & Wang, D. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134688 · Cited by 67 · cited 1×
“The lack of specific tools to support intrateam communication has also been addressed by Hou and Wang [25].”
BackgroundHackathons in the formal learning process
da Gama, K. S., Gonçalves, B. A., & Alessio, P. M. (2018) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3197091.3197138 · Cited by 67 · cited 3×
“As hackathon outcomes, we perceive the diverse direct results of a hackathon, such as pro-totypes, networking, learning and others [21].”
BackgroundNolte, A., Pe‐Than, E. P. P., Filippova, A., Bird, C., Scallen, S., & Herbsleb, J. D. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274398 · Cited by 65 · cited 3×
“[34] reported that preparation prior to a hackathon can influence the continuation of hackathon projects [34].”
BackgroundInfluentialSelf-citationThe innovation-driven hackathon: one means for accelerating innovation
Frey, F. J., & Luks, M. (2016) · European Conference on Pattern Languages of Programs · doi:10.1145/3011784.3011794 · Cited by 64 · cited 3×
“…Coaching and mentoring to the winning teams [1, 35], a showcase of technical artifacts developed during an event at a forum [1], post-hackathon prizes [31, 35], the release of the productive version of technical artifacts [1, 20], recruitment of new team members [35], and grant writing [35].”
BackgroundThe hackathon model to spur innovation around global mHealth
Angelidis, P. A., Berman, L., Casas-Perez, M. D. L. L., Celi, L. A., Dafoulas, G., Dagan, A. S., Escobar, B., López, D. M., Noguez, J., Osorio-Valencia, J. S., et al. (2016) · Journal of Medical Engineering & Technology · doi:10.1080/03091902.2016.1213903 · Cited by 61 · cited 2×
“Participants can also engage in networking [46] by meeting new people, thus, creating opportunities for collaboration [3, 39].”
BackgroundHow Can Hackathons Accelerate Corporate Innovation?
Flores, M., Golob, M., Maklin, D., Herrera, M., Tucci, C. L., Al-Ashaab, A., Williams, L., Encinas‐Oropesa, A., Martinez, V., Zaki, M., et al. (2018) · IFIP advances in information and communication technology · doi:10.1007/978-3-319-99704-9_21 · Cited by 61 · cited 2×
“A full list of all references considered in this literature review is available here: https://bit.ly/2CDIezF O6 Ideas [40, 47] O7 Entrepreneurship [11, 31] O8 Fostering existing enterprise [10, 19] O9 Fostering awareness about hackathon theme [2, 46, 50] Intangible hackathon outcomes.”
BackgroundReappropriating Hackathons: The Production Work of the CHI4Good Day of Service
Porter, E., Bopp, C., Gerber, E. M., & Voida, A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025637 · Cited by 60 · cited 1×
“Participants can also engage in networking [46] by meeting new people, thus, creating opportunities for collaboration [3, 39].”
BackgroundFilippova, A., Trainer, E., & Herbsleb, J. D. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.22 · Cited by 55 · cited 1×
“[18] found that “brainstorming impacts satisfaction with outcome indirectly by increasing clarity of goals” [18], however, details regarding the particulars of the ideation process during hackathons are still missing.”
BackgroundBirbeck, N., Lawson, S., Morrissey, K., Rapley, T., & Olivier, P. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025931 · Cited by 53 · cited 7×
“They have a large array of responsibilities [42] such as marketing an event [6, 17, 48], defining prerequi-sites for outcomes [42], and recruiting participants [1, 5, 10] based on specific participant selection criteria [5, 10, 22].”
BackgroundInfluentialHackathons as a means of accelerating scientific discoveries and knowledge transfer
Ghouila, A., Siwo, G., Entfellner, J.-B. D., Panji, S., Button-Simons, K. A., Davis, S., Fadlelmola, F. M., Participants, T. D. O. M. H., Ferdig, M. T., & Mulder, N. (2018) · Genome Research · doi:10.1101/gr.228460.117 · Cited by 51 · cited 3×
“In addition, Ghoulia et al. [23] stated that by establishing a strong intrateam relationship , teams may be more likely to continue working together after the hackathon ends.”
BackgroundHack for the homeless: A humanitarian technology hackathon
Linnell, N., Figueira, S. M., Chintala, N., Falzarano, L., & Ciancio, V. (2014) · IEEE Global Humanitarian Technology Conference (GHTC) · doi:10.1109/ghtc.2014.6970341 · Cited by 43 · cited 4×
“The goals of hackathon organizers also certainly affect the design of a hackathon which can potentially the sustainability of hackathon outcomes [5, 30].”
BackgroundInfluentialFROM APPFEST TO ENTREPRENEURS: USING A HACKATHON EVENT TO SEED A UNIVERSITY STUDENT-LED ENTERPRISE
Cobham, D., Gowen, C., Jacques, K. L., Laurel, J., & Ringham, S. (2017) · INTED proceedings · doi:10.21125/inted.2017.0265 · Cited by 43 · cited 7×
“Other intangible outcomes may include entrepreneurship [11, 31] (i.e. the creation of new startups), fostering existing enterprises [19], and fostering awareness about the theme of a hackathon [2, 50].”
BackgroundInfluentialBeyond the Classroom: The Impact of a University-Based Civic Hackathon Addressing Homelessness
Wilson, J. H., Bender, K., & DeChants, J. P. (2019) · Journal of Social Work Education · doi:10.1080/10437797.2019.1633975 · Cited by 39 · cited 9×
“Participants commonly attend hackathons based on individual motivations such as e.g. having fun or learning [40, 50].”
BackgroundMethodInfluentialHackathon – A Method for Digital Innovative Success: A Comparative Descriptive Study
Soltani, P. M., Pessi, K., Ahlin, K., & Wernered, I. (2014) · Cited by 36
Project-Based Learning Among Engineering Students During Short-Form Hackathon Events
Horton, P., Jordan, S. S., Weiner, S. W., & Lande, M. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--30901 · Cited by 36 · cited 1×
“If a hackathon takes place as a competitive event, feedback [24] can also be provided by the jury, who evaluates projects and selects winner teams that receive prizes [29].”
BackgroundTHE USE OF THE “HACKATHON” IN DESIGN EDUCATION: AN OPPORTUNISTIC EXPLORATION.
Page, F., Sweeney, S., Bruce, F. S., & Baxter, S. (2016) · Discovery Research Portal (University of Dundee) · Cited by 35
Alba, M., Avalos, M., Guzman, C., & Larios, V. M. (2016) · IEEE International Smart Cities Conference (ISC2) · doi:10.1109/isc2.2016.7580877 · Cited by 33 · cited 5×
“…have offered: Coaching and mentoring to the winning teams [1, 35], a showcase of technical artifacts developed during an event at a forum [1], post-hackathon prizes [31, 35], the release of the productive version of technical artifacts [1, 20], recruitment of new team members [35], and…”
BackgroundInfluential‘You can dance your prototype if you like’: independent filmmakers adapting the hackathon
Karlsen, J., & Løvlie, A. S. (2017) · Digital Creativity · doi:10.1080/14626268.2017.1351992 · Cited by 33 · cited 3×
“In a team, participants typically agree on the tools [9] they use to work on their project [38] and select a leader [16, 26] for the duration of a hackathon.”
BackgroundCitizens’ Motivations for Engaging in Open Data Hackathons
Purwanto, A., Zuiderwijk, A., & Janssen, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-27397-2_11 · Cited by 32 · cited 5×
“Participants commonly attend hackathons based on individual motivations such as e.g. having fun or learning [40, 50].”
BackgroundInfluentialCobham, D., Gowen, C., Hargrave, B., Jacques, K. L., Laurel, J., & Ringham, S. (2017) · EDULEARN proceedings · doi:10.21125/edulearn.2017.1172 · Cited by 28 · cited 3×
“In this case, there were more team members than tasks needed to be completed, which meant that “ too often some members were idle awaiting others to complete dependent tasks ” [12].”
BackgroundCommunity and Code: Nine Lessons from Nine NESCent Hackathons
Stoltzfus, A., Rosenberg, M. S., Lapp, H., Budd, A., Cranston, K., Pontelli, E., Oliver, S., & Vos, R. A. (2017) · F1000Research · doi:10.12688/f1000research.11429.1 · Cited by 22 · cited 6×
“In order to sustain networking after the hackathon had ended, participants can join a mailing list [46], but the extent to which a mailing list can sustain networking remains questionable.”
BackgroundInfluentialHealth Hackathons Drive Affordable Medical Technology Innovation Through Community Engagement
Mantzavinou, A., Ranger, B. J., Gudapakkam, S., Hutchins, K. G. B., Bailey, E., & Olson, K. R. (2018) · Springer eBooks · doi:10.1007/978-3-319-91068-0_8 · Cited by 22 · cited 4×
“…Coaching and mentoring to the winning teams [1, 35], a showcase of technical artifacts developed during an event at a forum [1], post-hackathon prizes [31, 35], the release of the productive version of technical artifacts [1, 20], recruitment of new team members [35], and grant writing [35].”
BackgroundInfluentialChandrasekaran, S., Juckeland, G., Lin, M., Otten, M., Pleiter, D., Stone, J. E., Lucio-Vega, J., Zingale, M., & Foertter, F. (2018) · Computing in Science & Engineering · doi:10.1109/mcse.2018.042781332 · Cited by 20 · cited 4×
“In a team, participants typically agree on the tools [9] they use to work on their project [38] and select a leader [16, 26] for the duration of a hackathon.”
BackgroundMethodInfluentialBoisen, K. A., Boisen, A. S., Thomsen, S. L., Matthiesen, S. M., Hjerming, M., & Hertz, P. G. (2015) · International Journal of Adolescent Medicine and Health · doi:10.1515/ijamh-2015-0072 · Cited by 19 · cited 4×
“They have a large array of responsibilities [42] such as marketing an event [6, 17, 48], defining prerequi-sites for outcomes [42], and recruiting participants [1, 5, 10] based on specific participant selection criteria [5, 10, 22].”
BackgroundInfluentialHow do the design features of health hackathons contribute to participatory medicine?
Day, K., Humphrey, G., & Cockcroft, S. (2017) · AJIS. Australasian journal of information systems/AJIS. Australian journal of information systems/Australian journal of information systems · doi:10.3127/ajis.v21i0.1383 · Cited by 19 · cited 3×
“Teams have different sizes that can also fluctuate during the course of a hackathon [14].”
BackgroundSadovykh, A., Truşcan, D., Pierini, P., Widforss, G., Ashraf, A., Brunelière, H., Smrž, P., Bagnato, A., Afzal, W., & Hortelano, A. E. (2019) · Design, Automation, and Test in Europe · doi:10.23919/date.2019.8715247 · Cited by 18 · cited 1×
“They have a large array of responsibilities [42] such as marketing an event [6, 17, 48], defining prerequi-sites for outcomes [42], and recruiting participants [1, 5, 10] based on specific participant selection criteria [5, 10, 22].”
BackgroundDoes it matter why we hack? – Exploring the impact of goal alignment in hackathons
Angarita, M. A. M., & Nolte, A. (2019) · Gesellschaft für Informatik (GI) · doi:10.18420/ecscw2019_ep01 · Cited by 18 · cited 1×
“It is still uncertain how participant goals [32] relate to their behavior during hackathons and how their individual goals can affect the sustainability of outcomes.”
BackgroundSelf-citationBig Team Game Jams: A framework to emulate big production using game jams with big teams
Farhan, E., & Kocher, M. (2016) · International Conference on Game Jams, Hackathons, and Game Creation Events (ICGJ) · doi:10.1145/2897167.2897171 · Cited by 14 · cited 1×
“In a team, participants typically agree on the tools [9] they use to work on their project [38] and select a leader [16, 26] for the duration of a hackathon.”
BackgroundSocial Aspects of the Game Development Process in the Global Gam Jam
Pirker, J., Lesjak, I., Punz, A., & Drachen, A. (2018) · International Conference on Game Jams, Hackathons, and Game Creation Events (ICGJ) · doi:10.1145/3196697.3196700 · Cited by 12 · cited 3×
“In a team, participants typically agree on the tools [9] they use to work on their project [38] and select a leader [16, 26] for the duration of a hackathon.”
BackgroundMethodMapathons and Hackathons to Crowdsource the Generation and Usage of Geographic Data
da Gama, K. S., Rautenbach, V., Green, C., Gonçalves, B. A., Coetzee, S., Fourie, N., & Sastry, N. (2019) · International Conference on Game Jams, Hackathons, and Game Creation Events (ICGJ) · doi:10.1145/3316287.3316293 · Cited by 12 · cited 5×
“They have a large array of responsibilities [42] such as marketing an event [6, 17, 48], defining prerequi-sites for outcomes [42], and recruiting participants [1, 5, 10] based on specific participant selection criteria [5, 10, 22].”
BackgroundInfluentialStrategies for Empowering Collective Design
Aryana, B., Naderi, E., & Balis, G. (2019) · The Design Journal · doi:10.1080/14606925.2019.1594931 · Cited by 11 · cited 1×
“Some teams form clear objectives and requirements for their project [4, 36] while others choose a less structured approach.”
BackgroundFostering a culture of collaboration through playful Design Jams
Tang, T., & Vezzani (2017) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 8
Dainotti, A., Katz-Bassett, E., & Dimitropolous, X. (2018) · ACM SIGCOMM Computer Communication Review · doi:10.1145/3243157.3243169 · Cited by 7 · cited 2×
“They might also provide opportunities for participants to meet prior to the event [13, 17, 26].”
BackgroundConference report: 2018 materials and data science hackathon (MATDAT18)
Ferguson, A. L., Mueller, T., Rajasekaran, S., & Reich, B. J. (2019) · Molecular Systems Design & Engineering · doi:10.1039/c9me90018g · Cited by 4 · cited 3×
“They might also provide opportunities for participants to meet prior to the event [13, 17, 26].”
Background
Research works printed but not linked: 5
- Chandrasekaran, S., et al.: The OLCF GPU hackathon series: the story behind advancing scientific applications with a sustained impact. In: EduHPC-17: Workshop on Education for High-Performance Computing (2017)
- Drouhard, M., et al.: A typology of hackathon events. In: Conference on Computer-Supported Cooperative Work and Social Media (2017)
- Safarova, B., et al.: Learning from collaborative integration. In: 33rd eCAADe Conference (2015)
- Starov, O., et al.: Hacking the Innovations with University-Industry Hackathons. University-Industry Interaction Conference (2015)
- Suominen, A.H., et al.: Educational hackathon: innovation contest for innovation pedagogy. innovation, the name of the game. In: Proceedings of the 2018 ISPIM Innovation Conference (Stockholm), 18 (2018)
Cited by
The Impact of Artificial Intelligence on Workers' Skills: Upskilling and Reskilling in Organisations
Morandini, S., Fraboni, F., De Angelis, M., Puzzo, G., Giusino, D., & Pietrantoni, L. (2023) · Informing Science: The International Journal of an Emerging Transdiscipline · doi:10.28945/5078 · Cited by 439
Hack your organizational innovation: literature review and integrative model for running hackathons
Heller, B., Amir, A., Waxman, R., & Maaravi, Y. (2023) · Journal of Innovation and Entrepreneurship · doi:10.1186/s13731-023-00269-0 · Cited by 82
“In addition, issue-oriented hack-athons require diverse players’ joint work and expertise, complicating the realization of its products after the event (Angarita & Nolte, 2020).”
BackgroundGarcia, M. B. (2022) · Computer Applications in Engineering Education · doi:10.1002/cae.22564 · Cited by 67
“The only available systematic reviews on the hackathon domain have primarily focused on the connection between the event outcomes and design aspects [50], and how to organize it online [34].”
BackgroundBoosting innovative business ideas through hackathons. The “Hack for Travel” case study
Franco, S., Presenza, A., & Petruzzelli, A. M. (2021) · European Journal of Innovation Management · doi:10.1108/ejim-06-2021-0300 · Cited by 64
“In any case, the time, size and shape of a hackathon differ depending on the competencies available to the organizers and the considerable resources to prepare, run and follow-up that they are called to invest on the event (Medina Angarita and Nolte, 2020).”
BackgroundHow to organize an in-person, online or hybrid hackathon -- A revised planning kitPreprint
Affia-Jomants, A.-A. O., da Gama, K. S., Herbsleb, J. D., & Nolte, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.08025 · Cited by 60
Self-citation
Metadata
- Authors
- Medina Angarita, M. A., & Nolte, A.
- Affiliations
- University of Tartu (EE)
- Corresponding author
- Nolte, Alexander
- ORCID
- Nolte, Alexander
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2020
- Pages
- 50-64
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W3081921942
- Cited by
- 86
- References
- 43
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 25 · 2023: 12 · 2022: 14 · 2021: 9 · 2020: 1
- FWCI
- 16.34
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Lecture notes in computer science (publishedVersion)
- Topics
- Physical Sciences › Engineering › Biomedical Engineering › Biomedical and Engineering Education
- Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- hackathons, innovation, community building
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.