The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 3 of the 18 sources its claims cite were checked for support.
Game industry problems: An extensive analysis of the gray literature
Open accessOverview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 5 Not valid · 14 Not checked · 110 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“We analyse 200 postmortems written between 1997 and 2019 available in our public dataset [19]”
- Also stated in: § 3.1. Overall Dataset · plain
“The dataset contains 200 video-game projects from 1997 to 2019”
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > Method:
What the sources say
- Disputedcites [19] Politowski, 2018, Learning from the past: A process recommendation system for video game projects using postmortems experiences, Inf. Softw. Technol. sourceExperimental — accuracy not yet measured
Our copy of Politowski et al. (2018) reports a different corpus and a different time span: Section 4.1 (p. 6) states 'The total amount of game projects was 234 from 1997 (which is the oldest postmortem we found) to 2016,' and in the same section 'among the 100 postmortems that we analyzed we kept only 55 that had sufficient information to create a development process'; Section 7 (p. 17) confirms tests used 'a dataset with 55 samples.' Only the 1997 start year matches the claim; neither a count of 200 postmortems nor coverage to 2019 appears anywhere (searches for the dataset size, the year range, and a publicly released dataset returned only these passages, plus Section 3.1 p. 4, which describes gathering postmortems from Gamasutra/Wikipedia, not publishing a public dataset). The '200 postmortems written between 1997 and 2019 available in our public dataset' figure is therefore contrary to what this source states rather than supported by it. (LLM, conf=0.72)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Also stated in: § 3.1. Overall Dataset · plain
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Although open source, Godot 18 is not yet as mature as its proprietary counterparts [18].”
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > 6.9. Inadequate or Missing Tools
What the sources say
- Could not verifycites [18] Nassif, 2016, Neural network models for software development effort estimation: a comparative study, Neural Comput. Appl. sourceExperimental — accuracy not yet measured
The Introduction (p. 2) compares neural-network models for software effort estimation, and the Datasets section (p. 6) describes software-project data. Searches for Godot, game engines, and proprietary counterparts found no passage addressing Godot’s maturity. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These postmortems include 927 problems that we categorized into 20 types.”
- Also stated in: § 3.1. Overall Dataset · plain
“describing 927 problems.”
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > Method:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
- Also stated in: § 3.1. Overall Dataset · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 78 problems pertain to the three platforms (multi-platform games).”
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > Group Production
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These postmortems include 927 problems that we categorized into 20 types.”
- Also stated in: § Abstract · plain
“resulting in 927 problems divided into 20 types.”
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > Method:
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
- Also stated in: § Abstract · plain
53 more claims
Create a free account to see all 129 claims and to verify the 15 references still open.
- It may state two separate claims as one. 50
- Probably not a substantial claim. 37
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Post-mortem studies have shown that difficulties in production often stem from soft skills, team coordination, and organisational dynamics rather than mere technical obstacles [54].”
From Generative AI in Game Development: A Qualitative Research Synthesis (2026) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > 3.2. Problems Over the Years
“To normalise the numbers of problems each year, we divide their numbers by the total numbers of problems that year.”
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > Research bias:
“we mitigated this imbalance by dividing the numbers of problems by the numbers of postmortems per year for the historical analysis.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > Method:
“Problems related to teams increased in the last decade;”
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > 5.2. Petrillo et al., 2009
“we reported that problems related to teams and communication remain constant over the years.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > Third-Party Technology
“in our dataset, only one problem is related to unit testing.”
§ Game Industry Problems: an Extensive Analysis of the Gray Literature > 7.4. Game Testing
“Indeed, postmortems mention playtesting but do not mention unit testing or integration testing.”
Citations
Cites
The mythical man-month: essays on software engineering
(1996) · Choice Reviews Online · doi:10.5860/choice.33-5142 · Cited by 2,452
Godel, Escher, Bach: An Eternal Golden Braid.
Lieberson, J., & Hofstadter, D. R. (1980) · The Journal of Philosophy · doi:10.2307/2025598 · Cited by 1,914
Gray literature: An important resource in systematic reviews
Paez, A. (2017) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12266 · Cited by 1,459
A Theory of Fun for Game Design
Koster, R., & Wright, W. (2004) · Cited by 1,110
Grey literature: An important resource in systematic reviews
Páez, A. (2017) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12265 · Cited by 603
Grounded theory in software engineering research: a critical review and guidelines
Stol, K.-J., Ralph, P., & Fitzgerald, B. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884833 · Cited by 470 · cited 1×
“coding technique from Grounded Theory [25].”
Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0
Bourque, P., & Fairley, R. E. (2014) · IEEE Computer Society Press eBooks · Cited by 456
Game Development: Harder Than You Think
Blow, J. (2004) · Queue · doi:10.1145/971564.971590 · Cited by 176
Requirements engineering and the creative process in the video game industry
Callele, D., Neufeld, E. M., & Schneider, K. A. (2005) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2005.58 · Cited by 149
Software Engineering Challenges in Game Development
Kanode, C. M., & Haddad, H. M. (2009) · International Conference on Information Technology: New Generations · doi:10.1109/itng.2009.74 · Cited by 145
Rethinking Productivity in Software Engineering
Sadowski, C., & Zimmermann, T. (2019) · Apress eBooks · doi:10.1007/978-1-4842-4221-6 · Cited by 144
What went wrong? A survey of problems in game development
Petrillo, F. D. S., Pimenta, M. S., Trindade, F., & Dietrich, C. A. (2009) · Computers in entertainment · doi:10.1145/1486508.1486521 · Cited by 131
Self-citationNeural network models for software development effort estimation: a comparative study
Nassif, A. B., Azzeh, M., Capretz, L. F., & Ho, D. (2016) · Neural Computing and Applications · doi:10.1007/s00521-015-2127-1 · Cited by 122
VIDEOGAMES AS INTERACTIVE EXPERIENTIAL PRODUCTS AND THEIR MANNER OF DEVELOPMENT
Tschang, F. T. (2005) · International Journal of Innovation Management · doi:10.1142/s1363919605001198 · Cited by 102
What went right and what went wrong: an analysis of 155 postmortems from game development
Washburn, M. P., Sathiyanarayanan, P., Nagappan, M., Zimmermann, T., & Bird, C. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889253 · Cited by 75
Studying the urgent updates of popular games on the Steam platform
Lin, D., Bezemer, C., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9480-2 · Cited by 68
Kude, T., Mithas, S., Schmidt, C. T., & Heinzl, A. (2019) · Information Systems Research · doi:10.1287/isre.2019.0856 · Cited by 67
Why Does Code Review Work for Open Source Software Communities?
Alami, A., Cohn, M. L., & Wąsowski, A. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00111 · Cited by 56
Crunch Time: The Reasons and Effects of Unpaid Overtime in the Games Industry
Edholm, H. H., Lidström, M., Steghöfer, J.-P., & Burden, H. (2017) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2017.18 · Cited by 41
Dataset of Video Game Development Problems
Politowski, C., Petrillo, F. D. S., Ullmann, G. C., Werly, J. D. A., & Guéhéneuc, Y. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387486 · Cited by 37
Self-citationPolitowski, C., Fontoura, L. M., Petrillo, F., & Guéhéneuc, Y.-G. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.04.003 · Cited by 17
Self-citation
Research works printed but not linked: 1
- Politowski, 2020, Are game engines software frameworks? A three-perspective study, J. Syst. Softw., 171
Cited by
Trends in Gaming - Views of Industry Professionals and Game Scholars
Kultima, A. (2025) · Proceedings of the 20th International Conference on the Foundations of Digital Games · doi:10.1145/3723498.3723743 · Cited by 4
Generative AI in Game Development: A Qualitative Research Synthesis
Ternar, D.-A., Denisova, A., Cunha, J. M., Kultima, A., & Guckelsberger, C. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791206 · Cited by 6
Generative AI in Game Development: A Qualitative Research Synthesis
Ternar, A., Denisova, A., Cunha, J. M., Kultima, A., & Guckelsberger, C. (2025) · arXiv
“Studio Studio cases further show that friction in integrating outputs into pipelines can slow or stall adoption, even where initial enthusiasm is high ( C6.”
Metadata
- Authors
- Politowski, C., Petrillo, F., Ullmann, G. C., & Guéhéneuc, Y.-G.
- Affiliations
- Concordia University (CA)
- Universidade Regional do Noroeste do Estado do Rio Grande do Sul (BR)
- Corresponding author
- Politowski, Cristiano
- Venue
- Information and Software Technology
- Publisher
- Elsevier BV
- Published
- 2021-06
- Volume
- 134
- Pages
- 106538
- ISSN
- 0950-5849 (ISSN-L)
- OpenAlex
- W3083073468
- Cited by
- 54
- References
- 21
- Citations by year
- 2026: 10 · 2025: 14 · 2024: 6 · 2023: 11 · 2022: 6 · 2021: 6
- FWCI
- 6.55
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Information and Software Technology (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Espace ÉTS (ETS) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Canada Research Chairs
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Sociology and Political Science › Digital Games and Media
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- video game development, autopsy, game developers, marketing problems, team problems, management problems, production problems, business challenges, state of practice
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.