A Disruptive Research Playbook for Studying Disruptive Innovations
Open accessThe DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Abstract
As researchers today, we are witnessing a fundamental change in our technologically-enabled world due to the advent and diffusion of highly disruptive technologies such as generative Artificial Intelligence (AI), Augmented Reality (AR) and Virtual Reality (VR). In particular, software engineering has been profoundly affected by the transformative power of disruptive innovations for decades, with a significant impact of technical advancements on social dynamics due to its socio-technical nature. In this article, we reflect on the importance of formulating and addressing research problems in software engineering through a socio-technical lens, thus ensuring a holistic understanding of the complex phenomena in this field. We propose a research playbook with the aim of providing a guide to formulate compelling and socially relevant research questions and to identify the appropriate research strategies for empirical investigations, with an eye on the long-term implications of technologies or their use. We showcase how to apply the research playbook. Firstly, we show how it can be used retrospectively to reflect on a prior disruptive technology, Stack Overflow, and its impact on software development. Secondly, we show how it can be used to question the impact of two current disruptive technologies: AI and AR/VR. Finally, we introduce a specialized GPT model to support the researcher in framing future investigations. We conclude by discussing the broader implications of adopting the playbook for both researchers and practitioners in software engineering and beyond.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460
Disruptive technologies: catching the wave
(1995) · Long Range Planning · doi:10.1016/0024-6301(95)91075-1 · Cited by 2,402
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Selecting Empirical Methods for Software Engineering Research
Easterbrook, S., Singer, J., Storey, M.-A., & Damian, D. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_11 · Cited by 1,167 · cited 3×
“We designed and developed a research playbook to guide and assist in the evaluation and prediction of how disruptive innovations may impact software engineering.”
Self-citationMeeting the Challenge of Disruptive Change
Christensen, C. M., & Overdorf, M. (2000) · Cited by 1,062
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
What are developers talking about? An analysis of topics and trends in Stack Overflow
Barua, A., Thomas, S. W., & Hassan, A. E. (2014) · Empirical Software Engineering · doi:10.1007/s10664-012-9231-y · Cited by 612
Design lessons from the fastest q&a site in the west
Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., & Hartmann, B. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979366 · Cited by 553
McLuhan, M. (2017) · communication theory · doi:10.4324/9781315080918-31 · Cited by 540
Disruptive Technologies: Catching the Wave
Bower, J. L., & Christensen, C. M. (1996) · Journal of Product Innovation Management · Cited by 426
How are Java software developers using the Eclipse IDE?
Murphy, G. C., Kersten, M., & Findlater, L. (2006) · IEEE Software · doi:10.1109/ms.2006.105 · Cited by 413
How do programmers ask and answer questions on the web? (NIER track)
Treude, C., Barzilay, O., & Storey, M.-A. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985907 · Cited by 405
Self-citationSteering user behavior with badges
Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488398 · Cited by 357
Generative AI for Software Practitioners
Ebert, C., & Louridas, P. (2023) · IEEE Software · doi:10.1109/ms.2023.3265877 · Cited by 293
Stack Overflow Considered Harmful? The Impact of Copy&Paste on Android Application Security
Fischer, F., Bottinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., & Fahl, S. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.31 · Cited by 280
StackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge
Vasilescu, B., Filkov, V., & Serebrenik, A. (2013) · 2013 International Conference on Social Computing · doi:10.1109/socialcom.2013.35 · Cited by 277
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
You Get Where You're Looking for: The Impact of Information Sources on Code Security
Acar, Y., Backes, M., Fahl, S., Kim, D., Mazurek, M. L., & Stransky, C. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.25 · Cited by 252
Collaborative virtual environments: An introductory review of issues and systems
Churchill, E. F., & Snowdon, D. (1998) · Virtual Reality · doi:10.1007/bf01409793 · Cited by 251
Building Theories in Software Engineering
Sjøberg, D. I. K., Dybå, T., Anda, B. C. D., & Hannay, J. E. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_12 · Cited by 244
Augmenting API documentation with insights from stack overflow
Treude, C., & Robillard, M. P. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884800 · Cited by 239
Mokyr, J. (1992) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195074772.001.0001 · Cited by 226
The who, what, how of software engineering research: a socio-technical framework
Storey, M.-A., Ernst, N., Williams, C., & Kalliamvakou, E. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09858-z · Cited by 175
Self-citationParadise unplugged: identifying barriers for female participation on stack overflow
Ford, D., Smith, J., Guo, P. J., & Parnin, C. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2950331 · Cited by 168
Are code examples on an online Q&A forum reliable?: a study of API misuse on stack overflow
Zhang, T., Upadhyaya, G., Reinhardt, A., Rajan, H., & Kim, M. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180260 · Cited by 167
Encouraging user behaviour with achievements: An empirical study
Grant, S., & Betts, B. (2013) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624007 · Cited by 149
Toxic Code Snippets on Stack Overflow
Ragkhitwetsagul, C., Krinke, J., Paixão, M. H. E., Bianco, G., & Oliveto, R. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2900307 · Cited by 136
How do developers utilize source code from stack overflow?
Wu, Y., Wang, S., Bezemer, C.-P., & Inoue, K. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9634-5 · Cited by 127
Applications of AI in classical software engineering
Barenkamp, M., Rebstadt, J., & Thomas, O. (2020) · AI Perspectives · doi:10.1186/s42467-020-00005-4 · Cited by 123
How software engineering research aligns with design science: a review
Engström, E., Storey, M.-A., Runeson, P., Höst, M., & Baldassarre, M. T. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09818-7 · Cited by 122
Self-citationIs GitHub copilot a substitute for human pair-programming?: an empirical study
Imai, S. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3522684 · Cited by 116
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2308.10620 · Cited by 114
On code reuse from StackOverflow: An exploratory study on Android apps
Abdalkareem, R., Shihab, E., & Rilling, J. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.04.005 · Cited by 104
Optical see-through augmented reality can induce severe motion sickness
Kaufeld, M., Mundt, M., Forst, S., & Hecht, H. (2022) · Displays · doi:10.1016/j.displa.2022.102283 · Cited by 103
Towards discovering the role of emotions in stack overflow
Novielli, N., Calefato, F., & Lanubile, F. (2014) · International Workshop on Social Software Engineering (SSE) · doi:10.1145/2661685.2661689 · Cited by 99
Self-citationSeahawk: Stack Overflow in the IDE
Ponzanelli, L., Bacchelli, A., & Lanza, M. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606701 · Cited by 95
Automated Query Reformulation for Efficient Search Based on Query Logs From Stack Overflow
Cao, K., Chen, C., Baltes, S., Treude, C., & Chen, X. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00116 · Cited by 82
Lukasik, S. J. (2010) · IEEE Annals of the History of Computing · doi:10.1109/mahc.2010.11 · Cited by 78
Among the Machines: Human-Bot Interaction on Social Q&A Websites
Murgia, A., Janssens, D., Demeyer, S., & Vasilescu, B. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2892311 · Cited by 74
Perry, T. S. (2015) · IEEE Spectrum · doi:10.1109/mspec.2016.7367470 · Cited by 62
Harnessing Stack Overflow for the IDE
Bacchelli, A., Ponzanelli, L., & Lanza, M. (2012) · 2012 Third International Workshop on Recommendation Systems for Software Engineering (RSSE) · doi:10.1109/rsse.2012.6233404 · Cited by 60
Understanding Stack Overflow Code Fragments
Treude, C., & Robillard, M. P. (2017) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2017.24 · Cited by 59
Chatterjee, P., Kong, M., & Pollock, L. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.110454 · Cited by 58
Ford, D., Lustig, K., Banks, J., & Parnin, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174182 · Cited by 55
Conversational Bot for Newcomers Onboarding to Open Source Projects
Dominic, J., Houser, J. A., Steinmacher, I., Ritter, C., & Rodeghero, P. (2020) · International Conference on Software Engineering · doi:10.1145/3387940.3391534 · Cited by 52
Understanding stack overflow code quality: A recommendation of caution
Meldrum, S., Licorish, S. A., Owen, C. A., & Savarimuthu, B. T. R. (2020) · Science of Computer Programming · doi:10.1016/j.scico.2020.102516 · Cited by 49
Hoppe, A. H., van de Camp, F., & Stiefelhagen, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432950 · Cited by 47
Yılmaz, M., O’Farrell, E., & Clarke, P. M. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2531 · Cited by 47
Mining Successful Answers in Stack Overflow
Calefato, F., Lanubile, F., Marasciulo, M. C., & Novielli, N. (2015) · 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories · doi:10.1109/msr.2015.56 · Cited by 45
Self-citationFacilitating Crowd Sourced Software Engineering via Stack Overflow
Barzilay, O., Treude, C., & Zagalsky, A. (2013) · Springer eBooks · doi:10.1007/978-1-4614-6596-6_15 · Cited by 44
Community evolution on Stack Overflow
Moutidis, I., & Williams, H. T. P. (2021) · PLoS ONE · doi:10.1371/journal.pone.0253010 · Cited by 43
Uncovering theories in software engineering
Stol, K.-J., & Fitzgerald, B. F. (2013) · doi:10.1109/gtse.2013.6613863 · Cited by 39
The Consequences of Generative AI for UGC and Online Community EngagementPreprint
Burtch, G., Lee, D., & Chen, Z. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4521754 · Cited by 38
Towards improving bug tracking systems with game mechanisms
Lotufo, R., Passos, L. M. S., & Czarnecki, K. (2012) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2012.6224293 · Cited by 35
Fernandes, F., & Werner, C. M. L. (2022) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres_a_00371 · Cited by 35
Reusing Code from StackOverflow: The Effect on Technical Debt
Δίγκας, Γ., Nikolaidis, N., Ampatzoglou, A., & Chatzigeorgiou, A. (2019) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2019.00022 · Cited by 33
Signals Matter: Understanding Popularity and Impact of Users on Stack Overflow
Merchant, A., Shah, D., Bhatia, G. S., Ghosh, A., & Kumaraguru, P. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313583 · Cited by 32
"A Bit of Code": How the Stack Overflow Community Creates Quality Postings
Squire, M., & Funkhouser, C. (2014) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2014.185 · Cited by 19
Social Interactions in Virtual Reality: What Cues Do People Use Most and How
Fang, J., Chang, V., Gao, G., & Wang, H. (2021) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3462204.3481772 · Cited by 18
Collaborative program comprehension via software visualization in extended reality
Krause-Glau, A., Hansen, M., & Hasselbring, W. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107007 · Cited by 17
Social science theories in software engineering research
Lorey, T., Ralph, P., & Felderer, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510076 · Cited by 14
On using Stack Overflow comment-edit pairs to recommend code maintenance changes
Tang, H., & Nadi, S. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09954-8 · Cited by 8
Challenges of Teaching Programming in StackOverflow Era
Pöial, J. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-68198-2_65 · Cited by 5
(2021) · IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase52884.2021.00002 · Cited by 5
Improved retrieval of programming solutions with code examples using a multi-featured score
Silva, R. F., Rahman, M. M., Dantas, C. E., Roy, C. K., Khomh, F., & Maia, M. D. A. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111063 · Cited by 4
Research works printed but not linked: 14
- Muhammad Asaduzzaman, Ahmed Shah Mashiyat, Chanchal K. Roy, and Kevin A. Schneider. 2013. Answering questions about unanswered questions of Stack Overflow. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 97–100.
- Amiangshu Bosu, Christopher S. Corley, Dustin Heaton, Debarshi Chatterji, Jeffrey C. Carver, and Nicholas A Kraft. 2013. Building reputation in StackOverflow: An empirical investigation. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 89–92.
- Eduardo C. Campos, Martin Monperrus, and Marcelo A. Maia. 2016. Searching stack overflow for API-usage-related bug fixes using snippet-based queries. In Proceedings of the 26th Annual International Conference on Computer Science and Software Engineering (CASCON ’16). IBM Corp., USA, 232–242.
- Mark Chen Jakub Tworek Heewoo Jun Qijing Yuan Henryk P. de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Niru Joseph Greg Brockman Alexander Ray Rishita Puri Gabriel Krueger Mike Petrov Hala Khlaaf Girish Sastry Pamela Mishkin Ben Chan Scott Gray Nick Ryder Maxim Pavlov Alex Power Lukasz Kaiser Maximilian Bavarian Carolyn Winter Philippe Tillet Felipe P. Such David Cummings Matthias Plappert Fotios Chantzis Eric Barnes Ariel Herbert-Voss William H. Guss Alex Nichol Andrew Paino Nick Tezak Jerry Tang Igor Babuschkin Sujith Balaji Shivendra Jain Will Saunders Christopher Hesse Andrew N. Carr Jan Leike Joshua Achiam Vedant Misra Eri Morikawa Alec Radford Melody Knight Miles Brundage Matej Murati Katja Mayer Peter Welinder Brian McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https://arxiv.org/abs/2107.03374 · arXiv
- Adrian Hoff, Christoph Seidl, Mircea F Lungu, and Michele Lanza. 2023. Preparing software re-engineering via freehand sketches in virtual reality. In Proceedings of the 39th IEEE International Conference on Software Maintenance and Evolution. IEEE.
- Joseph E. McGrath. 1995. Methodology matters: Doing research in the behavioral and social sciences. In Readings in Human–Computer Interaction. Elsevier, 152–169.
- Marshall McLuhan. 1977. Laws of the Media. ETC: A Review of General Semantics 34, 2 (1977), 173–179.
- Sarah Meldrum Sherlock A. Licorish and Bastin Tony Roy Savarimuthu. 2020. Exploring research interest in stack overflow – A systematic mapping study and quality evaluation. arXiv:2010.12282. Retrieved from https://arxiv.org/abs/2010.12282 · arXiv
- OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https://arxiv.org/abs/2303.08774 · arXiv
- Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirere. 2023. The impact of AI on developer productivity: Evidence from GitHub. arXiv:2302.06590. Retrieved from http://arxiv.org/abs/2302.06590 · arXiv
- P. Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de França Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Martínez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarimäki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2020. ACM SIGSOFT empirical standards. arXiv:2010.03525. Retrieved from https://arxiv.org/abs/2010.03525 · arXiv
- Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de França Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Martínez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarimäki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2020. Empirical standards for software engineering research. arXiv:2010.03525. Retrieved from https://arxiv.org/abs/2010.03525 · arXiv
- Siddharth Subramanian and Reid Holmes. 2013. Making sense of online code snippets. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 85–88.
- Stefan Wagner and Günter Ruhe. 2018. A systematic review of productivity factors in software development. arXiv:1801.06475. Retrieved from https://arxiv.org/abs/1801.06475 · arXiv
Cited by
The Impact of Generative AI on Creativity in Software Development: A Research Agenda
Jackson, V., Vasilescu, B., Russo, D., Ralph, P., Prikladnicki, R., Izadi, M., D’Angelo, S., Inman, S., Andrade, A., & van der Hoek, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708523 · Cited by 23
Self-citationFrom Disruptions to Discussions: How GenAI Impacts Human Interactions in Software Development
Salomon, M., Koshchenko, E., Sergeyuk, A., Holmes, R., Murphy, G. C., & Fritz, T. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655626 · Cited by 1
“We chose to apply this model as it has been previously used for such purposes for similar software engineering impact assessments (e.g., [42]–[44]).”
Metadata
- Authors
- Storey, M.-A., Russo, D., Novielli, N., Kobayashi, T., & Wang, D.
- Affiliations
- University of Victoria (CA)
- Aalborg University (DK)
- University of Bari Aldo Moro (IT)
- Tokyo Institute of Technology (JP)
- Tianjin University (CN)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2024-11-19
- Volume
- 33
- Issue
- 8
- Pages
- 1-29
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4400652308
- Cited by
- 15
- References
- 69
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 2
- FWCI
- 5.66
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
- VBN Forskningsportal (Aalborg Universitet) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- CINECA IRIS Institutional Research Information System (University of Bari Aldo Moro) (submittedVersion, cc-by)
- Tokyo Tech Research Repository (Tokyo Institute of Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- The Copenhagen Symposium on Human-Centered Software Engineering AI (G-2023-21020 and CF23-0208)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Keywords
- disruptive innovation, research question formulation, socio-technical perspective, generative AI, augmented reality, virtual reality, Stack Overflow, software engineering research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.