A Disruptive Research Playbook for Studying Disruptive Innovations
Open accessThe DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Abstract
As researchers today, we are witnessing a fundamental change in our technologically-enabled world due to the advent and diffusion of highly disruptive technologies such as generative Artificial Intelligence (AI), Augmented Reality (AR) and Virtual Reality (VR). In particular, software engineering has been profoundly affected by the transformative power of disruptive innovations for decades, with a significant impact of technical advancements on social dynamics due to its socio-technical nature. In this article, we reflect on the importance of formulating and addressing research problems in software engineering through a socio-technical lens, thus ensuring a holistic understanding of the complex phenomena in this field. We propose a research playbook with the aim of providing a guide to formulate compelling and socially relevant research questions and to identify the appropriate research strategies for empirical investigations, with an eye on the long-term implications of technologies or their use. We showcase how to apply the research playbook. Firstly, we show how it can be used retrospectively to reflect on a prior disruptive technology, Stack Overflow, and its impact on software development. Secondly, we show how it can be used to question the impact of two current disruptive technologies: AI and AR/VR. Finally, we introduce a specialized GPT model to support the researcher in framing future investigations. We conclude by discussing the broader implications of adopting the playbook for both researchers and practitioners in software engineering and beyond.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460
Disruptive technologies: catching the wave
(1995) · Long Range Planning · doi:10.1016/0024-6301(95)91075-1 · Cited by 2,402
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Selecting Empirical Methods for Software Engineering Research
Easterbrook, S., Singer, J., Storey, M.-A., & Damian, D. (2008) · Guide to Advanced Empirical Software Engineering · doi:10.1007/978-1-84800-044-5_11 · Cited by 1,167 · cited 3×
“We designed and developed a research playbook to guide and assist in the evaluation and prediction of how disruptive innovations may impact software engineering.”
Self-citationMeeting the Challenge of Disruptive Change
Christensen, C. M., & Overdorf, M. (2000) · Cited by 1,062
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
What are developers talking about? An analysis of topics and trends in Stack Overflow
Barua, A., Thomas, S. W., & Hassan, A. E. (2014) · Empirical Software Engineering · doi:10.1007/s10664-012-9231-y · Cited by 612
Design lessons from the fastest q&a site in the west
Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., & Hartmann, B. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979366 · Cited by 553
McLuhan, M. (2017) · communication theory · doi:10.4324/9781315080918-31 · Cited by 540
Research works printed but not linked: 14
- Muhammad Asaduzzaman, Ahmed Shah Mashiyat, Chanchal K. Roy, and Kevin A. Schneider. 2013. Answering questions about unanswered questions of Stack Overflow. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 97–100.
- Amiangshu Bosu, Christopher S. Corley, Dustin Heaton, Debarshi Chatterji, Jeffrey C. Carver, and Nicholas A Kraft. 2013. Building reputation in StackOverflow: An empirical investigation. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 89–92.
- Eduardo C. Campos, Martin Monperrus, and Marcelo A. Maia. 2016. Searching stack overflow for API-usage-related bug fixes using snippet-based queries. In Proceedings of the 26th Annual International Conference on Computer Science and Software Engineering (CASCON ’16). IBM Corp., USA, 232–242.
- Mark Chen Jakub Tworek Heewoo Jun Qijing Yuan Henryk P. de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Niru Joseph Greg Brockman Alexander Ray Rishita Puri Gabriel Krueger Mike Petrov Hala Khlaaf Girish Sastry Pamela Mishkin Ben Chan Scott Gray Nick Ryder Maxim Pavlov Alex Power Lukasz Kaiser Maximilian Bavarian Carolyn Winter Philippe Tillet Felipe P. Such David Cummings Matthias Plappert Fotios Chantzis Eric Barnes Ariel Herbert-Voss William H. Guss Alex Nichol Andrew Paino Nick Tezak Jerry Tang Igor Babuschkin Sujith Balaji Shivendra Jain Will Saunders Christopher Hesse Andrew N. Carr Jan Leike Joshua Achiam Vedant Misra Eri Morikawa Alec Radford Melody Knight Miles Brundage Matej Murati Katja Mayer Peter Welinder Brian McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https://arxiv.org/abs/2107.03374 · arXiv
- Adrian Hoff, Christoph Seidl, Mircea F Lungu, and Michele Lanza. 2023. Preparing software re-engineering via freehand sketches in virtual reality. In Proceedings of the 39th IEEE International Conference on Software Maintenance and Evolution. IEEE.
- Joseph E. McGrath. 1995. Methodology matters: Doing research in the behavioral and social sciences. In Readings in Human–Computer Interaction. Elsevier, 152–169.
- Marshall McLuhan. 1977. Laws of the Media. ETC: A Review of General Semantics 34, 2 (1977), 173–179.
- Sarah Meldrum Sherlock A. Licorish and Bastin Tony Roy Savarimuthu. 2020. Exploring research interest in stack overflow – A systematic mapping study and quality evaluation. arXiv:2010.12282. Retrieved from https://arxiv.org/abs/2010.12282 · arXiv
- OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https://arxiv.org/abs/2303.08774 · arXiv
- Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirere. 2023. The impact of AI on developer productivity: Evidence from GitHub. arXiv:2302.06590. Retrieved from http://arxiv.org/abs/2302.06590 · arXiv
- P. Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de França Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Martínez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarimäki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2020. ACM SIGSOFT empirical standards. arXiv:2010.03525. Retrieved from https://arxiv.org/abs/2010.03525 · arXiv
- Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de França Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Martínez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarimäki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2020. Empirical standards for software engineering research. arXiv:2010.03525. Retrieved from https://arxiv.org/abs/2010.03525 · arXiv
- Siddharth Subramanian and Reid Holmes. 2013. Making sense of online code snippets. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 85–88.
- Stefan Wagner and Günter Ruhe. 2018. A systematic review of productivity factors in software development. arXiv:1801.06475. Retrieved from https://arxiv.org/abs/1801.06475 · arXiv
Cited by
The Impact of Generative AI on Creativity in Software Development: A Research Agenda
Jackson, V., Vasilescu, B., Russo, D., Ralph, P., Prikladnicki, R., Izadi, M., D’Angelo, S., Inman, S., Andrade, A., & van der Hoek, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708523 · Cited by 23
Self-citationFrom Disruptions to Discussions: How GenAI Impacts Human Interactions in Software Development
Salomon, M., Koshchenko, E., Sergeyuk, A., Holmes, R., Murphy, G. C., & Fritz, T. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3655626 · Cited by 1
“We chose to apply this model as it has been previously used for such purposes for similar software engineering impact assessments (e.g., [42]–[44]).”
Metadata
- Authors
- Storey, M.-A., Russo, D., Novielli, N., Kobayashi, T., & Wang, D.
- Affiliations
- University of Victoria (CA)
- Aalborg University (DK)
- University of Bari Aldo Moro (IT)
- Tokyo Institute of Technology (JP)
- Tianjin University (CN)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2024-11-19
- Volume
- 33
- Issue
- 8
- Pages
- 1-29
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4400652308
- Cited by
- 15
- References
- 69
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 2
- FWCI
- 5.66
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
- VBN Forskningsportal (Aalborg Universitet) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- CINECA IRIS Institutional Research Information System (University of Bari Aldo Moro) (submittedVersion, cc-by)
- Tokyo Tech Research Repository (Tokyo Institute of Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- The Copenhagen Symposium on Human-Centered Software Engineering AI (G-2023-21020 and CF23-0208)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
- Keywords
- disruptive innovation, research question formulation, socio-technical perspective, generative AI, augmented reality, virtual reality, Stack Overflow, software engineering research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.