Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.

A Disruptive Research Playbook for Studying Disruptive Innovations

Open access
Storey, Margaret-Anne, Russo, Daniel, Novielli, Nicole, Kobayashi, Takashi, Wang, Dong · 2024 · ACM Transactions on Software Engineering and Methodology

Abstract

As researchers today, we are witnessing a fundamental change in our technologically-enabled world due to the advent and diffusion of highly disruptive technologies such as generative Artificial Intelligence (AI), Augmented Reality (AR) and Virtual Reality (VR). In particular, software engineering has been profoundly affected by the transformative power of disruptive innovations for decades, with a significant impact of technical advancements on social dynamics due to its socio-technical nature. In this article, we reflect on the importance of formulating and addressing research problems in software engineering through a socio-technical lens, thus ensuring a holistic understanding of the complex phenomena in this field. We propose a research playbook with the aim of providing a guide to formulate compelling and socially relevant research questions and to identify the appropriate research strategies for empirical investigations, with an eye on the long-term implications of technologies or their use. We showcase how to apply the research playbook. Firstly, we show how it can be used retrospectively to reflect on a prior disruptive technology, Stack Overflow, and its impact on software development. Secondly, we show how it can be used to question the impact of two current disruptive technologies: AI and AR/VR. Finally, we introduce a specialized GPT model to support the researcher in framing future investigations. We conclude by discussing the broader implications of adopting the playbook for both researchers and practitioners in software engineering and beyond.

Overview

Integrity signals

Registry

Crossref · 10.1145/3678172

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 68 references

Research works printed but not linked: 14

  1. Muhammad Asaduzzaman, Ahmed Shah Mashiyat, Chanchal K. Roy, and Kevin A. Schneider. 2013. Answering questions about unanswered questions of Stack Overflow. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 97–100.
  2. Amiangshu Bosu, Christopher S. Corley, Dustin Heaton, Debarshi Chatterji, Jeffrey C. Carver, and Nicholas A Kraft. 2013. Building reputation in StackOverflow: An empirical investigation. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 89–92.
  3. Eduardo C. Campos, Martin Monperrus, and Marcelo A. Maia. 2016. Searching stack overflow for API-usage-related bug fixes using snippet-based queries. In Proceedings of the 26th Annual International Conference on Computer Science and Software Engineering (CASCON ’16). IBM Corp., USA, 232–242.
  4. Mark Chen Jakub Tworek Heewoo Jun Qijing Yuan Henryk P. de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Niru Joseph Greg Brockman Alexander Ray Rishita Puri Gabriel Krueger Mike Petrov Hala Khlaaf Girish Sastry Pamela Mishkin Ben Chan Scott Gray Nick Ryder Maxim Pavlov Alex Power Lukasz Kaiser Maximilian Bavarian Carolyn Winter Philippe Tillet Felipe P. Such David Cummings Matthias Plappert Fotios Chantzis Eric Barnes Ariel Herbert-Voss William H. Guss Alex Nichol Andrew Paino Nick Tezak Jerry Tang Igor Babuschkin Sujith Balaji Shivendra Jain Will Saunders Christopher Hesse Andrew N. Carr Jan Leike Joshua Achiam Vedant Misra Eri Morikawa Alec Radford Melody Knight Miles Brundage Matej Murati Katja Mayer Peter Welinder Brian McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https://arxiv.org/abs/2107.03374 · arXiv
  5. Adrian Hoff, Christoph Seidl, Mircea F Lungu, and Michele Lanza. 2023. Preparing software re-engineering via freehand sketches in virtual reality. In Proceedings of the 39th IEEE International Conference on Software Maintenance and Evolution. IEEE.
  6. Joseph E. McGrath. 1995. Methodology matters: Doing research in the behavioral and social sciences. In Readings in Human–Computer Interaction. Elsevier, 152–169.
  7. Marshall McLuhan. 1977. Laws of the Media. ETC: A Review of General Semantics 34, 2 (1977), 173–179.
  8. Sarah Meldrum Sherlock A. Licorish and Bastin Tony Roy Savarimuthu. 2020. Exploring research interest in stack overflow – A systematic mapping study and quality evaluation. arXiv:2010.12282. Retrieved from https://arxiv.org/abs/2010.12282 · arXiv
  9. OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https://arxiv.org/abs/2303.08774 · arXiv
  10. Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirere. 2023. The impact of AI on developer productivity: Evidence from GitHub. arXiv:2302.06590. Retrieved from http://arxiv.org/abs/2302.06590 · arXiv
  11. P. Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de França Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Martínez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarimäki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2020. ACM SIGSOFT empirical standards. arXiv:2010.03525. Retrieved from https://arxiv.org/abs/2010.03525 · arXiv
  12. Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de França Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Martínez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarimäki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2020. Empirical standards for software engineering research. arXiv:2010.03525. Retrieved from https://arxiv.org/abs/2010.03525 · arXiv
  13. Siddharth Subramanian and Reid Holmes. 2013. Making sense of online code snippets. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 85–88.
  14. Stefan Wagner and Günter Ruhe. 2018. A systematic review of productivity factors in software development. arXiv:1801.06475. Retrieved from https://arxiv.org/abs/1801.06475 · arXiv

Cited by

See all 15

Metadata

Authors
Storey, M.-A., Russo, D., Novielli, N., Kobayashi, T., & Wang, D.
Affiliations
  • University of Victoria (CA)
  • Aalborg University (DK)
  • University of Bari Aldo Moro (IT)
  • Tokyo Institute of Technology (JP)
  • Tianjin University (CN)
Venue
ACM Transactions on Software Engineering and Methodology
Published
2024-11-19
Volume
33
Issue
8
Pages
1-29
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
OpenAlex
W4400652308
Cited by
15
References
68
Citations by year
2026: 4 · 2025: 9 · 2024: 2
FWCI
5.66
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
  • VBN Forskningsportal (Aalborg Universitet) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • CINECA IRIS Institutional Research Information System (University of Bari Aldo Moro) (submittedVersion, cc-by)
  • Tokyo Tech Research Repository (Tokyo Institute of Technology) (submittedVersion)
  • arXiv (Cornell University)
Funders
The Copenhagen Symposium on Human-Centered Software Engineering AI (G-2023-21020 and CF23-0208)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Social Sciences › Business, Management and Accounting › Management Information Systems › Information Systems Theories and Implementation
Keywords
disruptive innovation, research question formulation, socio-technical perspective, generative AI, augmented reality, virtual reality, Stack Overflow, software engineering research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Disruptive Research Playbook for Studying Disruptive Innovations. https://app.cochat.ai/papers/10.1145/3678172
The paper
Storey, M.-A., Russo, D., Novielli, N., Kobayashi, T., & Wang, D. (2024). A Disruptive Research Playbook for Studying Disruptive Innovations. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3678172

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.