Lessons Learned from a Two-Round Delphi-based Scenario Study
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
The Delphi technique is a suitable methodology for structuring group communication to answer current and prospective research questions within several rounds. The method is used in many disciplines and characterized by anonymity, iteration, controlled feedback, and statistical "group response" (Rowe & Wright, 2001). This technical paper presents practical details and lessons learned from a two-round Delphi-based scenario study in which projections (Delphi statements, questions or hypotheses) were developed with findings from expert interviews and an expert workshop. This Delphi study provides answers to future-related questions for which other research methods are inappropriate. This is depicted as a five-step process, making it easy to follow and replicable, for example to help first-time Delphi-method researchers. In doing so, the authors aim to provide the community with valuable technical insights and guidance for studies applying the Delphi technique both to prospective questions and in other research settings.•Conducting a Delphi study can be a slow process with respect to receiving feedback from the panelists. Planning an appropriate period for distributing the questionnaire may produce a higher return rate. A sufficient time buffer should be incorporated into project planning.•Projections that create dissent among the panelists may provide valuable results.•Data analytics, software programs and online social networks can support researchers, save time and resources, and provide further insights in the process of conducting a Delphi study.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The third international consensus definitions for sepsis and septic shock (Sepsis-3).
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar-Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. (2016) · JAMA · doi:10.1001/jama.2016.0287 · Cited by 28,250
Stang, A. (2010) · European Journal of Epidemiology · doi:10.1007/s10654-010-9491-z · Cited by 17,396
The Neuropsychiatric Inventory: Comprehensive assessment of psychopathology in dementia
Cummings, J. L., Mega, M., Gray, K., Rosenberg-Thompson, S., Carusi, D. A., & Gornbein, J. (1994) · Neurology · doi:10.1212/wnl.44.12.2308 · Cited by 7,568
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061
The Delphi method as a research tool: an example, design considerations and applications
Okoli, C., & Pawlowski, S. D. (2004) · Information & Management · doi:10.1016/j.im.2003.11.002 · Cited by 4,771
The Delphi Technique: Making Sense of Consensus
Hsu, C.-C., & Sandford, B. A. (2007) · University of Massachusetts Amherst · doi:10.7275/pdz9-th90 · Cited by 4,500
Deardorff, D. K. (2006) · Journal of Studies in International Education · doi:10.1177/1028315306287002 · Cited by 3,489
NbClust : An R Package for Determining the Relevant Number of Clusters in a Data Set
Charrad, M., Ghazzali, N., Boiteau, V., & Niknafs, A. (2014) · Journal of Statistical Software · doi:10.18637/jss.v061.i06 · Cited by 3,396 · cited 2×
“’s [10] majority rule may provide support.”
MethodZhang, W., Moskowitz, R. W., Nuki, G., Abramson, S., Altman, R. D., Arden, N., Bierma-Zeinstra, S., Brandt, K. D., Croft, P., Doherty, M., et al. (2008) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2007.12.013 · Cited by 2,796
Jünger, S., Payne, S. A., Brine, J., Radbruch, L., & Brearley, S. G. (2017) · Palliative Medicine · doi:10.1177/0269216317690685 · Cited by 2,210
Brooks, N., Neil Adger, W., & Mick Kelly, P. (2005) · Global Environmental Change · doi:10.1016/j.gloenvcha.2004.12.006 · Cited by 2,049
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831
Current validity of the Delphi method in social sciences
Landeta, J. (2006) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2005.09.002 · Cited by 1,353
Belton, I., MacDonald, A., Wright, G., & Hamlin, I. (2019) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2019.07.002 · Cited by 474 · cited 1×
“Most “lessons learned” papers were published around 2011, and fewer have been published in the last five years [5,19,21] .”
BackgroundAnalyzing the past to prepare for the future
(2002) · MIS Quarterly · doi:10.5555/2017160.2017162 · Cited by 179
A clustering approach for topic filtering within systematic literature reviews
Weißer, T., Saßmannshausen, T., Ohrndorf, D., Burggräf, P., & Wagner, J. (2020) · MethodsX · doi:10.1016/j.mex.2020.100831 · Cited by 64
Cited by
Use of Delphi in health sciences research: A narrative review
Shang, Z. (2023) · Medicine · doi:10.1097/md.0000000000032829 · Cited by 657
Beiderbeck, D., Frevel, N., von der Gracht, H. A., Schmidt, S. L., & Schweitzer, V. M. (2021) · MethodsX · doi:10.1016/j.mex.2021.101401 · Cited by 642
Wehrle, M., Birkel, H., von der Gracht, H. A., & Hartmann, E. (2022) · Journal of Purchasing and Supply Management · doi:10.1016/j.pursup.2021.100732 · Cited by 49
Defining the goals of Product Passports by circular product strategies
Reich, R. H., Ayan, J., Alaerts, L., & Van Acker, K. (2023) · Procedia CIRP · doi:10.1016/j.procir.2023.02.044 · Cited by 30
Oo, C. T. L., Wider, W., Pang, N. T. P., Koh, E. B. Y., Vasanthi, R. K., Thet, K. Z. Z., Ramalho, R., Özdemir, B. N., & Mahboob, K. (2026) · International Journal of Qualitative Studies on Health and Well-being · doi:10.1080/17482631.2026.2621802 · Cited by 5
“The interpretation of the findings relies heavily on the expertise and judgement of the participating professionals (Schmalz et al., 2021).”
Metadata
- Authors
- Schmalz, U., Spinler, S., & Ringbeck, J.
- Venue
- MethodsX
- Publisher
- Elsevier BV
- Published
- 2021
- Volume
- 8
- Pages
- 101179
- ISSN
- 2215-0161 (ISSN-L)
- OpenAlex
- W3112421677
- Cited by
- 136
- References
- 17
- FWCI
- 17.85
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.