How Delphi studies in the health sciences find consensus: a scoping review
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Background: Delphi studies are primarily used in the health sciences to find consensus. They inform clinical practice and influence structures, processes, and framework conditions of healthcare. The practical research—how Delphi studies are conducted—has seldom been discussed methodologically or documented systematically. The aim of this scoping review is to fill this research gap and to identify shortcomings in the methodological presentation in the literature. On the basis of the analysis, we derive recommendations for the quality-assured implementation of Delphi studies. Methods: Forming the basis of this scoping review are publications on consensus Delphi studies in the health sciences between January 1, 2018, and April 21, 2021, in the databases Scopus, MEDLINE via PubMed, CINAHL, and Epistemonikos. Included were publications in German and English containing the words “Delphi” in the title and “health” and “consensus” in the title or abstract. The practical research was analyzed for the qualitative content of the publications according to three deductive main categories, to which an influence on the result of Delphi studies can be imputed (expert panel, questionnaire design, process and feedback design). Results: A total of 287 consensus Delphi studies were included in the review, whereby 43% reported having carried out a modified Delphi. In most cases, heterogeneous expert groups from research, clinical practice, health economics, and health policy were surveyed. In about a quarter of the Delphi studies, affected parties, such as patients, were part of the expert panel. In the Delphi questionnaires it was most common for standardized Likert scales to be combined with open-ended questions. Which method was used to analyze the open-ended responses was not reported in 62% of the Delphi studies. Consensus is largely (81%) defined as percentage agreement. Conclusions: The results show considerable differences in how Delphi studies are carried out, making assessments and comparisons between them difficult. Sometimes an approach points to unintended effects, or biases in the individual judgments of the respondents and, thus, in the overall results of Delphi studies. For this reason, we extrapolate suggestions for how certain comparability and quality assurance can be achieved for Delphi studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061
Research guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 5×
“In a classic Delphi study, the experts’ judgments are typically collected anonymously using (online) questionnaires [11].”
InfluentialThe Delphi method as a research tool: an example, design considerations and applications
Okoli, C., & Pawlowski, S. D. (2004) · Information & Management · doi:10.1016/j.im.2003.11.002 · Cited by 4,771
The Psychology of Survey Response
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) · doi:10.1017/cbo9780511819322 · Cited by 4,734
Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.12.002 · Cited by 3,740
Jünger, S., Payne, S. A., Brine, J., Radbruch, L., & Brearley, S. G. (2017) · Palliative Medicine · doi:10.1177/0269216317690685 · Cited by 2,210 · cited 3×
“This diversity has been a consistent topic of discussion for decades in critical assessments of Delphi studies [15, 75, 76].”
Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020476 · Cited by 2,045 · cited 1×
“In a systematic review of Delphi studies that identify health-care quality indicators ( n = 80), Boulkedid et al. [13] found that more than half of the Delphi studies reported following a modified Delphi.”
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831
Delphi methodology in healthcare research: How to decide its appropriateness
Nasa, P., Jain, R., & Juneja, D. (2021) · World Journal of Methodology · doi:10.5662/wjm.v11.i4.116 · Cited by 1,821
Bishop, D. V. M., Snowling, M. J., Thompson, P. A., Greenhalgh, T., & and the CATALISE‐2 consortium (2017) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12721 · Cited by 1,641
Probabilistic mental models: a Brunswikian theory of confidence
Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991) · Psychological Review · doi:10.1037/0033-295x.98.4.506 · Cited by 1,506 · cited 2×
“…experts are often required to place the question in a larger context and generate transformation knowledge beyond the scope of their specialty [22, 31], e.g., in regard to the consequences of the judgment for affected groups [32], future generations [33] or other specialized areas in an…”
Delphi Technique in Health Sciences: A Map
Niederberger, M., & Spranger, J. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00457 · Cited by 1,072
Self-citationUsing the Delphi expert consensus method in mental health research
Jorm, A. F. (2015) · Australian & New Zealand Journal of Psychiatry · doi:10.1177/0004867415600891 · Cited by 765 · cited 6×
“1), and evaluate questions [3, 22].”
InfluentialBeiderbeck, D., Frevel, N., von der Gracht, H. A., Schmidt, S. L., & Schweitzer, V. M. (2021) · MethodsX · doi:10.1016/j.mex.2021.101401 · Cited by 642
De Meyer, D., Kottner, J., Beele, H., Schmitt, J., Lange, T., Van Hecke, A., Verhaeghe, S., & Beeckman, D. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.03.011 · Cited by 119
Boel, A., Navarro-Compán, V., Landewé, R., & van der Heijde, D. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.09.034 · Cited by 118 · cited 2×
“To reduce the probability that important perspectives are lost, Boel et al. [74] recommend, based on a methodological experiment, always inviting all of the experts to each Delphi round regardless of their participation or nonparticipation in the previous round.”
Patient and Public Involvement in research: A journey to co-production
Price, A., Clarke, M., Staniszewska, S., Chu, L., Tembo, D., Kirkpatrick, M., & Nelken, Y. (2022) · Patient Education and Counseling · doi:10.1016/j.pec.2021.07.021 · Cited by 90
Cited by
Using the Delphi Method to Establish Expert Consensus: A Practical Guide
Jorm, A. (2025) · Advancing Methods for Interdisciplinarity in the Social Sciences · doi:10.1007/978-981-96-8357-4 · Cited by 22
Metadata
- Authors
- Schifano, J., & Niederberger, M.
- Venue
- Systematic Reviews
- Published
- 2025-01-14
- Volume
- 14
- Issue
- 1
- Pages
- 14-14
- ISSN
- 2046-4053 (ISSN-L)
- OpenAlex
- W4406342287
- Cited by
- 163
- References
- 20
- FWCI
- 288.24
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.