The Delphi Technique: Making Sense of Consensus
Open accessThe DOI matches its DataCite record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
The Delphi technique is a widely used and accepted method for gathering data from respondents within their domain of expertise. The technique is designed as a group communication process which aims to achieve a convergence of opinion on a specific real-world issue. The Delphi process has been used in various fields of study such as program planning, needs assessment, policy determination, and resource utilization to develop a full range of alternatives, explore or expose underlying assumptions, as well as correlate judgments on a topic spanning a wide range of disciplines. The Delphi technique is well suited as a method for consensus-building by using a series of questionnaires delivered using multiple iterations to collect data from a panel of selected subjects. Subject selection, time frames for conducting and completing a study, the possibility of low response rates, and unintentionally guiding feedback from the respondent group are areas which should be considered when designing and implementing a Delphi study. Accessed 68,465 times on https://pareonline.net from August 30, 2007 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“factors with a level of agreement ≥ 70% were extracted and included in the present review [19].”
From Healthcare professionals’ perspectives on artificial intelligence in patient care: a systematic review of hindering and facilitating factors on different levels (2025) · cites this paper as [19]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“To narrow the list into a usable checklist, we then used a three-step Delphi survey process [31].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [31]
The cited source says what the claim attributes to it.
In the citing paper
“The iteration characteristics of the Delphi technique have intrinsic limitations and might lead to groupthink, where participants might conform to dominant opinions of the group, which could affect objectivity.197”
From Definition and diagnostic criteria of clinical obesity (2025) · cites this paper as [197]
In the citing paper
“This anonymous and iterative group technique is designed to gather individual opinions from experts and transform these into a group consensus 6.”
From Diagnosis and treatment of chronic bacterial prostatitis and chronic prostatitis/chronic pelvic pain syndrome: a consensus guideline (2015) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Foundations of Behavioral Research
Stockton, R., & Kerlinger, F. N. (1974) · American Educational Research Journal · doi:10.2307/1162206 · Cited by 7,347
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061
Research guidelines for the Delphi survey technique
Hasson, F. (2000) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2000.01567.x · Cited by 5,951 · cited 1×
“As such, Ludwig (1994) specifically addresses subject motivation as the key to the successful implementation of a Delphi study and investigators need to play an active role in this area to help ensure as high a response rate as possible.”
The Delphi Method: An Experimental Study of Group Opinion
(1969) · RAND Corporation eBooks · doi:10.7249/rm5888 · Cited by 1,628
Gazing into the oracle : the Delphi method and its application to social policy and public health
Ziglio, E., & Adler, M. E. (1996) · Jessica Kingsley Publishers eBooks · Cited by 1,429
Group Techniques for Program Planning
de Ven, A. V., & Delbecq, A. L. (1975) · Cited by 907
Technological forecasting for decision making
Martino, J. P. (1972) · Cited by 813
Experimental Study of Channel Patterns
SCHUMM, S. A., & KHAN, H. R. (1972) · Geological Society of America Bulletin · doi:10.1130/0016-7606(1972)83[1755:esocp]2.0.co;2 · Cited by 776
An experimental study of group opinion
Dalkey, N. (1969) · Futures · doi:10.1016/s0016-3287(69)80025-x · Cited by 750
Miller, G. (2001) · Tourism Management · doi:10.1016/s0261-5177(00)00067-4 · Cited by 604
The Delphi method: Substance, context, a critique and an annotated bibliography
Pill, J. (1971) · Socio-Economic Planning Sciences · doi:10.1016/0038-0121(71)90041-3 · Cited by 579 · cited 2×
“Regarding the criteria used to guide the selection of Delphi subjects, individuals are considered eligible to be invited to participate in a Delphi study if they have somewhat related backgrounds and experiences concerning the target issue, are capable of contributing helpful inputs, and are willing to revise their in…”
Planning and Conducting Needs Assessments: A Practical Guide
Witkin, B. R., & Altschuld, J. W. (1995) · Cited by 548
On the Epistemology of the Inexact Sciences
Helmer, O., & Rescher, N. (1959) · Management Science · doi:10.1287/mnsc.6.1.25 · Cited by 528
The Modified Delphi Technique - A Rotational Modification
Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999) · Journal of Career and Technical Education · doi:10.21061/jcte.v15i2.702 · Cited by 520
STUDIES IN THE QUALITY OF LIFE; DELPHI AND DECISION-MAKING.
Dalkey, N. C. (1972) · Lexington Books · Cited by 495
Using Experts` Opinions Through Delphi Technique
Yousuf, M. İ. (2020) · doi:10.7275/rrph-t210 · Cited by 476
Instructional Technology: Past, Present, and Future
Anglin, G. J. (1995) · Medical Entomology and Zoology · Cited by 469
Predicting the Future: Have you considered using the Delphi Methodology?
Ludwig, B. (1997) · Journal of Extension · doi:10.66752/1077-5315.8000 · Cited by 345
The methodological worth of the Delphi forecasting technique
Hill, K. Q., & Fowles, J. (1975) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(75)90057-8 · Cited by 278
The Delphi Technique: A Case Study.
Cyphert, F. R., & Gant, W. L. (1971) · Phi Delta Kappan · Cited by 201
Assessing Needs in Educational and Social Programs
Witkin (1984) · Cited by 200
Delphi Technique: Expanding Applications.
Brooks, K. W. (1979) · The North Central Association quarterly · Cited by 166
Weaver, W. (1971) · Phi Delta Kappan · Cited by 158
Forecasting Technology for Planning Decisions
Jones, H., & Twiss, B. C. (1978) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-03134-4 · Cited by 135
Use of Delphi methods in higher education
Judd, R. C. (1972) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(72)90013-3 · Cited by 115
Experimental Assessment of Delphi Procedures with Group Value Judgments
Dalkey, N. C., & Rourke, D. (1971) · Cited by 90
Ludwig, B. G. (1994) · OhioLink ETD Center (Ohio Library and Information Network) · Cited by 86
Delivery Methodology of the Delphi: A Comparison of Two Approaches
Young, S., & Jamieson, L. M. (2001) · Journal of Park and Recreation Administration · Cited by 83
Essential assessment criteria for physical education teacher education programs: A Delphi study.
Jacobs, J. M. (1996) · doi:10.33915/etd.12913 · Cited by 56
Delphi method applied to tourism.
Taylor, R. E., Judd, L. L., Witt, S. F., & Moutinho, L. (1989) · Cited by 49
Methodological problems associated with the Use of the Delphi technique
Shields, T. J., Silcock, G. W. H., Donegan, H. A., & Bell, Y. A. (1987) · Fire Technology · doi:10.1007/bf01036934 · Cited by 44
Considerations in the Application of a Modified Scree Test for Delphi Survey Data
Altschuld, J. W., & Thomas, P. M. (1991) · Evaluation Review · doi:10.1177/0193841x9101500201 · Cited by 32
Forecasting through hierarchical Delphi
Oh, K. H. (1974) · OhioLink ETD Center (Ohio Library and Information Network) · Cited by 28
Predicting Future Trends in Adult Fitness Using the Delphi Approach
Murray, W. F., & Jarman, B. O. (1987) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1987.10605436 · Cited by 27
Forecasting risks in the tourism industry using the Delphi technique
Cunliffe, S. K. (2002) · Cited by 25
The utilization of expert opinion in decision‐making
Klee, A. J. (1972) · AIChE Journal · doi:10.1002/aic.690180604 · Cited by 23
Using the Delphi process to identify significant recreation research-based innovations.
Anderson, D. H. (1993) · Journal of Park and Recreation Administration · Cited by 22
Methodological problems associated with the use of the Delphi technique ? some comments
Marchant, E. W. (1988) · Fire Technology · doi:10.1007/bf01039641 · Cited by 21
Development of an instrument to evaluate intercollegiate athletic coaches : a modified Delphi study
Eckman, C. A. (1983) · Medical Entomology and Zoology · Cited by 16
Douglas, D. C. (1983) · OhioLink ETD Center (Ohio Library and Information Network) · Cited by 13
Research works printed but not linked: 7
- Adams, S. J. (2001). Projecting the next decade in safety management: A Delphi technique study. Professional Safety, 46 (10), 26-29.
- Green, P. J. (1982, March). The content of a college-level outdoor leadership course. Paper presented at the Conference of the Northwest District Association for the American Alliance for Health, Physical Education, Recreation, and Dance, Spokane, WA.
- Jones, C. G. (1975). A Delphi evaluation of agreement between organizations. In H. A. Linstone, & M. Turoff (Eds.). The Delphi method: Techniques and applications (pp. 160-167). Reading, MA: Addison-Wesley Publishing Company.
- Kaplan, L. M. (1971). The use of the Delphi method in organizational communication: A case study. Unpublished master's thesis, The Ohio State University, Columbus.
- Linstone, H. A., & Turoff, M. (1975). Introduction. In H. A. Linstone, & M. Turoff (Eds.). The Delphi method: Techniques and applications (pp. 3-12). Reading, MA: Addison-Wesley Publishing Company.
- Ludlow, J. (1975). Delphi inquiries and knowledge utilization. In H. A. Linstone, & M. Turoff (Eds.). The Delphi method: Techniques and applications (pp. 102-123). Reading, MA: Addison-Wesley Publishing Company.
- Scheibe, M., Skutsch, M., & Schofer, J. (1975). Experiments in Delphi methodology. In H. A. Linstone, & M. Turoff (Eds.). The Delphi method: Techniques and applications (pp. 262-287). Reading, MA: Addison-Wesley Publishing Company.
Cited by
James, P. A., Oparil, S., Carter, B. L., Cushman, W. C., Dennison-Himmelfarb, C., Handler, J., Lackland, D. T., LeFevre, M. L., MacKenzie, T. D., Ogedegbe, O., et al. (2014) · JAMA · doi:10.1001/jama.2013.284427 · Cited by 7,950
Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.12.002 · Cited by 3,740
“Quality score components Were criteria for participants reproducible? Yes 65 No 33 Was the number of rounds to be performed stated? Yes 81 No 17 Were criteria for dropping items clear? Yes or not applicable 59 No 39 Stopping criteria other than rounds specified? Yes 15 No 83 Quality score (items) 0 3 1 16 2 37 3 38 4…”
BackgroundMethodRethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
“To narrow the list into a usable checklist, we then used a three-step Delphi survey process [31].”
Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS)
Downes, M. J., Brennan, M. L., Williams, H. C., & Dean, R. S. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011458 · Cited by 2,376
Metadata
- Authors
- Hsu, C.-C., & Sandford, B. A.
- Venue
- University of Massachusetts Amherst
- Publisher
- University of Massachusetts Amherst
- Published
- 2007
- Volume
- 12
- Issue
- 10
- Pages
- 10
- OpenAlex
- W2583504027
- Cited by
- 4,500
- References
- 40
- Citations by year
- 2026: 160 · 2025: 361 · 2024: 381 · 2023: 406 · 2022: 373 · 2021: 381 · 2020: 383 · 2019: 375 · 2018: 358 · 2017: 287 · 2016: 298 · 2015: 240 · 2014: 176 · 2013: 129 · 2012: 91
- FWCI
- 516.01
- Type
- journal article
- Language
- en
- Open access
- Licence
- cc-by-nc-nd
- Locations
- ScholarWorks@UMassAmherst (University of Massachusetts Amherst) (submittedVersion, cc-by-nc-nd)
- University of Massachusetts (UMass) Amherst
- Practical assessment, research & evaluation
- Topics
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Management Science and Operations Research › Multi-Criteria Decision Making
- Social Sciences › Sociology and Political Science › Focus Groups and Qualitative Methods
- Keywords
- Delphi method, consensus building, expert opinion, participant selection, response rate, needs assessment, program planning, policy determination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.