CoChat
Sign Up
Cited by 1,821Delphi Technique in ResearchReliability and Agreement in MeasurementMeta-analysis and systematic reviews

Delphi methodology in healthcare research: How to decide its appropriateness

Open access
Nasa, Prashant, Jain, Ravi, Juneja, Deven·2021·World Journal of Methodology
DOI 10.5662/wjm.v11.i4.116PMC PMC8299905
Open this paper in CoChat — free account

Abstract

The Delphi technique is a systematic process of forecasting using the collective opinion of panel members. The structured method of developing consensus among panel members using Delphi methodology has gained acceptance in diverse fields of medicine. The Delphi methods assumed a pivotal role in the last few decades to develop best practice guidance using collective intelligence where research is limited, ethically/logistically difficult or evidence is conflicting. However, the attempts to assess the quality standard of Delphi studies have reported significant variance, and details of the process followed are usually unclear. We recommend systematic quality tools for evaluation of Delphi methodology; identification of problem area of research, selection of panel, anonymity of panelists, controlled feedback, iterative Delphi rounds, consensus criteria, analysis of consensus, closing criteria, and stability of the results. Based on these nine qualitative evaluation points, we assessed the quality of Delphi studies in the medical field related to coronavirus disease 2019. There was inconsistency in reporting vital elements of Delphi methods such as identification of panel members, defining consensus, closing criteria for rounds, and presenting the results. We propose our evaluation points for researchers, medical journal editorial boards, and reviewers to evaluate the quality of the Delphi methods in healthcare research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.5662/wjm.v11.i4.116

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Delphi process is a structured method that uses multiple rounds of anonymous surveys with a panel of experts, where participants refine their opinions based on feedback from previous rounds until consensus emerges, designed to harness collective expert judgment while avoiding biases of face-to-face group dynamics.

    “The Delphi process is a structured method that uses multiple rounds of anonymous surveys with a panel of experts, where participants refine their opinions based on feedback from previous rounds until consensus emerges. It is designed to harness collective expert judgment while avoiding the biases of face‐to‐face group dynamics [24].”

    From Consensus‐Based Recommendations on Pulp Therapies in Primary and Permanent Teeth: <scp>IAPD</scp> Porto Forum (2026) · cites this paper as [24]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. [3]Goodman CM The Delphi technique: a critique J Adv Nurs 1987 12 729 734 3320139
  2. [4]McGrath BA Brenner MJ Warrillow SJ Pandian V Arora A Cameron TS Añon JM Hernández Martínez G Truog RD Block SD Lui GCY McDonald C Rassekh CH Atkins J Qiang L Vergez S Dulguerov P Zenk J Antonelli M Pelosi P Walsh BK Ward E Shang Y Gasparini S Donati A Singer M Openshaw PJM Tolley N Markel H Feller-Kopman DJ Tracheostomy in the COVID-19 era: global and multidisciplinary guidance Lancet Respir Med 2020 8 717 725 32422180 PMC7228735
  3. [5]Jünger S Payne S Brearley S Ploenes V Radbruch L Consensus building in palliative care: a Europe-wide delphi study on common understandings and conceptual differences J Pain Symptom Manage 2012 44 192 205 22704058
  4. [6]Vogel C Zwolinsky S Griffiths C Hobbs M Henderson E Wilkins E A Delphi study to build consensus on the definition and use of big data in obesity research Int J Obes (Lond) 2019 43 2573 2586 30655580 PMC6892733
  5. [7]Singer M Deutschman CS Seymour CW Shankar-Hari M Annane D Bauer M Bellomo R Bernard GR Chiche JD Coopersmith CM Hotchkiss RS Levy MM Marshall JC Martin GS Opal SM Rubenfeld GD van der Poll T Vincent JL Angus DC The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) JAMA 2016 315 801 810 26903338 PMC4968574
  6. [8]Han H Ahn DH Song J Hwang TY Roh S Development of mental health indicators in Korea Psychiatry Investig 2012 9 311 318 PMC3521105 23251193
  7. [9]van Hasselt FM Oud MJ Loonen AJ Practical recommendations for improvement of the physical health care of patients with severe mental illness Acta Psychiatr Scand 2015 131 387 396 25495118
  8. [11]Boulkedid R Abdoul H Loustau M Sibony O Alberti C Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review PLoS One 2011 6 e20476 21694759 PMC3111406
  9. [13]Green B Jones M Hughes D Williams A Applying the Delphi technique in a study of GPs' information requirements Health Soc Care Community 1999 7 198 205 11560634
  10. [14]Robba C Poole D Citerio G Taccone FS Rasulo FA Consensus on brain ultrasonography in critical care group Brain Ultrasonography Consensus on Skill Recommendations and Competence Levels Within the Critical Care Setting Neurocrit Care 2020 32 502 511 31264072
  11. [15]Niederberger M Spranger J Delphi Technique in Health Sciences: A Map Front Public Health 2020 8 457 33072683 PMC7536299
  12. [16]Santaguida P Dolovich L Oliver D Lamarche L Gilsing A Griffith LE Richardson J Mangin D Kastner M Raina P Protocol for a Delphi consensus exercise to identify a core set of criteria for selecting health related outcome measures (HROM) to be used in primary health care BMC Fam Pract 2018 19 152 30185172 PMC6123958
  13. [17]Dunn WN Public policy analysis: an introduction. Upper Saddle River: Pearson Prentice Hall, 2004
  14. [20]Grant S Booth M Khodyakov D Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies J Clin Epidemiol 2018 99 96 105 29559324
  15. [22]Holey EA Feeley JL Dixon J Whittaker VJ An exploration of the use of simple statistics to measure consensus and stability in Delphi studies BMC Med Res Methodol 2007 7 52 18045508 PMC2216026
  16. [24]Vitacca M Lazzeri M Guffanti E Frigerio P D'Abrosca F Gianola S Carone M Paneroni M Ceriana P Pasqua F Banfi P Gigliotti F Simonelli C Cirio S Rossi V Beccaluva CG Retucci M Santambrogio M Lanza A Gallo F Fumagalli A Mantero M Castellini G Calabrese M Castellana G Volpato E Ciriello M Garofano M Clini E Ambrosino N Arir Associazione Riabilitatori dell'Insufficienza Respiratoria Sip Società Italiana di Pneumologia Aifi Associazione Italiana Fisioterapisti And Sifir Società Italiana di Fisioterapia E Riabilitazione OBOAAIPO Italian suggestions for pulmonary rehabilitation in COVID-19 patients recovering from acute respiratory failure: results of a Delphi process Monaldi Arch Chest Dis 2020 90 32573175
  17. [25]Mikuls TR Johnson SR Fraenkel L Arasaratnam RJ Baden LR Bermas BL Chatham W Cohen S Costenbader K Gravallese EM Kalil AC Weinblatt ME Winthrop K Mudano AS Turner A Saag KG American College of Rheumatology Guidance for the Management of Rheumatic Disease in Adult Patients During the COVID-19 Pandemic: Version 1 Arthritis Rheumatol 2020 72 1241 1251 32349183
  18. [26]Greenhalgh T Thompson P Weiringa S Neves AL Husain L Dunlop M Rushforth A Nunan D de Lusignan S Delaney B What items should be included in an early warning score for remote assessment of suspected COVID-19? BMJ Open 2020 10 e042626 PMC7662139 33184088
  19. [27]Lamb CR Desai NR Angel L Chaddha U Sachdeva A Sethi S Bencheqroun H Mehta H Akulian J Argento AC Diaz-Mendoza J Musani A Murgu S Use of Tracheostomy During the COVID-19 Pandemic: American College of Chest Physicians/American Association for Bronchology and Interventional Pulmonology/Association of Interventional Pulmonology Program Directors Expert Panel Report Chest 2020 158 1499 1514 32512006 PMC7274948
  20. [28]Welsh Surgical Research Initiative (WSRI) Collaborative Surgery during the COVID-19 pandemic: operating room suggestions from an international Delphi process Br J Surg 2020 107 1450 1458 32395829 PMC7273074
  21. [29]Eibensteiner F Ritschl V Ariceta G Jankauskiene A Klaus G Paglialonga F Edefonti A Ranchin B Schmitt CP Shroff R Stefanidis CJ Walle JV Verrina E Vondrak K Zurowska A Stamm T Aufricht C European Pediatric Dialysis Working Group Rapid response in the COVID-19 pandemic: a Delphi study from the European Pediatric Dialysis Working Group Pediatr Nephrol 2020 35 1669 1678 32418146 PMC7230035
  22. [30]Bhandari P Subramaniam S Bourke MJ Alkandari A Chiu PWY Brown JF Keswani RN Bisschops R Hassan C Raju GS Muthusamy VR Sethi A May GR Albéniz E Bruno M Kaminski MF Alkhatry M Almadi M Ibrahim M Emura F Moura E Navarrete C Wulfson A Khor C Ponnudurai R Inoue H Saito Y Yahagi N Kashin S Nikonov E Yu H Maydeo AP Reddy DN Wallace MB Pochapin MB Rösch T Sharma P Repici A Recovery of endoscopy services in the era of COVID-19: recommendations from an international Delphi consensus Gut 2020 69 1915 1924 32816921
  23. [31]Guckenberger M Belka C Bezjak A Bradley J Daly ME DeRuysscher D Dziadziuszko R Faivre-Finn C Flentje M Gore E Higgins KA Iyengar P Kavanagh BD Kumar S Le Pechoux C Lievens Y Lindberg K McDonald F Ramella S Rengan R Ricardi U Rimner A Rodrigues GB Schild SE Senan S Simone CB 2nd Slotman BJ Stuschke M Videtic G Widder J Yom SS Palma D Practice recommendations for lung cancer radiotherapy during the COVID-19 pandemic: An ESTRO-ASTRO consensus statement Radiother Oncol 2020 146 223 229 32342863 PMC7252074
  24. [32]Aj B C B T A Harper E R Rl H Rj E Welsh Surgical Research Initiative (WSRI) Collaborative International surgical guidance for COVID-19: Validation using an international Delphi process - Cross-sectional study Int J Surg 2020 79 309 316 32531308 PMC7282765
  25. [33]Gelfand JM Armstrong AW Bell S Anesi GL Blauvelt A Calabrese C Dommasch ED Feldman SR Gladman D Kircik L Lebwohl M Lo Re V 3rd Martin G Merola JF Scher JU Schwartzman S Treat JR Van Voorhees AS Ellebrecht CT Fenner J Ocon A Syed MN Weinstein EJ Smith J Gondo G Heydon S Koons S Ritchlin CT National Psoriasis Foundation COVID-19 Task Force Guidance for Management of Psoriatic Disease During the Pandemic: Version 1 J Am Acad Dermatol 2020 83 1704 1716 32891785 PMC7471802
  26. [34]Allan M Mahawar K Blackwell S Catena F Chand M Dames N Goel R Graham YN Kothari SN Laidlaw L Mayol J Moug S Petersen RP Pryor AD Smart NJ Taylor M Toogood GJ Wexner SD Zevin B Wilson MS PRODUCE study COVID-19 research priorities in surgery (PRODUCE study): A modified Delphi process Br J Surg 2020 107 e538 e540 32871017 PMC7929143
  27. [35]Shanbehzadeh M Kazemi-Arpanahi H Mazhab-Jafari K Haghiri H Coronavirus disease 2019 (COVID-19) surveillance system: Development of COVID-19 minimum data set and interoperable reporting framework J Educ Health Promot 2020 9 203 33062736 PMC7530432
  28. [36]Bergman C Stall NM Haimowitz D Aronson L Lynn J Steinberg K Wasserman M Recommendations for Welcoming Back Nursing Home Visitors During the COVID-19 Pandemic: Results of a Delphi Panel J Am Med Dir Assoc 2020 21 1759 1766 33256956 PMC7539058
  29. [37]Daigle P Leung V Yin V Kalin-Hajdu E Nijhawan N Personal protective equipment (PPE) during the COVID-19 pandemic for oculofacial plastic and orbital surgery Orbit 2020 1 6 32552229
  30. [38]Sorbello M Rosenblatt W Hofmeyr R Greif R Urdaneta F Aerosol boxes and barrier enclosures for airway management in COVID-19 patients: a scoping review and narrative synthesis Br J Anaesth 2020 125 880 894 32977955 PMC7470712
  31. [39]Jheon S Ahmed AD Fang VW Jung W Khan AZ Lee JM Sihoe AD Thongcharoen P Tsuboi M Turna A Nakajima J Thoracic cancer surgery during the COVID-19 pandemic: a consensus statement from the Thoracic Domain of the Asian Society for Cardiovascular and Thoracic Surgery Asian Cardiovasc Thorac Ann 2020 28 322 329 32609557
  32. [40]Olmos-Gómez MDC Sex and Careers of University Students in Educational Practices as Factors of Individual Differences in Learning Environment and Psychological Factors during COVID-19 Int J Environ Res Public Health 2020 17 PMC7400233 32668769
  33. [41]Sawhney MS Bilal M Pohl H Kushnir VM Khashab MA Schulman AR Berzin TM Chahal P Muthusamy VR Varadarajulu S Banerjee S Ginsberg GG Raju GS Feuerstein JD Triaging advanced GI endoscopy procedures during the COVID-19 pandemic: consensus recommendations using the Delphi method Gastrointest Endosc 2020 92 535 542 32425235 PMC7229945
  34. [42]Sciubba DM Ehresman J Pennington Z Lubelski D Feghali J Bydon A Chou D Elder BD Elsamadicy AA Goodwin CR Goodwin ML Harrop J Klineberg EO Laufer I Lo SL Neuman BJ Passias PG Protopsaltis T Shin JH Theodore N Witham TF Benzel EC Scoring System to Triage Patients for Spine Surgery in the Setting of Limited Resources: Application to the Coronavirus Disease 2019 (COVID-19) Pandemic and Beyond World Neurosurg 2020 140 e373 e380 32479913 PMC7256646
  35. [43]Errett NA Howarth M Shoaf K Couture M Ramsey S Rosselli R Webb S Bennett A Miller A Developing an Environmental Health Sciences COVID-19 Research Agenda: Results from the NIEHS Disaster Research Response (DR2) Work Group's Modified Delphi Method Int J Environ Res Public Health 2020 17 PMC7557963 32961660
  36. [44]Arezzo A Francis N Mintz Y Adamina M Antoniou SA Bouvy N Copaescu C de Manzini N Di Lorenzo N Morales-Conde S Müller-Stich BP Nickel F Popa D Tait D Thomas C Nimmo S Paraskevis D Pietrabissa A EAES Group of Experts for Recovery Amid COVID-19 Pandemic EAES Recommendations for Recovery Plan in Minimally Invasive Surgery Amid COVID-19 Pandemic Surg Endosc 2021 35 1 17 33170335 PMC7653984
  37. [45]Dashash M Almasri B Takaleh E Halawah AA Sahyouni A Educational perspective for the identification of essential competencies required for approaching patients with COVID-19 East Mediterr Health J 2020 26 1011 1017 33047791
  38. [46]Ramalho R Adiukwu F Gashi Bytyçi D El Hayek S Gonzalez-Diaz JM Larnaout A Grandinetti P Nofal M Pereira-Sanchez V Pinto da Costa M Ransing R Teixeira ALS Shalbafan M Soler-Vidal J Syarif Z Orsolini L Telepsychiatry During the COVID-19 Pandemic: Development of a Protocol for Telemental Health Care Front Psychiatry 2020 11 552450 33173507 PMC7538900
  39. [47]Saldarriaga Rivera LM Fernández Ávila D Bautista Molano W Jaramillo Arroyave D Bautista Ramírez AJ Díaz Maldonado A Hernán Izquierdo J Jáuregui E Latorre Muñoz MC Restrepo JP Segura Charry JS Recommendations on the management of adult patients with rheumatic diseases in the context of SARS-CoV-2/COVID-19 infection. Colombian Association of Rheumatology Reumatol Clin 2020 16 437 446 32800760 PMC7386365
  40. [48]Tchouaket Nguemeleu E Boivin S Robins S Sia D Kilpatrick K Brousseau S Dubreuil B Larouche C Parisien N Development and validation of a time and motion guide to assess the costs of prevention and control interventions for nosocomial infections: A Delphi method among experts PLoS One 2020 15 e0242212 33180833 PMC7660509
  41. [49]Santana RF Silva MBD Marcos DADSR Rosa CDS Wetzel Junior W Delvalle R Nursing recommendations for facing dissemination of COVID-19 in Brazilian Nursing Homes Rev Bras Enferm 2020 73 e20200260 32965401
  42. [50]Tang Z Sun B Xu B A quick evaluation method of nosocomial infection risk for cancer hospitals during the COVID-19 pandemic J Cancer Res Clin Oncol 2020 146 1891 1892 32342200 PMC7184945
  43. [51]Jiménez-Rodríguez D Ruiz-Salvador D Rodríguez Salvador MDM Pérez-Heredia M Muñoz Ronda FJ Arrogante O Consensus on Criteria for Good Practices in Video Consultation: A Delphi Study Int J Environ Res Public Health 2020 17 PMC7432677 32727042
  44. [52]Reina Ortiz M Grijalva MJ Turell MJ Waters WF Montalvo AC Mathias D Sharma V Renoy CF Suits P Thomas SJ Leon R Biosafety at Home: How to Translate Biomedical Laboratory Safety Precautions for Everyday Use in the Context of COVID-19 Am J Trop Med Hyg 2020 103 838 840 32597388 PMC7410461
  45. [53]Douillet D Mahieu R Boiveau V Vandamme YM Armand A Morin F Savary D Dubée V Annweiler C Roy PM HOME-CoV expert group Outpatient management or hospitalization of patients with proven or suspected SARS-CoV-2 infection: the HOME-CoV rule Intern Emerg Med 2020 15 1525 1531 32888112 PMC7550768
  46. [54]Richez C Flipo RM Berenbaum F Cantagrel A Claudepierre P Debiais F Dieudé P Goupille P Roux C Schaeverbeke T Wendling D Pham T Thomas T Managing patients with rheumatic diseases during the COVID-19 pandemic: The French Society of Rheumatology answers to most frequently asked questions up to May 2020 Joint Bone Spine 2020 87 431 437 32473418 PMC7255274
  47. [55]Yılmaz Yalçınkaya E Karadağ Saygı E Özyemişci Taşkıran Ö Çapan N Kutlay Ş Sonel Tur B El Ö Ünlü Akyüz E Tekin S Ofluoğlu D Zİnnuroğlu M Akpınar P Özekli Mısırlıoğlu T Hüner B Nur H Çağlar S Sezgin M Tıkız C Öneş K İçağasıoğlu A Aydın R Consensus recommendations for botulinum toxin injections in the spasticity management of children with cerebral palsy during COVID-19 outbreak Turk J Med Sci 2021 51 385 392 33350298 PMC8203129
  48. [56]Tanasijevic AM Revette A Klepin HD Zeidan A Townsley D DiNardo CD Sebert M DeZern AE Stone RM Magnavita ES Chen R Sekeres MA Abel GA Consensus minimum hemoglobin level above which patients with myelodysplastic syndromes can safely forgo transfusions Leuk Lymphoma 2020 61 2900 2904 32667230
  49. [57]Alarcón MA Sanz-Sánchez I Shibli JA Treviño Santos A Caram S Lanis A Jiménez P Dueñas R Torres R Alvarado J Avendaño A Galindo R Umanzor V Shedden M Invernizzi C Yibrin C Collins J León R Contreras L Bueno L López-Pacheco A Málaga-Figueroa L Sanz M Delphi Project on the trends in Implant Dentistry in the COVID-19 era: Perspectives from Latin America Clin Oral Implants Res 2021 32 521 537 33595844 PMC8014337

Cited by

Sorted by citations

Authors
Nasa, P., Jain, R., & Juneja, D.
Affiliations
Max Super Speciality Hospital (IN)
ORCID
Venue
World Journal of Methodology
Publisher
Baishideng Publishing Group Inc.
Published
2021-07-20
Volume
11
Issue
4
Pages
116-129
ISSN
2222-0682 (ISSN-L)
DOI
10.5662/wjm.v11.i4.116
PMC
PMC8299905
OpenAlex
W3185797914
Cited by
1,821
References
2
Citations by year
2026: 552 · 2025: 552 · 2024: 463 · 2023: 201 · 2022: 49 · 2021: 4
FWCI
34.47
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • World Journal of Methodology (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
Delphi method, consensus building, study quality assessment, reporting quality, panel selection, consensus criteria, COVID-19 pandemic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review