CoChat
Sign Up
Cited by 189Delphi Technique in ResearchSurvey Methodology and NonresponseReliability and Agreement in Measurement

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response

Open access
Veugelers, Rebekka, Gaakeer, Menno I., Patka, Peter, Huijsman, Robbert·2020·BMC Medical Research Methodology
DOI 10.1186/s12874-020-01029-4PMC PMC7294633
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: A proper application of the Delphi technique is essential for obtaining valid research results. Medical researchers regularly use Delphi studies, but reports often lack detailed information on methodology and controlled feedback: in the medical literature, papers focusing on Delphi methodology issues are rare. Since the introduction of electronic surveys, details on response times remain scarce. We aim to bridge a number of gaps by providing a real world example covering methodological choices and response times in detail. METHODS: The objective of our e(lectronic)-Delphi study was to determine minimum standards for emergency departments (EDs) in the Netherlands. We opted for a two-part design with explicit decision rules. Part 1 focused on gathering and defining items; Part 2 addressed the main research question using an online survey tool. A two-person consensus rule was applied throughout: even after consensus on specific items was reached, panellists could reopen the discussion as long as at least two panellists argued similarly. Per round, the number of reminders sent and individual response times were noted. We also recorded the methodological considerations and evaluations made by the research team prior to as well as during the study. RESULTS: The study was performed in eight rounds and an additional confirmation round. Response rates were 100% in all rounds, resulting in 100% consensus in Part 1 and 96% consensus in Part 2. Our decision rules proved to be stable and easily applicable. Items with negative advice required more rounds before consensus was reached. Response delays were mostly due to late starts, but once panellists started, they nearly always finished the questionnaire on the same day. Reminders often yielded rapid responses. Intra-individual differences in response time were large, but quick responders remained quick. CONCLUSIONS: We advise those considering Delphi study to follow the CREDES guideline, consider a two-part design, invest in personal commitment of the panellists, set clear decision rules, use a consistent lay-out and send out your reminders early. Adopting this overall approach may assist researchers in future Delphi studies and may help to improve the quality of Delphi designs in terms of improved rigor and higher response rates.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12874-020-01029-4

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Components of online survey. The design was identical for each page of the survey and consisted of the following components
Fig. 1 Components of online survey. The design was identical for each page of the survey and consisted of the following componentsCC BY · PubMed Central
Fig. 2 Flowchart for Part 1: gathering and defining items. Preliminary consensus = consensus (agreement 70% or above) but retained based on other decision rule
Fig. 2 Flowchart for Part 1: gathering and defining items. Preliminary consensus = consensus (agreement 70% or above) but retained based on other decision ruleCC BY · PubMed Central
Fig. 3 Flowchart for Part 2. the main research topic: minimum ED standards in three domains (ED facilities, diagnostics and medical specialist availability)
Fig. 3 Flowchart for Part 2. the main research topic: minimum ED standards in three domains (ED facilities, diagnostics and medical specialist availability)CC BY · PubMed Central
Fig. 4 Deemed necessity and influence on number of rounds to reach consensus
Fig. 4 Deemed necessity and influence on number of rounds to reach consensusCC BY · PubMed Central
Fig. 5 Median time to complete survey per round. The vertical axis displays the response time in days; the horizontal axis lists the individual rounds. The figure represents the median, the interquartile range and two outliers in Part 1 Round 4
Fig. 5 Median time to complete survey per round. The vertical axis displays the response time in days; the horizontal axis lists the individual rounds. The figure represents the median, the interquartile range and two outliers in Part 1 Round 4CC BY · PubMed Central
Fig. 6 Response times and reminders. The horizontal axis presents the date; the vertical axis presents the cumulative returned surveys. The dotted lines represent one or more reminders that were sent
Fig. 6 Response times and reminders. The horizontal axis presents the date; the vertical axis presents the cumulative returned surveys. The dotted lines represent one or more reminders that were sentCC BY · PubMed Central
Fig. 7 Number of reminders sent to individual panellists in total and per round
Fig. 7 Number of reminders sent to individual panellists in total and per roundCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 28 references

Research works printed but not linked: 1

  1. Helmer O. Social technology. New York: Basic Books; 1966.

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Veugelers, R., Gaakeer, M. I., Patka, P., & Huijsman, R.
Affiliations
  • Admiraal De Ruyter Ziekenhuis (NL)
  • Erasmus MC (NL)
  • Erasmus University Rotterdam (NL)
Corresponding author
Veugelers, Rebekka
ORCID
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2020-06-15
Volume
20
Issue
1
Pages
156-156
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/s12874-020-01029-4
PMC
PMC7294633
OpenAlex
W3035611753
Cited by
189
References
28
Citations by year
2026: 28 · 2025: 35 · 2024: 38 · 2023: 42 · 2022: 20 · 2021: 23 · 2020: 2
FWCI
29.07
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion, other-oa)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Stichting Achmea Health (Z634)
Topics
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
Delphi method, electronic surveys, response time, decision rules
MeSH terms
Delphi Technique, Humans, Medicine, Netherlands, Physicians, Consensus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review