CoChat
Sign Up
Cited by 176Delphi Technique in ResearchMeta-analysis and systematic reviewsReliability and Agreement in Measurement

The DOI matches its Crossref record; 38 of the 39 references it cites were found in a registry or an index, and 1 are not checked yet.

Sample size in multistakeholder Delphi surveys: at what minimum sample size do replicability of results stabilize?

Open access
Manyara, Anthony Muchai, Purvis, Anthony, Ciani, Oriana, Collins, Gary S., Taylor, Rod S.·2024·Journal of Clinical Epidemiology
DOI 10.1016/j.jclinepi.2024.111485PMC PMC7617918
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND AND OBJECTIVE: The minimum sample size for multistakeholder Delphi surveys remains understudied. Drawing from three large international multistakeholder Delphi surveys, this study aimed to: 1) investigate the effect of increasing sample size on replicability of results; 2) assess whether the level of replicability of results differed with participant characteristics: for example, gender, age, and profession. METHODS: We used data from Delphi surveys to develop guidance for improved reporting of health-care intervention trials: SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and CONSORT (Consolidated Standards of Reporting Trials) extension for surrogate end points (n = 175, 22 items rated); CONSORT-SPI [CONSORT extension for Social and Psychological Interventions] (n = 333, 77 items rated); and core outcome set for burn care (n = 553, 88 items rated). Resampling with replacement was used to draw random subsamples from the participant data set in each of the three surveys. For each subsample, the median value of all rated survey items was calculated and compared to the medians from the full participant data set. The median number (and interquartile range) of medians replicated was used to calculate the percentage replicability (and variability). High replicability was defined as ≥80% and moderate as 60% and <80% RESULTS: The average median replicability (variability) as a percentage of total number of items rated from the three datasets was 81% (10%) at a sample size of 60. In one of the datasets (CONSORT-SPI), a ≥80% replicability was reached at a sample size of 80. On average, increasing the sample size from 80 to 160 increased the replicability of results by a further 3% and reduced variability by 1%. For subgroup analysis based on participant characteristics (eg, gender, age, professional role), using resampled samples of 20 to 100 showed that a sample size of 20 to 30 resulted to moderate replicability levels of 64% to 77%. CONCLUSION: We found that a minimum sample size of 60-80 participants in multistakeholder Delphi surveys provides a high level of replicability (≥80%) in the results. For Delphi studies limited to individual stakeholder groups (such as researchers, clinicians, patients), a sample size of 20 to 30 per group may be sufficient.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2024.111485

Cites

38 of 39 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure Boxplots showing replicability of results with increasing sample size in the three datasets: random resampled samples (20-500) on the x-axis and the median number of medians replicated (median replicability) from the full sample ratings (y-axis). Note for each panel, the maximum value of the y-axis is the number of items in each survey (22 for SPIRIT|CONSORT-Surrogate, 77 for CONSORT-SPI and 88 for COSB-i, respectively)).
Figure Boxplots showing replicability of results with increasing sample size in the three datasets: random resampled samples (20-500) on the x-axis and the median number of medians replicated (median replicability) from the full sample ratings (y-axis). Note for each panel, the maximum value of the y-axis is the number of items in each survey (22 for SPIRIT|CONSORT-Surrogate, 77 for CONSORT-SPI and 88 for COSB-i, respectively)).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

Sorted by citations

See all 176

Related

See all

Metadata

Authors
Manyara, A. M., Purvis, A., Ciani, O., Collins, G. S., & Taylor, R. S.
Affiliations
  • University of Bristol (GB)
  • University of Glasgow (GB)
  • Bocconi University (IT)
  • MRC/CSO Social and Public Health Sciences Unit (GB)
Corresponding author
Manyara, Anthony Muchai
ORCID
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2024-10
Volume
174
Pages
111485
ISSN
0895-4356 (ISSN-L)
DOI
10.1016/j.jclinepi.2024.111485
PMC
PMC7617918
OpenAlex
W4401027269
Cited by
163
References
36
Citations by year
2026: 93 · 2025: 67 · 2024: 3
FWCI
116.49
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Clinical Epidemiology (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
  • ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion, cc-by-sa)
  • PubMed Central (acceptedVersion, cc-by)
  • Archivio istituzionale della Ricerca - Bocconi (submittedVersion, cc-by-nc-nd)
  • Explore Bristol Research (submittedVersion, cc-by)
Funders
  • Cancer Research UK (C49297/A27294)
  • Economic and Social Research Council
  • National Institute for Health and Care Research
  • UKRI MRC (MR/V038400/1, 304823-01)
Topics
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
sample size, minimum sample size, resampling, participant characteristics, gender, age, professional roles, reporting guidelines, alcoholic beverages, core outcome set, burn treatment, surrogate endpoints, stakeholder consensus, subgroup analysis
MeSH terms
Adult, Delphi Technique, Female, Humans, Male, Middle Aged, Surveys and Questionnaires, Research Design, Reproducibility of Results, Sample Size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review