Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Methods for developing a core outcome or information set require involvement of key stakeholders to prioritise many items and achieve agreement as to the core set. The Delphi technique requires participants to rate the importance of items in sequential questionnaires (or rounds) with feedback provided in each subsequent round such that participants are able to consider the views of others. This study examines the impact of receiving feedback from different stakeholder groups, on the subsequent rating of items and the level of agreement between stakeholders. METHODS: Randomized controlled trials were nested within the development of three core sets each including a Delphi process with two rounds of questionnaires, completed by patients and health professionals. Participants rated items from 1 (not essential) to 9 (absolutely essential). For round 2, participants were randomized to receive feedback from their peer stakeholder group only (peer) or both stakeholder groups separately (multiple). Decisions as to which items to retain following each round were determined by pre-specified criteria. RESULTS: Whilst type of feedback did not impact on the percentage of items for which a participant subsequently changed their rating, or the magnitude of change, it did impact on items retained at the end of round 2. Each core set contained discordant items retained by one feedback group but not the other (3-22 % discordant items). Consensus between patients and professionals in items to retain was greater amongst those receiving multiple group feedback in each core set (65-82 % agreement for peer-only feedback versus 74-94 % for multiple feedback). In addition, differences in round 2 scores were smaller between stakeholder groups receiving multiple feedback than between those receiving peer group feedback only. Variability in item scores across stakeholders was reduced following any feedback but this reduction was consistently greater amongst the multiple feedback group. CONCLUSIONS: In the development of a core outcome or information set, providing feedback within Delphi questionnaires from all stakeholder groups separately may influence the final core set and improve consensus between the groups. Further work is needed to better understand how participants rate and re-rate items within a Delphi process. TRIAL REGISTRATION: The three randomized controlled trials reported here were each nested within the development of a core information or outcome set to investigate processes in core outcome and information set development. Outcomes were not health-related and therefore trial registration was not applicable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Principles of Biomedical Ethics
(2012) · Thieves of Virtue · doi:10.7551/mitpress/9079.003.0009 · Cited by 13,453
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061
Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.12.002 · Cited by 3,740
Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020476 · Cited by 2,045 · cited 1×
“Whilst most Delphi present feedback in the form of summary statistics [11], the majority of research has focussed on the impact of the presentation of rationale in addition to summary statistics [12], rather than comparing different presentations of quantitative data.”
BackgroundWillis, G. (2005) · doi:10.4135/9781412983655 · Cited by 1,778
Developing core outcome sets for clinical trials: issues to consider
Williamson, P. R., Altman, D. G., Blazeby, J. M., Clarke, M., Devane, D., Gargon, E., & Tugwell, P. (2012) · Trials · doi:10.1186/1745-6215-13-132 · Cited by 1,661
Self-citationGroup polarization: A critical review and meta-analysis.
Isenberg, D. J. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.50.6.1141 · Cited by 1,044
Sinha, I. P., Smyth, R. L., & Williamson, P. R. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000393 · Cited by 891 · cited 3×
“Patients and health professionals were the only stakeholders recruited as these were considered the key groups to inform the core sets [13].”
BackgroundSelf-citation
Cited by
The COMET Handbook: version 1.0
Williamson, P. R., Altman, D. G., Bagley, H., Barnes, K. L., Blazeby, J. M., Brookes, S. T., Clarke, M., Gargon, E., Gorst, S., Harman, N., et al. (2017) · Trials · doi:10.1186/s13063-017-1978-4 · Cited by 2,011
“proved agreement in items to retain at the end of the Delphi process [130].”
BackgroundMethodInfluentialSelf-citationDelphi Technique in Health Sciences: A Map
Niederberger, M., & Spranger, J. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00457 · Cited by 1,072
Core outcome set–STAndards for reporting: the COS-STAR statement
Kirkham, J. J., Gorst, S., Altman, D. G., Blazeby, J. M., Clarke, M., Devane, D., Gargon, E., Moher, D., Schmitt, J., Tugwell, P., et al. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002148 · Cited by 650
Self-citationConsensus in the delphi method: What makes a decision change?
Barrios, M., Guilera, G., Nuño, L., & Gómez-Benito, J. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2020.120484 · Cited by 377
Core Outcome Set-STAndardised Protocol Items: the COS-STAP Statement
for the COS-STAP Group, Kirkham, J. J., Gorst, S., Altman, D. G., Blazeby, J. M., Clarke, M., Tunis, S., & Williamson, P. R. (2019) · Trials · doi:10.1186/s13063-019-3230-x · Cited by 264
Self-citation
Metadata
- Authors
- Brookes, S. T., Macefield, R. C., Williamson, P. R., McNair, A. G., Potter, S., Blencowe, N. S., Strong, S., & Blazeby, J. M.
- Venue
- Trials
- Published
- 2016-08-17
- Volume
- 17
- Issue
- 1
- Pages
- 409-409
- ISSN
- 1745-6215 (ISSN-L)
- OpenAlex
- W2475534915
- Cited by
- 122
- References
- 8
- FWCI
- 26.68
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. https://app.cochat.ai/papers/10.1186/s13063-016-1479-x
- The paper
- Brookes, S. T., Macefield, R. C., Williamson, P. R., McNair, A. G., Potter, S., Blencowe, N. S., Strong, S., & Blazeby, J. M. (2016). Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. https://doi.org/10.1186/s13063-016-1479-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.