The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Consensus in the delphi method: What makes a decision change?
Open accessAbstract
Abstract We examined whether giving feedback to participants in a Delphi study about the level of agreement across the expert panel had an effect on opinion change between rounds. We also considered the potential influence of participants’ sociodemographic and professional characteristics. Five three-round Delphi studies were conducted independently, in which a total of 628 mental health experts responded to all three rounds. In each study, participants had to decide, based on their experience, whether a series of categories were relevant. The percentage of group agreement (i.e., percentage of participants who considered each category as relevant) in round 2 was shown as feedback in round 3, and responses in rounds 2 and 3 were considered to analyze opinion change. Results showed that when the feedback given in round 3 indicated that ≥75% of experts considered a category to be relevant, there was a further shift in opinion towards the group opinion (i.e., the category then yielded even greater consensus), whereas if the feedback indicated
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
the delphi method: techniques and applications
Linstone, H. A., & Turoff, M. (1975) · Bibliothèque et Archives nationales du Québec (Québec government) · Cited by 5,231
Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.12.002 · Cited by 3,740
The Delphi technique: myths and realities
Powell, C. (2003) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2003.02537.x · Cited by 2,584
The Delphi Method: Techniques and Applications
J., J. E., Linstone, H. A., & Turoff, M. (1976) · Technometrics · doi:10.2307/1268751 · Cited by 2,228
Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020476 · Cited by 2,045
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831
The Delphi technique as a forecasting tool: issues and analysis
Rowe, G., & Wright, G. (1999) · International Journal of Forecasting · doi:10.1016/s0169-2070(99)00018-7 · Cited by 1,789
Consulting the oracle: ten lessons from using the Delphi technique in nursing research
Keeney, S., Hasson, F., & McKenna, H. (2006) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2006.03716.x · Cited by 1,511
Gazing into the oracle : the Delphi method and its application to social policy and public health
Ziglio, E., & Adler, M. E. (1996) · Jessica Kingsley Publishers eBooks · Cited by 1,429
Managing Delphi Surveys Using Nonparametric Statistical Techniques*
Schmidt, R. C. (1997) · Decision Sciences · doi:10.1111/j.1540-5915.1997.tb01330.x · Cited by 1,141
Sinha, I. P., Smyth, R. L., & Williamson, P. R. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000393 · Cited by 891
Delphi methodology in health research: how to do it?
Trevelyan, E. G., & Robinson, P. N. (2015) · European Journal of Integrative Medicine · doi:10.1016/j.eujim.2015.07.002 · Cited by 745
The Use of the Delphi and Other Consensus Group Methods in Medical Education Research: A Review
Humphrey-Murto, S., Varpio, L., Wood, T. J., Gonsalves, C., Ufholz, L.-A., Mascioli, K., Wang, C., & Foth, T. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001812 · Cited by 613
Jiang, R., Kleer, R., & Piller, F. T. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2017.01.006 · Cited by 480
Stability and agreement criteria for the termination of Delphi studies
Dajani, J. S., Sincoff, M. Z., & Talley, W. K. (1979) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(79)90007-6 · Cited by 354
Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions
Donohoe, H. M., & Needham, R. D. (2009) · International Journal of Tourism Research · doi:10.1002/jtr.709 · Cited by 335
Prinsen, C. A. C., Vohra, S., Rose, M. R., King-Jones, S., Ishaque, S., Bhaloo, Z., Adams, D., & Terwee, C. B. (2014) · Trials · doi:10.1186/1745-6215-15-247 · Cited by 332
Selb, M., Escorpizo, R. S., Kostanjsek, N. F. I., Stucki, G., Üstün, B., & Cieza, A. (2015) · PubMed · Cited by 286
van der Vaart, R., Witting, M., Riper, H., Kooistra, L., Bohlmeijer, E. T., & van Gemert-Pijnen, L. J. (2014) · BMC Psychiatry · doi:10.1186/s12888-014-0355-z · Cited by 273
Review of Delphi-based scenario studies: Quality and design considerations
Nowack, M., Endrikat, J., & Guenther, E. (2011) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2011.03.006 · Cited by 268
Using advice from multiple sources to revise and improve judgments
Yaniv, I., & Milyavsky, M. (2007) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2006.05.006 · Cited by 242
The use of Delphi and Nominal Group Technique in nursing education: A review
Foth, T., Efstathiou, N., Vanderspank-Wright, B., Ufholz, L.-A., Dütthorn, N., Zimansky, M., & Humphrey-Murto, S. (2016) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2016.04.015 · Cited by 233
International Classification of Functioning
Iwarsson, S. (2014) · Encyclopedia of Quality of Life and Well-Being Research · doi:10.1007/978-94-007-0753-5_1499 · Cited by 187
Kahan, B. C., Forbes, G., Ali, Y., Jairath, V., Bremner, S., Harhay, M. O., Hooper, R., Wright, N., Eldridge, S. M., & Leyrat, C. (2016) · Trials · doi:10.1186/s13063-016-1571-2 · Cited by 185
Lange, T., Kopkow, C., Lützner, J., Günther, K.-P., Gravius, S., Scharf, H.-P., Stöve, J., Wagner, R., & Schmitt, J. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0912-8 · Cited by 184
Research Areas in Distance Education: A Delphi Study
Zawacki-Richter, O. (2009) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v10i3.674 · Cited by 183
Rowe, G., Wright, G., & McColl, A. (2004) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2004.03.004 · Cited by 153
Quantifying the development of agreement among experts in Delphi studies
Meijering, J. V., Kampen, J. K., & Tobi, H. (2013) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2013.01.003 · Cited by 151
Birko, S., Dove, E. S., & Özdemir, V. (2015) · PLoS ONE · doi:10.1371/journal.pone.0135162 · Cited by 149
Desirability bias in foresight: Consequences for decision quality based on Delphi results
Ecken, P., Gnatzy, T., & von der Gracht, H. A. (2011) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2011.05.006 · Cited by 146
Harman, N., Bruce, I. A., Kirkham, J. J., Tierney, S., Callery, P., O'Brien, K. D., Bennett, A., Chorbachi, R., Hall, P. N., Harding‐Bell, A., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0129514 · Cited by 136
Hussler, C., Müller, P., & Rondé, P. (2011) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2011.07.008 · Cited by 135
BROOKES, S. T., Macefield, R. C., Williamson, P. R., McNair, A., Potter, S., Blencowe, N., Strong, S., & Blazeby, J. M. (2016) · Trials · doi:10.1186/s13063-016-1479-x · Cited by 131
Understanding cloud computing adoption issues: A Delphi study approach
El-Gazzar, R. F., Hustad, E., & Olsen, D. H. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.04.061 · Cited by 126
People consultation to construct the future: A Delphi application
Rodríguez, J. L., & Barrutia, J. (2010) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2010.04.001 · Cited by 122
The impact of task characteristics on the performance of structured group forecasting techniques
Rowe, G., & Wright, G. (1996) · International Journal of Forecasting · doi:10.1016/0169-2070(95)00658-3 · Cited by 118
Förster, B., & von der Gracht, H. A. (2013) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2013.07.012 · Cited by 116
The Delphi method in forecasting financial markets— An experimental study
Kauko, K., & Palmroos, P. (2014) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2013.09.007 · Cited by 97
A review of Delphi forecasting research in tourism
Lin, V. S., & Song, H. (2014) · Current Issues in Tourism · doi:10.1080/13683500.2014.967187 · Cited by 96
Zimmermann, M., Darkow, I., & von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.05.016 · Cited by 80
Lynch, T. S., Minkara, A. A., Aoki, S. K., Bedi, A., Bharam, S., Clohisy, J. C. F., Harris, J. D., Larson, C. M., Nepple, J. J., Nho, S. J., et al. (2019) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-18-00041 · Cited by 77
Policy experts' propensity to change their opinion along Delphi rounds
Makkonen, M., Hujala, T., & Uusivuori, J. (2016) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2016.05.020 · Cited by 71
Bolger, F., Stranieri, A., Wright, G., & Yearwood, J. L. (2011) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2011.06.002 · Cited by 65
Turnbull, A. E., Dinglas, V. D., Friedman, L. A., Chessare, C. M., Sepúlveda, K. A., Bingham, C. O., & Needham, D. M. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.06.007 · Cited by 65
Sobaih, A. E. E., Ritchie, C., & Jones, E. (2012) · International Journal of Contemporary Hospitality Management · doi:10.1108/09596111211247227 · Cited by 63
Meijering, J. V., & Tobi, H. (2015) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2015.11.008 · Cited by 53
Nuño, L., Barrios, M., Rojo, E., Gómez‐Benito, J., & Guilera, G. (2018) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2018.05.012 · Cited by 31
Self-citationThe application of unsupervised deep learning in predictive models using electronic health records
Wang, L., Tong, L., Davis, D. A., Arnold, T., & Esposito, T. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-00923-1 · Cited by 29
Consensus Development Methods, and their Use in Creating Clinical Guidelines
Black, N., Murphy, M., Lamping, D. L., et al. (2001) · SAGE Publications Ltd eBooks · doi:10.4135/9781848608344.n24 · Cited by 22
The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial
Meijering, J. V., & Tobi, H. (2018) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2017.11.010 · Cited by 16
Functioning in schizophrenia from the perspective of psychologists: A worldwide study
Nuño, L., Guilera, G., Coenen, M., Rojo, E., Gómez‐Benito, J., & Barrios, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0217936 · Cited by 16
Self-citationNuño, L., Guilera, G., Bell, M. D., Rojo, E., Gómez‐Benito, J., Calderón, C., & Barrios, M. (2021) · American Journal of Occupational Therapy · doi:10.5014/ajot.2021.041509 · Cited by 13
Self-citationNuño, L., Barrios, M., Moller, M. D., Calderón, C., Rojo, E., Gómez‐Benito, J., & Guilera, G. (2019) · International Journal of Mental Health Nursing · doi:10.1111/inm.12586 · Cited by 12
Self-citationChild oral health from the professional perspective – a global ICF ‐ CY survey
Faulks, D., Molina, G. F., Eschevins, C., & Dougall, A. (2015) · International Journal of Paediatric Dentistry · doi:10.1111/ipd.12195 · Cited by 11
The Perspective of Social Workers on Functioning for Individuals With Schizophrenia: A Delphi Study
Nuño, L., Guilera, G., Solomon, P., Rojo, E., Gómez‐Benito, J., & Barrios, M. (2021) · Journal of the Society for Social Work and Research · doi:10.1086/715893 · Cited by 9
Self-citation
Cited by
Beiderbeck, D., Frevel, N., von der Gracht, H. A., Schmidt, S. L., & Schweitzer, V. M. (2021) · MethodsX · doi:10.1016/j.mex.2021.101401 · Cited by 642
Using the Delphi Method to Establish Expert Consensus: A Practical Guide
Jorm, A. (2025) · Advancing Methods for Interdisciplinarity in the Social Sciences · doi:10.1007/978-981-96-8357-4 · Cited by 22
Arribas, C. M., Arcos, R., & Gertrudix, M. (2025) · Cogent Social Sciences · doi:10.1080/23311886.2025.2501759 · Cited by 9
Metadata
- Authors
- Barrios, M., Guilera, G., Nuño, L., & Gómez-Benito, J.
- Venue
- Technological Forecasting and Social Change
- Publisher
- Elsevier BV
- Published
- 2021-02
- Volume
- 163
- Pages
- 120484
- ISSN
- 0040-1625 (ISSN-L)
- OpenAlex
- W3107815694
- Cited by
- 377
- References
- 56
- FWCI
- 28.42
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.