ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi
Open accessPublished version of a preprint
The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Background: In biomedical research, it is often desirable to seek consensus among individuals who have differing perspectives and experience. This is important when evidence is emerging, inconsistent, limited, or absent. Even when research evidence is abundant, clinical recommendations, policy decisions, and priority-setting may still require agreement from multiple, sometimes ideologically opposed parties. Despite their prominence and influence on key decisions, consensus methods are often poorly reported. Our aim was to develop the first reporting guideline dedicated to and applicable to all consensus methods used in biomedical research regardless of the objective of the consensus process, called ACCORD (ACcurate COnsensus Reporting Document). Methods and findings: We followed methodology recommended by the EQUATOR Network for the development of reporting guidelines: a systematic review was followed by a Delphi process and meetings to finalize the ACCORD checklist. The preliminary checklist was drawn from the systematic review of existing literature on the quality of reporting of consensus methods and suggestions from the Steering Committee. A Delphi panel ( n = 72) was recruited with representation from 6 continents and a broad range of experience, including clinical, research, policy, and patient perspectives. The 3 rounds of the Delphi process were completed by 58, 54, and 51 panelists. The preliminary checklist of 56 items was refined to a final checklist of 35 items relating to the article title ( n = 1), introduction ( n = 3), methods ( n = 21), results ( n = 5), discussion ( n = 2), and other information ( n = 3). Conclusions: The ACCORD checklist is the first reporting guideline applicable to all consensus-based studies. It will support authors in writing accurate, detailed manuscripts, thereby improving the completeness and transparency of reporting and providing readers with clarity regarding the methods used to reach agreement. Furthermore, the checklist will make the rigor of the consensus methods used to guide the recommendations clear for readers. Reporting consensus studies with greater clarity and transparency may enhance trust in the recommendations made by consensus panels.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Rigor was supported by a priori definitions of objectives, panel composition and recruitment strategy, scoring approach and consensus rules, number of rounds, and stopping criteria; controlled feedback between rounds; investigator review of borderline items; and documentation of wording changes across iterations, consistent with Accurate Consensus Reporting Document recommendations for transparent”
From Principles for Responsible AI in Health Professions Education, Research, and Care: Health CARE-AI (Contextual, Accountable, Responsible, and Equitable Artificial Intelligence) Framework Delphi Consensus Study (2026) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The reporting of the consensus based methods used in the development of TRIPOD+AI followed the ACCORD (Accurate Consensus Reporting Document) recommendations.80”
From TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods (2024) · cites this paper as [80]
The cited source says what the claim attributes to it.
In the citing paper
“Reporting follows the Accurate Consensus Reporting Document guideline for consensus methods [15].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2_v1 · Cited by 8,090
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/v7gm2 · Cited by 5,177
Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.12.002 · Cited by 3,740
Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes
(1976) · The Journal of Applied Behavioral Science · doi:10.1177/002188637601200414 · Cited by 2,813 · cited 1×
“Delphi [5,6] • Anonymity • Iteration over multiple rounds of voting • Feedback after each round Nominal group technique [7] A face-to-face group interaction comprising 4 stages: • Solo idea generation • Round-robin feedback of ideas • Clarification of ideas through discussion • Voting to prioritize…”
BackgroundEvidence for a Collective Intelligence Factor in the Performance of Human Groups
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010) · Science · doi:10.1126/science.1193147 · Cited by 2,685
Jünger, S., Payne, S. A., Brine, J., Radbruch, L., & Brearley, S. G. (2017) · Palliative Medicine · doi:10.1177/0269216317690685 · Cited by 2,210 · cited 2×
“The Conducting and Reporting Delphi Studies (CREDES) guideline was designed to support the conduct and reporting of Delphi studies, with a focus on palliative care [26].”
Method(2001) · Defense Technical Information Center (DTIC) · Cited by 2,158
Consensus development methods, and their use in clinical guideline development.
Murphy, Black, Lamping, McKee, Sanderson, Askham, & Marteau (1998) · Health Technology Assessment · doi:10.3310/hta2030 · Cited by 2,051 · cited 2×
“Coordinating this process may be best achieved through the use of formal consensus methods [4] such as those described in Table 1.”
BackgroundMethodStaniszewska, S., Brett, J., Simera, I., Seers, K., Mockford, C., Goodlad, S., Altman, D. G., Moher, D., Barber, R., Denegri, S., et al. (2017) · BMJ · doi:10.1136/bmj.j3453 · Cited by 1,983
Consensus measurement in Delphi studies
von der Gracht, H. A. (2012) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2012.04.013 · Cited by 1,831
Staniszewska, S., Brett, J., Simera, I., Seers, K., Mockford, C., Goodlad, S., Altman, D. G., Moher, D., Barber, R., Denegri, S., et al. (2017) · Research Involvement and Engagement · doi:10.1186/s40900-017-0062-2 · Cited by 1,643
Guidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437
Zuberbier, T., Abdul Latiff, A. H., Abuzakouk, M., Aquilina, S., Asero, R., Baker, D., Ballmer‐Weber, B., Bangert, C., Ben‐Shoshan, M., Bernstein, J. A., et al. (2022) · Allergy · doi:10.1111/all.15090 · Cited by 1,263 · cited 2×
“…the evidence is uncertain, and incorporating multiple perspectives; creating clinical recommendations based on experience Diagnostic guidelines [11] Defining the markers, signs, and symptoms or thresholds that indicate a specific condition Disease classification [11] Classifying disease type…”
BackgroundJoint international consensus statement for ending stigma of obesity
Rubino, F., Puhl, R. M., Cummings, D. E., Eckel, R. H., Ryan, D. H., Mechanick, J. I., Nadglowski, J., Ramos Salas, X., Schauer, P. R., Twenefour, D., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0803-x · Cited by 1,152 · cited 1×
“…limited resources Developing core outcome sets [13] Defining the most important and clinically significant outcomes in research Formulating policy [14] Analyzing and interpreting evidence, its biases and strengths, to inform policies. types of consensus methods across the spectrum of health…”
BackgroundConsensus development methods, and their use in clinical guideline development.
Murphy, M. P., Black, N. A., Lamping, D. L., et al. (1998) · PubMed · Cited by 965
van Melick, N., van Cingel, R. E. H., Brooijmans, F., Neeter, C., van Tienen, T., Hullegie, W., & Nijhuis-van der Sanden, M. W. G. (2016) · British Journal of Sports Medicine · doi:10.1136/bjsports-2015-095898 · Cited by 918
The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines
Brouwers, M. C., Kerkvliet, K., Spithoff, K., & AGREE Next Steps Consortium (2016) · BMJ · doi:10.1136/bmj.i1152 · Cited by 785
Core outcome set–STAndards for reporting: the COS-STAR statement
Kirkham, J. J., Gorst, S., Altman, D. G., Blazeby, J. M., Clarke, M., Devane, D., Gargon, E., Moher, D., Schmitt, J., Tugwell, P., et al. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002148 · Cited by 650 · cited 2×
“COS-STAR includes only 3 items around the definition of consensus and a “description of how the consensus process was undertaken” [28].”
MethodTurner, L., Shamseer, L., Altman, D. G., Schulz, K. F., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-60 · Cited by 628 · cited 1×
“Reporting guidelines can enhance the reporting quality of research [20–22], and the absence of a universal reporting guideline for studies using consensus methods may contribute to their well-documented suboptimal reporting quality [5,19,23–25].”
BackgroundTheory and applications of the Delphi technique: A bibliography (1975–1994)
Gupta, U. G., & Clarke, R. E. (1996) · Technological Forecasting and Social Change · doi:10.1016/s0040-1625(96)00094-7 · Cited by 573 · cited 1×
“The need for optimal reporting of consensus methods has been documented for decades [19,24].”
BackgroundWoudenberg, F. (1991) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(91)90002-w · Cited by 520
Reporting guidelines for Delphi techniques in health sciences: A methodological review
Spranger, J., Homberg, A., Sonnberger, M., & Niederberger, M. (2022) · Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2022.04.025 · Cited by 262 · cited 3×
“Reporting guidelines can enhance the reporting quality of research [20–22], and the absence of a universal reporting guideline for studies using consensus methods may contribute to their well-documented suboptimal reporting quality [5,19,23–25].”
BackgroundMunblit, D., Nicholson, T., Akrami, A., Apfelbacher, C., Chen, J., De Groote, W., Diaz, J. V., Gorst, S. L., Harman, N., Kokorina, A., et al. (2022) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(22)00169-2 · Cited by 232
Boosting medical diagnostics by pooling independent judgments
Kurvers, R. H. J. M., Herzog, S. M., Hertwig, R., Krause, J., Carney, P. A., Bogart, A., Argenziano, G., Zalaudek, I., & Wolf, M. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1601827113 · Cited by 198
Dechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171
Will COVID-19 be evidence-based medicine’s nemesis?
Greenhalgh, T. (2020) · PLOS Medicine · doi:10.1371/journal.pmed.1003266 · Cited by 146 · cited 1×
“…or guidance are needed and there is uncertainty on the best course of action, such as when evidence is emergent, inconsistent, limited, or absent—not least in rapidly evolving fields such as pandemics [3]—the collation and dissemination of knowledge, experience, and expertise becomes critical.”
BackgroundBarnes, C., Boutron, I., Giraudeau, B., Porcher, R., Altman, D. G., & Ravaud, P. (2015) · BMC Medicine · doi:10.1186/s12916-015-0460-y · Cited by 135
Gattrell, W. T., Hungin, A. P., Price, A., Winchester, C. C., Tovey, D., Hughes, E. L., van Zuuren, E. J., Goldman, K., Logullo, P., Matheis, R., et al. (2022) · Research Integrity and Peer Review · doi:10.1186/s41073-022-00122-0 · Cited by 116
Self-citationEvidence-Based Medicine and Clinical Research: Both Are Needed, Neither Is Perfect
Szajewska, H. (2018) · Annals of Nutrition and Metabolism · doi:10.1159/000487375 · Cited by 83 · cited 1×
“Evidence-based medicine relies on (1) the best available evidence; (2) patients’ values, preferences, and knowledge; and (3) healthcare professionals’ experience and expertise [1,2].”
BackgroundDodd, S., Gorst, S. L., Young, A., Lucas, S. W., & Williamson, P. R. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.03.022 · Cited by 81 · cited 1×
“However, the experience represented by the Steering Committee recognized the value of patient participation in consensus recommendations, the importance of which is reported in the literature [38], and voted to include this item.”
BackgroundSadowski, D. C., Camilleri, M., Chey, W. D., Leontiadis, G. I., Marshall, J. K., Shaffer, E. A., Tse, F., & Walters, J. R. F. (2020) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2019.08.062 · Cited by 74 · cited 1×
“Clinical practice guidelines [10] Translating evidence into clinical recommendations, particularly where the evidence is uncertain, and incorporating multiple perspectives; creating clinical recommendations based on experience Diagnostic guidelines [11] Defining the markers, signs, and symptoms or…”
MethodClavisi, O., Bragge, P., Tavender, E. J., Turner, T., & Gruen, R. L. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.04.002 · Cited by 67 · cited 1×
“Priority-setting and core outcome identification are just 2 areas where patient participation in consensus exercises is becoming standard [35–37].”
BackgroundReporting guidelines used varying methodology to develop recommendations
Schlussel, M. M., Sharp, M. K., de Beyer, J. A., Kirtley, S., Logullo, P., Dhiman, P., MacCarthy, A., Koroleva, A., Speich, B., Bullock, G. S., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.03.018 · Cited by 59
Self-citationvan Zuuren, E. J., Logullo, P., Price, A. I., Fedorowicz, Z., Hughes, E. L., & Gattrell, W. T. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-065154 · Cited by 57
Self-citationIt is time for consensus on ‘consensus statements’
Blazey, P., Crossley, K. M., Ardern, C. L., van Middelkoop, M., Scott, A., & Khan, K. M. (2021) · British Journal of Sports Medicine · doi:10.1136/bjsports-2021-104578 · Cited by 49
Self-citationSadowski, D. C., Camilleri, M., Chey, W. D., Leontiadis, G. I., Marshall, J. K., Shaffer, E. A., Tse, F., & Walters, J. R. F. (2019) · Journal of the Canadian Association of Gastroenterology · doi:10.1093/jcag/gwz038 · Cited by 38
Clayton-Smith, M., Narayanan, H., Shelton, C., Bates, L., Brennan, F., Diedo, B., Donnellon, M., Dorey, J., Evans, B., Gower, J., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-066622 · Cited by 22 · cited 1×
“…or thresholds that indicate a specific condition Disease classification [11] Classifying disease type or severity Establishing research priorities [12] Defining and ranking priorities in the context of limited resources Developing core outcome sets [13] Defining the most important and clinically…”
BackgroundDijkstra, H. P., Auliffe, S. M., Ardern, C. L., Kemp, J. L., Mosler, A., Price, A. I., Blazey, P., Richards, D. P., Farooq, A., Serner, A., et al. (2023) · British Journal of Sports Medicine · doi:10.1136/bjsports-2022-106094 · Cited by 3
Self-citation
Research works printed but not linked: 4
- J. Surowiecki, 2004, The wisdom of crowds
- Open Science Framework, 2022, ACCORD registration with the Open Science Framework
- The EQUATOR Network, 2021, ACCORD registration at the EQUATOR Network
- The EQUATOR Network, 2021, EQUATOR Network Newsletter October 2021
Cited by
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Self-citationNiederberger, M., Schifano, J., Deckert, S., Hirt, J., Homberg, A., Köberich, S., Kuhn, R., Rommel, A., Sonnberger, M., & the DEWISS network (2024) · PLOS ONE · doi:10.1371/journal.pone.0304651 · Cited by 220
Manyara, A. M., Purvis, A., Ciani, O., Collins, G. S., & Taylor, R. S. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111485 · Cited by 163
“For specific topics or research areas, multistakeholder Delphi surveys (ie, recruiting various stakeholders) should be recruited for representativeness, for example, the appropriate range of stakeholder and user groups and geographies [8,13].”
Revisiting the Delphi technique - Research thinking and practice: A discussion paper
Hasson, F., Keeney, S., & McKenna, H. (2025) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2025.105119 · Cited by 183
Metadata
- Authors
- Gattrell, W. T., Logullo, P., van Zuuren, E. J., Price, A., Hughes, E. L., Blazey, P., Winchester, C. C., Tovey, D., Goldman, K., Hungin, A. P., & Harrison, N.
- Affiliations
- Bristol-Myers Squibb (United Kingdom) (GB)
- Stanford Medicine (US)
- University of British Columbia (CA)
- University of Oxford (GB)
- AbbVie (United States) (US)
- Newcastle University (GB)
- Corresponding author
- Gattrell, William T.
- ORCID
- Venue
- PLOS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2024-01-23
- Volume
- 21
- Issue
- 1
- Pages
- e1004326
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC10805282
- OpenAlex
- W4391135337
- Cited by
- 701
- References
- 37
- Citations by year
- 2026: 340 · 2025: 251 · 2024: 107
- FWCI
- 499.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- medRxiv (acceptedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Newcastle University ePrints (Newcastle Univesity) (cc-by)
- Funders
- Bristol-Myers Squibb
- Cancer Research UK (27294)
- Universiteit Leiden
- Ipsen
- Topics
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- consensus methods, reporting guidelines, consensus study, reporting quality, transparency, clinical guidelines, EQUATOR Network, Delphi method, consensus panel
- MeSH terms
- Humans, Consensus, Trust, Biomedical Research, Checklist, Policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
![Fig 1 A flow diagram to show the development of checklist items. *Potential items from relevant information beyond the predefined data extraction form [19]. †New item (T1) proposed at checklist review meeting.](https://figures.cochat.ai/paper-figures/9c3286131c794438a049f33a5ee65803/accord-accurate-consensus-reporting-document-a-reporting-guideline-for-consensus-figure-1.jpg)