CoChat
Sign Up
Cited by 343Protein Structure and DynamicsMachine Learning in BioinformaticsComputational Drug Discovery Methods

The DOI matches its Crossref record; 56 of the 58 references it cites were found in a registry or an index.

Critical assessment of protein intrinsic disorder prediction

Open access
Necci, Marco, Piovesan, Damiano, CAID Predictors, Hoque, Md Tamjidul, Walsh, Ian, Iqbal, Sumaiya, Vendruscolo, Michele, Sormanni, Pietro, Wang, Chen, Raimondi, Daniele, Sharma, Ronesh, Zhou, Yaoqi, Litfin, Thomas, Galzitskaya, Oxana Valerianovna, Lobanov, Michail Yu., Vranken, Wim, Wallner, Björn, Mirabello, Claudio, Malhis, Nawar, Dosztányi, Zsuzsanna, Erdős, Gábor, Mészáros, Bálint, Gao, Jianzhao, Wang, Kui, Hu, Gang, Wu, Zhonghua, Sharma, Alok, Hanson, Jack, Paliwal, Kuldip, Callebaut, Isabelle, Bitard-Feildel, Tristan, Orlando, Gabriele, Peng, Zhenling, Xu, Jinbo, Wang, Sheng, Jones, David T., Cozzetto, Domenico, Meng, Fanchi, Yan, Jing, Gsponer, Jörg, Cheng, Jianlin, Wu, Tianqi, Kurgan, Lukasz, DisProt Curators, Promponas, Vasilis J., Tamana, Stella, Marino-Buslje, Cristina, Martínez-Pérez, Elizabeth, Chasapi, Anastasia, Ouzounis, Christos, Dunker, A. Keith, Kajava, Andrey V., Leclercq, Jeremy Y., Aykac-Fas, Burcu, Lambrughi, Matteo, Maiani, Emiliano, Papaleo, Elena, Chemes, Lucia Beatriz, Álvarez, Lucía, González-Foutel, Nicolás S., Iglesias, Valentin, Pujols, Jordi, Ventura, Salvador, Palopoli, Nicolás, Benítez, Guillermo Ignacio, Parisi, Gustavo, Bassot, Claudio, Elofsson, Arne, Govindarajan, Sudha, Lamb, John, Salvatore, Marco, Hatos, András, Monzon, Alexander Miguel, Bevilacqua, Martina, Mičetić, Ivan, Minervini, Giovanni, Paladin, Lisanna, Quaglia, Federica, Leonardi, Emanuela, Davey, Norman, Horvath, Tamas, Kovacs, Orsolya Panna, Murvai, Nikoletta, Pancsa, Rita, Schad, Eva, Szabo, Beata, Tantos, Agnes, Macedo-Ribeiro, Sandra, Manso, Jose Antonio, Pereira, Pedro José Barbosa, Davidović, Radoslav, Veljkovic, Nevena, Hajdu-Soltész, Borbála, Pajkos, Mátyás, Szaniszló, Tamás, Guharoy, Mainak, Lazar, Tamas, Macossay-Castillo, Mauricio, Tompa, Peter, Tosatto, Silvio C. E.·2021·Nature Methods
DOI 10.1038/s41592-021-01117-3PMC PMC8105172

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract Intrinsically disordered proteins, defying the traditional protein structure–function paradigm, are a challenge to study experimentally. Because a large part of our knowledge rests on computational predictions, it is crucial that their accuracy is high. The Critical Assessment of protein Intrinsic Disorder prediction (CAID) experiment was established as a community-based blind test to determine the state of the art in prediction of intrinsically disordered regions and the subset of residues involved in binding. A total of 43 methods were evaluated on a dataset of 646 proteins from DisProt. The best methods use deep learning techniques and notably outperform physicochemical methods. The top disorder predictor has F max = 0.483 on the full dataset and F max = 0.792 following filtering out of bona fide structured regions. Disordered binding regions remain hard to predict, with F max = 0.231. Interestingly, computing times among methods can vary by up to four orders of magnitude.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41592-021-01117-3

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 CAID dashboard. a, CAID timeline: phases of CAID from June 2018 to the present. The initial results were presented and discussed at the conferences Intelligent Systems for Molecular Biology (ISMB) and CASP. b, CAID process: iterative process of the CAID experiment in four phases. (1) Annotation: any process that produces unpublished annotation of IDR coordinates; in this edition, annotation refers to the DisProt round of annotation. (2) Prediction: annotations are used to build references with which we test predictors. (3) Evaluation: predictions are evaluated. (4) Report: a report of the evaluation is produced and published in peer-reviewed journals and on a web page that allows the reader to browse the evaluation of all CAID editions. c, Residue classification strategy for the DisProt and DisProt-PDB references. d, Number of residues for each class in different references. e, Number of proteins for each set of annotations that they contain. f, Number of proteins in each taxon.
Fig. 1 CAID dashboard. a, CAID timeline: phases of CAID from June 2018 to the present. The initial results were presented and discussed at the conferences Intelligent Systems for Molecular Biology (ISMB) and CASP. b, CAID process: iterative process of the CAID experiment in four phases. (1) Annotation: any process that produces unpublished annotation of IDR coordinates; in this edition, annotation refers to the DisProt round of annotation. (2) Prediction: annotations are used to build references with which we test predictors. (3) Evaluation: predictions are evaluated. (4) Report: a report of the evaluation is produced and published in peer-reviewed journals and on a web page that allows the reader to browse the evaluation of all CAID editions. c, Residue classification strategy for the DisProt and DisProt-PDB references. d, Number of residues for each class in different references. e, Number of proteins for each set of annotations that they contain. f, Number of proteins in each taxon.CC BY · PubMed Central
Fig. 2 Prediction success and CPU times for the ten top-ranking disorder predictors in the DisProt dataset. a, The reference used (DisProt, n = 646 proteins) in the analysis and how it was obtained. b–g, Performance of predictors expressed as maximum F1-score across all thresholds (Fmax) (b) and AUC (e) for the ten top-ranking methods (light gray) and baselines (white), and distribution of execution time per target (c,f) using the DisProt dataset. b,e, The horizontal line indicates, respectively, Fmax and AUC of the best baseline. d,g, Precision–recall (d) and ROC curves (g) of the ten top-ranking methods and baselines using the DisProt dataset, with level curves of F1-score and balanced accuracy, respectively. F, Fmax; C, coverage; A, AUC. c,f, Boxplots are defined as follows: the middle value of the dataset is the median (Q2/50th percentile) and box boundaries are the first quartile (Q1/25th percentile) and third quartile (Q3/75th percentile), respectively; maximum is Q3 + 1.5 × (Q3 – Q1) and minimum is Q1 – 1.5 × (Q3 – Q1). Outliers are hidden for clarity. c,f, Magenta dots indicate that the entire distribution of execution times is <1 s. Q1–Q3, first to third quartiles. TPR, true positive rate; FPR, false positive rate.
Fig. 2 Prediction success and CPU times for the ten top-ranking disorder predictors in the DisProt dataset. a, The reference used (DisProt, n = 646 proteins) in the analysis and how it was obtained. b–g, Performance of predictors expressed as maximum F1-score across all thresholds (Fmax) (b) and AUC (e) for the ten top-ranking methods (light gray) and baselines (white), and distribution of execution time per target (c,f) using the DisProt dataset. b,e, The horizontal line indicates, respectively, Fmax and AUC of the best baseline. d,g, Precision–recall (d) and ROC curves (g) of the ten top-ranking methods and baselines using the DisProt dataset, with level curves of F1-score and balanced accuracy, respectively. F, Fmax; C, coverage; A, AUC. c,f, Boxplots are defined as follows: the middle value of the dataset is the median (Q2/50th percentile) and box boundaries are the first quartile (Q1/25th percentile) and third quartile (Q3/75th percentile), respectively; maximum is Q3 + 1.5 × (Q3 – Q1) and minimum is Q1 – 1.5 × (Q3 – Q1). Outliers are hidden for clarity. c,f, Magenta dots indicate that the entire distribution of execution times is <1 s. Q1–Q3, first to third quartiles. TPR, true positive rate; FPR, false positive rate.CC BY · PubMed Central
Fig. 3 Prediction success and CPU times for the ten top-ranking disorder predictors in the DisProt-PDB dataset. a, The reference used (DisProt-PDB, n = 646 proteins) in the analysis and how it was obtained. b–g, Performance of predictors expressed as maximum F1-score across all thresholds (Fmax) (b) and AUC (e) for the ten top-ranking methods (light gray) and baselines (white), and distribution of execution time per target (c,f) using the DisProt-PDB dataset. b,e, The horizontal line indicates, respectively, Fmax and AUC of the best baseline. d,g, Precision–recall (d) and ROC curves (g) of the ten top-ranking methods and baselines using the DisProt-PDB dataset, with level curves of F1-score and balanced accuracy, respectively. c,f, boxplots are defined as follows: the middle value of the dataset is the median (Q2/50th percentile) and box boundaries are the first quartile (Q1/25th percentile) and third quartile (Q3/75th percentile), respectively; maximum is Q3 + 1.5 × (Q3 – Q1) and minimum is Q1 – 1.5 × (Q3 – Q1). Outliers are hidden for clarity. c,f, Magenta dots indicate that the entire distribution of execution times is <1 s.
Fig. 3 Prediction success and CPU times for the ten top-ranking disorder predictors in the DisProt-PDB dataset. a, The reference used (DisProt-PDB, n = 646 proteins) in the analysis and how it was obtained. b–g, Performance of predictors expressed as maximum F1-score across all thresholds (Fmax) (b) and AUC (e) for the ten top-ranking methods (light gray) and baselines (white), and distribution of execution time per target (c,f) using the DisProt-PDB dataset. b,e, The horizontal line indicates, respectively, Fmax and AUC of the best baseline. d,g, Precision–recall (d) and ROC curves (g) of the ten top-ranking methods and baselines using the DisProt-PDB dataset, with level curves of F1-score and balanced accuracy, respectively. c,f, boxplots are defined as follows: the middle value of the dataset is the median (Q2/50th percentile) and box boundaries are the first quartile (Q1/25th percentile) and third quartile (Q3/75th percentile), respectively; maximum is Q3 + 1.5 × (Q3 – Q1) and minimum is Q1 – 1.5 × (Q3 – Q1). Outliers are hidden for clarity. c,f, Magenta dots indicate that the entire distribution of execution times is <1 s.CC BY · PubMed Central
Fig. 4 Prediction success and CPU times for the ten top-ranking binding predictors in the DisProt-binding dataset. a, The reference used (DisProt-binding, n = 646 proteins) in the analysis and how it was obtained. b–g, Performance of predictors expressed as maximum F1-score across all thresholds (Fmax) (b) and AUC (e) for the ten top-ranking methods (light gray) and baselines (white), and distribution of execution time per target (c,f) using the DisProt-binding dataset. b,e, The horizontal line indicates, respectively, Fmax and AUC of the best baseline. d,g, Precision–recall (d) and ROC curves (g) of the ten top-ranking methods and baselines using the DisProt-binding dataset, with level curves of F1-score and balanced accuracy, respectively. c,f, boxplots are defined as follows: the middle value of the dataset is the median (Q2/50th percentile) and box boundaries are the first quartile (Q1/25th percentile) and third quartile (Q3/75th percentile), respectively; maximum is Q3 + 1.5 × (Q3 – Q1) and minimum is Q1 – 1.5 × (Q3 – Q1). Outliers are hidden for clarity. c,f, Magenta dots indicate that the entire distribution of execution times is <1 s.
Fig. 4 Prediction success and CPU times for the ten top-ranking binding predictors in the DisProt-binding dataset. a, The reference used (DisProt-binding, n = 646 proteins) in the analysis and how it was obtained. b–g, Performance of predictors expressed as maximum F1-score across all thresholds (Fmax) (b) and AUC (e) for the ten top-ranking methods (light gray) and baselines (white), and distribution of execution time per target (c,f) using the DisProt-binding dataset. b,e, The horizontal line indicates, respectively, Fmax and AUC of the best baseline. d,g, Precision–recall (d) and ROC curves (g) of the ten top-ranking methods and baselines using the DisProt-binding dataset, with level curves of F1-score and balanced accuracy, respectively. c,f, boxplots are defined as follows: the middle value of the dataset is the median (Q2/50th percentile) and box boundaries are the first quartile (Q1/25th percentile) and third quartile (Q3/75th percentile), respectively; maximum is Q3 + 1.5 × (Q3 – Q1) and minimum is Q1 – 1.5 × (Q3 – Q1). Outliers are hidden for clarity. c,f, Magenta dots indicate that the entire distribution of execution times is <1 s.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 58 references

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Necci, M., Piovesan, D., Predictors, C., Hoque, M. T., Walsh, I., Iqbal, S., Vendruscolo, M., Sormanni, P., Wang, C., Raimondi, D., Sharma, R., Zhou, Y., Litfin, T., Galzitskaya, O. V., Lobanov, M. Y., Vranken, W., Wallner, B., Mirabello, C., Malhis, N., Dosztányi, Z., Erdős, G., Mészáros, B., Gao, J., Wang, K., Hu, G., Wu, Z., Sharma, A., Hanson, J., Paliwal, K., Callebaut, I., Bitard-Feildel, T., Orlando, G., Peng, Z., Xu, J., Wang, S., Jones, D. T., Cozzetto, D., Meng, F., Yan, J., Gsponer, J., Cheng, J., Wu, T., Kurgan, L., Curators, D., Promponas, V. J., Tamana, S., Marino-Buslje, C., Martínez-Pérez, E., Chasapi, A., Ouzounis, C., Dunker, A. K., Kajava, A. V., Leclercq, J. Y., Aykac-Fas, B., Lambrughi, M., Maiani, E., Papaleo, E., Chemes, L. B., Álvarez, L., González-Foutel, N. S., Iglesias, V., Pujols, J., Ventura, S., Palopoli, N., Benítez, G. I., Parisi, G., Bassot, C., Elofsson, A., Govindarajan, S., Lamb, J., Salvatore, M., Hatos, A., Monzon, A. M., Bevilacqua, M., Mičetić, I., Minervini, G., Paladin, L., Quaglia, F., Leonardi, E., Davey, N., Horvath, T., Kovacs, O. P., Murvai, N., Pancsa, R., Schad, E., Szabo, B., Tantos, A., Macedo-Ribeiro, S., Manso, J. A., Pereira, P. J. B., Davidović, R., Veljkovic, N., Hajdu-Soltész, B., Pajkos, M., Szaniszló, T., Guharoy, M., Lazar, T., Macossay-Castillo, M., Tompa, P., & Tosatto, S. C. E.
Affiliations
  • University of Padua (IT)
  • LSU New Orleans (US)
  • Agency for Science, Technology and Research (SG)
  • Broad Institute (US)
  • University of Cambridge (GB)
  • Columbia University (US)
  • Fiji National University (FJ)
  • Griffith University (AU)
  • Russian Academy of Sciences (RU)
  • Linköping University (SE)
  • Eötvös Loránd University (HU)
  • Nankai University (CN)
  • Sorbonne Université (FR)
  • Tianjin University (CN)
  • Toyota Technological Institute at Chicago (US)
  • University College London (GB)
  • University of Alberta (CA)
  • University of British Columbia (CA)
  • University of Missouri (US)
  • Virginia Commonwealth University (US)
  • University of Cyprus (CY)
  • Fundación Instituto Leloir (AR)
  • Centre for Research and Technology Hellas (GR)
  • Université de Montpellier (FR)
  • Danish Cancer Society (DK)
  • Consejo Nacional de Investigaciones Científicas y Técnicas (AR)
  • Universitat Autònoma de Barcelona (ES)
  • National University of Quilmes (AR)
  • Stockholm University (SE)
  • University of Copenhagen (DK)
  • Institute of Cancer Research (GB)
  • Institute of Molecular Life Sciences (HU)
  • Universidade do Porto (PT)
  • Vrije Universiteit Brussel (BE)
Corresponding author
Tosatto, Silvio C. E.
ORCID
Venue
Nature Methods
Publisher
Springer Science and Business Media LLC
Published
2021-04-19
Volume
18
Issue
5
Pages
472-481
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
DOI
10.1038/s41592-021-01117-3
PMC
PMC8105172
OpenAlex
W3156051371
Cited by
343
References
58
Citations by year
2026: 35 · 2025: 64 · 2024: 68 · 2023: 72 · 2022: 69 · 2021: 32
FWCI
18.99
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HAL (submittedVersion, other-oa)
  • Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, cc-by)
  • Griffith Research Online (submittedVersion, cc-by)
  • CONICET Digital (CONICET) (submittedVersion, cc-by)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • ELTE Digital Institutional Repository (EDIT) (Eötvös Loránd University) (submittedVersion, other-oa)
  • UCL Discovery (University College London) (submittedVersion)
  • IUScholarWorks (Indiana University) (submittedVersion, cc-by)
  • VinaR (Institute of Nuclear Sciences "Vinča") (submittedVersion, cc-by)
  • VUBIR (Vrije Universiteit Brussel) (submittedVersion)
  • Padua Research Archive (University of Padova) (submittedVersion, cc-by)
  • Apollo
  • Apollo (cc-by)
Funders
  • EC | Horizon 2020 Framework Programme (778247)
  • Ministero dell’Istruzione, dell’Università e della Ricerca (2017483NH8)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
Keywords
intrinsically disordered proteins, CAID, blind test, deep learning, physicochemical methods, DisProt, prediction accuracy, benchmarking
MeSH terms
Amino Acid Sequence, Protein Binding, Protein Conformation, Software, Protein Folding, Computational Biology, Databases, Protein, Intrinsically Disordered Proteins

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review