Cited by 168Protein Structure and DynamicsEnzyme Structure and FunctionMachine Learning in Bioinformatics

Continuous Automated Model EvaluatiOn (CAMEO) complementing the critical assessment of structure prediction in CASP12

Open access
Haas, Jürgen, Barbato, Alessandro, Behringer, Dario, Studer, Gabriel, Roth, Steven, Bertoni, Martino, Mostaguir, Khaled, Gumienny, Rafal +1 more·2018·Proteins: Structure, Function, and Bioinformatics

Abstract

Every second year, the community experiment “Critical Assessment of Techniques for Structure Prediction” (CASP) is conducting an independent blind assessment of structure prediction methods, providing a framework for comparing the performance of different approaches and discussing the latest developments in the field. Yet, developers of automated computational modeling methods clearly benefit from more frequent evaluations based on larger sets of data. The “Continuous Automated Model EvaluatiOn (CAMEO)” platform complements the CASP experiment by conducting fully automated blind prediction assessments based on the weekly pre‐release of sequences of those structures, which are going to be published in the next release of the PDB Protein Data Bank. CAMEO publishes weekly benchmarking results based on models collected during a 4‐day prediction window, on average assessing ca. 100 targets during a time frame of 5 weeks. CAMEO benchmarking data is generated consistently for all participating methods at the same point in time, enabling developers to benchmark and cross‐validate their method's performance, and directly refer to the benchmarking results in publications. In order to facilitate server development and promote shorter release cycles, CAMEO sends weekly email with submission statistics and low performance warnings. Many participants of CASP have successfully employed CAMEO when preparing their methods for upcoming community experiments. CAMEO offers a variety of scores to allow benchmarking diverse aspects of structure prediction methods. By introducing new scoring schemes, CAMEO facilitates new development in areas of active research, for example, modeling quaternary structure, complexes, or ligand binding sites.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/prot.25431

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • During method development, a temporally held out validation set from CAMEO (August 2021 to January 2022) was used.

    “During method development (e.g. hyperparameter selection), we used a temporally held out validation set obtained from the Continuous Automated Model EvaluatiOn (CAMEO) server (36) by filtering from August 2021 to January 2022.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

3 of the 60 references couldn’t be matched to a known paper.

Research works printed but not linked: 1

  1. [53]Bittrich S F.H.a.D.L. eQuant - A Server for Fast Protein Model Quality Assessment by Integrating High-Dimensional Data and Machine Learning. Beyond Databases, Architectures and Structures Advanced Technologies for Data Mining and Knowledge Discovery 2016 Springer 14

Cited by

Sorted by citations

Authors
Haas, J., Barbato, A., Behringer, D., Studer, G., Roth, S., Bertoni, M., Mostaguir, K., Gumienny, R., & Schwede, T.
Affiliations
  • Biozentrum University of Basel Switzerland
  • SIB Swiss Institute of Bioinformatics (CH)
Corresponding author
Schwede, Torsten
Venue
Proteins: Structure, Function, and Bioinformatics
Publisher
Wiley
Published
2017-12-17
Volume
86
Issue
S1
Pages
387-398
ISSN
  • 0887-3585 (ISSN-L)
  • 1097-0134
OpenAlex
W2770647690
Cited by
168
References
60
Citations by year
2026: 4 · 2025: 20 · 2024: 23 · 2023: 24 · 2022: 20 · 2021: 26 · 2020: 21 · 2019: 21 · 2018: 8 · 2017: 1
FWCI
5.69
Type
journal article
Language
en
Open access
Locations
  • Proteins Structure Function and Bioinformatics (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
  • edoc (University of Basel) (submittedVersion)
Funders
National Institute of General Medical Sciences (U01 GM093324‐01)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Physical Sciences › Materials Science › Materials Chemistry › Enzyme Structure and Function
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Keywords
caspase-12, protein structure prediction, automated benchmarking, protein structure modeling, scoring system, ligand binding site prediction, method performance evaluation
MeSH terms
Binding Sites, Humans, Ligands, Models, Molecular, Protein Binding, Protein Conformation, Proteins, Computational Biology, Sequence Analysis, Protein, Databases, Protein

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review