CoChat

The self‐assessment trap: can we all be better than average?

Open access
Norel, Raquel, Rice, John Jeremy, Stolovitzky, Gustavo · 2011 · Molecular Systems Biology
Cited by 96

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Abstract

Correspondence11 October 2011Open Access The self-assessment trap: can we all be better than average? Raquel Norel Raquel Norel IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author John Jeremy Rice John Jeremy Rice IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author Gustavo Stolovitzky Gustavo Stolovitzky IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author Raquel Norel Raquel Norel IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author John Jeremy Rice John Jeremy Rice IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author Gustavo Stolovitzky Gustavo Stolovitzky IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author Author Information Raquel Norel1, John Jeremy Rice1 and Gustavo Stolovitzky1 1IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Molecular Systems Biology (2011)7:537https://doi.org/10.1038/msb.2011.70 PDFDownload PDF of article text and main figures. ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinked InMendeleyWechatReddit Figures & Info Computational systems biology seems to be caught in what we call the 'self-assessment trap', in which researchers wishing to publish their analytical methods are required by referees or by editorial policy (e.g., Bioinformatics, BMC Bioinformatics, Nucleic Acids Research) to compare the performance of their own algorithms against other methodologies, thus being forced to be judge, jury and executioner. The result is that the authors' method tends to be the best in an unreasonable majority of cases (Table I). In many instances, this bias is the result of selective reporting of performance in the niche in which the method is superior. Evidence of that is that most papers reporting best performance choose only one or two metrics of performance, but when the number of performance metrics is larger than two, most methods fail to be the best in all categories assessed (Table I). Choosing many metrics can dramatically change the determination of best performance (Supplementary Table S1). Selective reporting can be inadvertent, but in some cases biases are more disingenuous, involving hiding information or quietly cutting corners in the performance evaluation (similar problems have been discussed in assessments of the performance of supercomputers, e.g., Bailey (1991)). Table 1. Break out of 57 surveyed papers in which the authors assess their own methods Number of performance metrics Total number of studies surveyed Authors' method is the best in all metrics and all data sets Authors' method is the best in most metrics and most data sets 1 25 19 6 2 15 13 2 3 7 4 3 4 4 1 3 5 4 1 3 6 2 1 1 Note that we did not find any self-assessment paper where the presented method was not top ranked in at least one metric or data set. The survey was conducted over a large pool of scientific peer-reviewed papers selected as follows. First, a Google Scholar search using the keywords 'computational biology method assessment' was conducted. When papers with comparisons of methods were identified, we further examined (1) papers from the same journal issue and (2) downstream papers that cite the identified paper (as determined by Google Scholar). The 57 papers (see Supplementary information) resulting from the search span 22 journals. Most papers are in the categories of gene regulatory networks/reverse engineering (24/69), structure prediction/assessment (14/69) and DNA-protein interactions/regulatory element identification. An additional nine papers found in the same manner but not shown in the Table reported independently (not-self) assessed methods, of which only four were top performers, whereas five reported methods that ranked high but were not top performers. Even assuming that there is no selective reporting, we would like to argue that papers reporting good-yet-not-the-best methods (of which we found none in our literature survey of self-assessed papers listed in the Supplementary information) can still advance science. For example, a method that is not top ranked can still have value by unearthing biological results that are complementary to the results reported by other better performing methods. Furthermore, the effectiveness of a top-performing algorithm can be boosted when its results are aggregated with second and third best performers (Figure 1, and Supplementary Figures S1 and S2; Marbach et al, 2010; Prill et al, 2010). The discussion above suggests that self-evaluation is suspect and that insistence on publication of only best perf

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/msb.2011.70

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Replicating genotype–phenotype associations

    NCI-NHGRI Working Group on Replication in Association Studies, Chanock, S. J., Manolio, T., Boehnke, M., Boerwinkle, E., Hunter, D. J., Thomas, G., Hirschhorn, J. N., Abecasis, G., Altshuler, D., et al. (2007) · Nature · doi:10.1038/447655a · Cited by 1,352 · cited 1×

    “…or refute previous results To a large extent, the remedies suggested above have been addressed in the context of genome-wide association studies (Chanock et al , 2007), and are embodied in existing independent assessments presented to the scientific community in efforts such as CASP…”
    Self-citation
  • Towards a rigorous assessment of systems biology models: the DREAM3 challenges

    Prill, R. J., Marbach, D., Saez-Rodriguez, J., Sorger, P. K., Alexopoulos, L. G., Xue, X., Clarke, N. D., Altan-Bonnet, G., & Stolovitzky, G. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009202 · Cited by 439 · cited 1×

    “Furthermore, the effectiveness of a top-performing algorithm can be boosted when its results are aggregated with second and third best performers (Figure 1, and Supplementary Figures S1 and S2; Marbach et al , 2010; Prill et al , 2010).”
    Self-citation

Show all 2 references

Cited by

See all 96

Metadata

Authors
Norel, R., Rice, J. J., & Stolovitzky, G.
Venue
Molecular Systems Biology
Published
2011-01
Volume
7
Issue
1
Pages
537-537
ISSN
1744-4292 (ISSN-L)
OpenAlex
W2075740080
Cited by
96
References
2
FWCI
1.41
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The self‐assessment trap: can we all be better than average? https://app.cochat.ai/papers/10.1038/msb.2011.70
The paper
Norel, R., Rice, J. J., & Stolovitzky, G. (2011). The self‐assessment trap: can we all be better than average? Molecular Systems Biology. https://doi.org/10.1038/msb.2011.70

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.