The self‐assessment trap: can we all be better than average?
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Correspondence11 October 2011Open Access The self-assessment trap: can we all be better than average? Raquel Norel Raquel Norel IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author John Jeremy Rice John Jeremy Rice IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author Gustavo Stolovitzky Gustavo Stolovitzky IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author Raquel Norel Raquel Norel IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author John Jeremy Rice John Jeremy Rice IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author Gustavo Stolovitzky Gustavo Stolovitzky IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Search for more papers by this author Author Information Raquel Norel1, John Jeremy Rice1 and Gustavo Stolovitzky1 1IBM Computational Biology Center, IBM T.J. Watson Research Center, Yorktown Heights, NY, USA Molecular Systems Biology (2011)7:537https://doi.org/10.1038/msb.2011.70 PDFDownload PDF of article text and main figures. ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinked InMendeleyWechatReddit Figures & Info Computational systems biology seems to be caught in what we call the 'self-assessment trap', in which researchers wishing to publish their analytical methods are required by referees or by editorial policy (e.g., Bioinformatics, BMC Bioinformatics, Nucleic Acids Research) to compare the performance of their own algorithms against other methodologies, thus being forced to be judge, jury and executioner. The result is that the authors' method tends to be the best in an unreasonable majority of cases (Table I). In many instances, this bias is the result of selective reporting of performance in the niche in which the method is superior. Evidence of that is that most papers reporting best performance choose only one or two metrics of performance, but when the number of performance metrics is larger than two, most methods fail to be the best in all categories assessed (Table I). Choosing many metrics can dramatically change the determination of best performance (Supplementary Table S1). Selective reporting can be inadvertent, but in some cases biases are more disingenuous, involving hiding information or quietly cutting corners in the performance evaluation (similar problems have been discussed in assessments of the performance of supercomputers, e.g., Bailey (1991)). Table 1. Break out of 57 surveyed papers in which the authors assess their own methods Number of performance metrics Total number of studies surveyed Authors' method is the best in all metrics and all data sets Authors' method is the best in most metrics and most data sets 1 25 19 6 2 15 13 2 3 7 4 3 4 4 1 3 5 4 1 3 6 2 1 1 Note that we did not find any self-assessment paper where the presented method was not top ranked in at least one metric or data set. The survey was conducted over a large pool of scientific peer-reviewed papers selected as follows. First, a Google Scholar search using the keywords 'computational biology method assessment' was conducted. When papers with comparisons of methods were identified, we further examined (1) papers from the same journal issue and (2) downstream papers that cite the identified paper (as determined by Google Scholar). The 57 papers (see Supplementary information) resulting from the search span 22 journals. Most papers are in the categories of gene regulatory networks/reverse engineering (24/69), structure prediction/assessment (14/69) and DNA-protein interactions/regulatory element identification. An additional nine papers found in the same manner but not shown in the Table reported independently (not-self) assessed methods, of which only four were top performers, whereas five reported methods that ranked high but were not top performers. Even assuming that there is no selective reporting, we would like to argue that papers reporting good-yet-not-the-best methods (of which we found none in our literature survey of self-assessed papers listed in the Supplementary information) can still advance science. For example, a method that is not top ranked can still have value by unearthing biological results that are complementary to the results reported by other better performing methods. Furthermore, the effectiveness of a top-performing algorithm can be boosted when its results are aggregated with second and third best performers (Figure 1, and Supplementary Figures S1 and S2; Marbach et al, 2010; Prill et al, 2010). The discussion above suggests that self-evaluation is suspect and that insistence on publication of only best perf
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Replicating genotype–phenotype associations
NCI-NHGRI Working Group on Replication in Association Studies, Chanock, S. J., Manolio, T., Boehnke, M., Boerwinkle, E., Hunter, D. J., Thomas, G., Hirschhorn, J. N., Abecasis, G., Altshuler, D., et al. (2007) · Nature · doi:10.1038/447655a · Cited by 1,352 · cited 1×
“…or refute previous results To a large extent, the remedies suggested above have been addressed in the context of genome-wide association studies (Chanock et al , 2007), and are embodied in existing independent assessments presented to the scientific community in efforts such as CASP…”
Self-citationTowards a rigorous assessment of systems biology models: the DREAM3 challenges
Prill, R. J., Marbach, D., Saez-Rodriguez, J., Sorger, P. K., Alexopoulos, L. G., Xue, X., Clarke, N. D., Altan-Bonnet, G., & Stolovitzky, G. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009202 · Cited by 439 · cited 1×
“Furthermore, the effectiveness of a top-performing algorithm can be boosted when its results are aggregated with second and third best performers (Figure 1, and Supplementary Figures S1 and S2; Marbach et al , 2010; Prill et al , 2010).”
Self-citation
Cited by
A comparison of single-cell trajectory inference methods
Saelens, W., Cannoodt, R., Todorov, H., & Saeys, Y. (2019) · Nature Biotechnology · doi:10.1038/s41587-019-0071-9 · Cited by 1,670
A community computational challenge to predict the activity of pairs of compounds
Community, N.-D., Bansal, M., Yang, J., Karan, C., Menden, M. P., Costello, J. C., Tang, H., Xiao, G., Li, Y., Allen, J., et al. (2014) · Nature Biotechnology · doi:10.1038/nbt.3052 · Cited by 321
“The experimental data set was originally intended to validate SynGen, a method explicitly designed to predict synergy and not antagonism.”
MethodSelf-citationA comprehensive evaluation of module detection methods for gene expression data
Saelens, W., Cannoodt, R., & Saeys, Y. (2018) · Nature Communications · doi:10.1038/s41467-018-03424-4 · Cited by 311
Essential guidelines for computational method benchmarking
Weber, L. M., Saelens, W., Cannoodt, R., Soneson, C., Hapfelmeier, A., Gardner, P. P., Boulesteix, A.-L., Saeys, Y., & Robinson, M. D. (2019) · Genome Biology · doi:10.1186/s13059-019-1738-8 · Cited by 21
Bergquist, T., Schaffter, T., Yan, Y., Yu, T., Prosser, J., Gao, J., Chen, G., Charzewski, Ł., Nawalany, Z., Brugere, I., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad159 · Cited by 8
Metadata
- Authors
- Norel, R., Rice, J. J., & Stolovitzky, G.
- Venue
- Molecular Systems Biology
- Published
- 2011-01
- Volume
- 7
- Issue
- 1
- Pages
- 537-537
- ISSN
- 1744-4292 (ISSN-L)
- OpenAlex
- W2075740080
- Cited by
- 96
- References
- 2
- FWCI
- 1.41
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.