Evaluation of BioCreAtIvE assessment of task 2
Open accessThe DOI matches its Crossref record.
Abstract
Background: Molecular Biology accumulated substantial amounts of data concerning functions of genes and proteins. Information relating to functional descriptions is generally extracted manually from textual data and stored in biological databases to build up annotations for large collections of gene products. Those annotation databases are crucial for the interpretation of large scale analysis approaches using bioinformatics or experimental techniques. Due to the growing accumulation of functional descriptions in biomedical literature the need for text mining tools to facilitate the extraction of such annotations is urgent. In order to make text mining tools useable in real world scenarios, for instance to assist database curators during annotation of protein function, comparisons and evaluations of different approaches on full text articles are needed. Results: The Critical Assessment for Information Extraction in Biology (BioCreAtIvE) contest consists of a community wide competition aiming to evaluate different strategies for text mining tools, as applied to biomedical literature. We report on task two which addressed the automatic extraction and assignment of Gene Ontology (GO) annotations of human proteins, using full text articles. The predictions of task 2 are based on triplets of protein – GO term – article passage. The annotation-relevant text passages were returned by the participants and evaluated by expert curators of the GO annotation (GOA) team at the European Institute of Bioinformatics (EBI). Each participant could submit up to three results for each sub-task comprising task 2. In total more than 15,000 individual results were provided by the participants. The curators evaluated in addition to the annotation itself, whether the protein and the GO term were correctly predicted and traceable through the submitted text fragment. Conclusion: Concepts provided by GO are currently the most extended set of terms used for annotating gene products, thus they were explored to assess how effectively text mining tools are able to extract those annotations automatically. Although the obtained results are promising, they are still far from reaching the required performance demanded by real world applications. Among the principal difficulties encountered to address the proposed task, were the complex nature of the GO terms and protein names (the large range of variants which are used to express proteins and especially GO terms in free text), and the lack of a standard training set. A range of very different strategies were used to tackle this task. The dataset generated in line with the BioCreative challenge is publicly available and will allow new possibilities for training information extraction methods in the domain of molecular biology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Overview of BioCreAtIvE: critical assessment of information extraction for biology
Hirschman, L., Yeh, A., Blaschke, C., & Valencia, A. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-s1-s1 · Cited by 530
Self-citationReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Biomedical Ontologies in Action: Role in Knowledge Management, Data Integration and Decision Support
Bodenreider, O. (2008) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638585 · Cited by 368
“Here again, (semi-)automatic methods for acquiring annotations from text have been investigated recently [59-63], but annotations are still most often the product of manual curation.”
MethodAutomated protein function prediction--the genomic challenge
Friedberg, I. (2006) · Briefings in Bioinformatics · doi:10.1093/bib/bbl004 · Cited by 366
Overview of the protein-protein interaction annotation extraction task of BioCreative II
Krallinger, M., Leitner, F., Rodriguez-Penagos, C., & Valencia, A. (2008) · Genome Biology · doi:10.1186/gb-2008-9-s2-s4 · Cited by 298
Self-citation
Metadata
- Authors
- Blaschke, C., Leon, E. A., Krallinger, M., & Valencia, A.
- Venue
- BMC Bioinformatics
- Published
- 2005-05
- Volume
- 6
- Issue
- S1
- Pages
- S16-S16
- ISSN
- 1471-2105 (ISSN-L)
- OpenAlex
- W2056119436
- Cited by
- 139
- References
- 0
- FWCI
- 7.14
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluation of BioCreAtIvE assessment of task 2. https://app.cochat.ai/papers/10.1186/1471-2105-6-s1-s16
- The paper
- Blaschke, C., Leon, E. A., Krallinger, M., & Valencia, A. (2005). Evaluation of BioCreAtIvE assessment of task 2. BMC Bioinformatics. https://doi.org/10.1186/1471-2105-6-s1-s16
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.