Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 213

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Evaluation of text-mining systems for biology: overview of the Second BioCreative community challenge

Open access
Krallinger, Martin, Morgan, Alexander, Smith, Larry, Leitner, Florian, Tanabe, Lorraine, Wilbur, John, Hirschman, Lynette, Valencia, Alfonso·2008·Genome Biology

Abstract

Background: Genome sciences have experienced an increasing demand for efficient text-processing tools that can extract biologically relevant information from the growing amount of published literature. In response, a range of text-mining and information-extraction tools have recently been developed specifically for the biological domain. Such tools are only useful if they are designed to meet real-life tasks and if their performance can be estimated and compared. The BioCreative challenge (Critical Assessment of Information Extraction in Biology) consists of a collaborative initiative to provide a common evaluation framework for monitoring and assessing the state-of-the-art of text-mining systems applied to biologically relevant problems. Results: The Second BioCreative assessment (2006 to 2007) attracted 44 teams from 13 countries worldwide, with the aim of evaluating current information-extraction/text-mining technologies developed for one or more of the three tasks defined for this challenge evaluation. These tasks included the recognition of gene mentions in abstracts (gene mention task); the extraction of a list of unique identifiers for human genes mentioned in abstracts (gene normalization task); and finally the extraction of physical protein-protein interaction annotation-relevant information (protein-protein interaction task). The 'gold standard' data used for evaluating submissions for the third task was provided by the interaction databases MINT (Molecular Interaction Database) and IntAct. Conclusion: The Second BioCreative assessment almost doubled the number of participants for each individual task when compared with the first BioCreative assessment. An overall improvement in terms of balanced precision and recall was observed for the best submissions for the gene mention (F score 0.87); for the gene normalization task, the best results were comparable (F score 0.81) compared with results obtained for similar tasks posed at the first BioCreative challenge. In case of the protein-protein interaction task, the importance and difficulties of experimentally confirmed annotation extraction from full-text articles were explored, yielding different results depending on the step of the annotation extraction workflow. A common characteristic observed in all three tasks was that the combination of system outputs could yield better results than any single system. Finally, the development of the first text-mining meta-server was promoted within the context of this community challenge.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/gb-2008-9-s2-s1

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 20

  1. Menne K, Hermjakob H, Apweiler R: A comparison of signal sequence prediction methods using a test set of signal peptides. Bioinformatics. 2000, 16: 741-742. 10.1093/bioinformatics/16.8.741. · DOI
  2. Sprenger J, Fink J, Teasdale R: Evaluation and comparison of mammalian subcellular localization prediction methods. BMC Bioinformatics. 2006, 7: S3-10.1186/1471-2105-7-S5-S3. · doi:10.1186/1471-2105-7-s5-s3
  3. Dalal S, Balasubramanian S, Regan L: Protein alchemy: changing beta-sheet into alpha-helix. Nat Struct Biol. 1997, 4: 548-552. 10.1038/nsb0797-548. · DOI · Paper page
  4. Lopez G, Rojas A, Tress M, Valencia A: Assessment of predictions submitted for the CASP7 function prediction category. Proteins. 2007, 69 (suppl 8): 165-174. 10.1002/prot.21651. · DOI
  5. Guigo R, Flicek P, Abril J, Reymond A, Lagarde J, Denoeud F, Antonarakis S, Ashburner M, Bajic V, Birney E, Castelo R, Eyras E, Ucla C, Gingeras T, Harrow J, Hubbard T, Lewis S, Reese M: EGASP: the human ENCODE Genome Annotation Assessment Project. Genome Biol. 2006, 7: 1-31. 10.1186/gb-2006-7-s1-s2. · DOI
  6. Hirschman L: The evolution of evaluation: lessons from the message understanding conference. Computer Speech Lang. 1998, 12: 281-305. 10.1006/csla.1998.0102. · DOI
  7. Bar-Haim R, Dagan I, Dolan B, Ferro L, Giampiccolo D, Magnini B, Szpektor I: The Second PASCAL Recognising Textual Entailment Challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. 2006, [http://www.cs.biu.ac.il/~szpekti/papers/RTE2-organizers.pdf]
  8. Yeh A, Hirschman L, Morgan A: Evaluation of text data mining for database curation: lessons learned from the KDD Challenge Cup. Bioinformatics. 2003, 19: 331-339. 10.1093/bioinformatics/btg1046. · DOI · Paper page
  9. Nedellec C: Learning language in logic-genic interaction extraction challenge. Proceedings of the LLL05 workshop. 2005, [http://www.cs.york.ac.uk/aig/lll/lll05/lll05-nedellec.pdf]
  10. Kim J, Ohta T, Tsuruoka Y, Tateisi Y, Collier N: Introduction to the Bio-Entity Task at JNLPBA. Proceedings of the BioCreative Challenge Evaluation Workshop. 2004
  11. Hirschman L, Yeh A, Blaschke C, Valencia A: Overview of BioCreAtIvE: critical assessment of information extraction for biology. BMC Bioinformatics. 2005, 6: S1-10.1186/1471-2105-6-S1-S1. · DOI · Paper page
  12. Yeh A, Morgan A, Colosimo M, Hirschman L: BioCreAtIvE Task 1A: gene mention finding evaluation. BMC Bioinformatics. 2005, 6: S2-10.1186/1471-2105-6-S1-S2. · DOI · Paper page
  13. Hirschman L, Colosimo M, Morgan A, Yeh A: Overview of BioCreAtIvE task 1B: normalized gene lists. BMC Bioinformatics. 2005, 6: S11-10.1186/1471-2105-6-S1-S11. · DOI · Paper page
  14. Blaschke C, Andres Leon E, Krallinger M, Valencia A: Evaluation of BioCreative assessment of task 2. BMC Bioinformatics. 2005, 6: S16-10.1186/1471-2105-6-S1-S16. · DOI
  15. Morgan A, Lu Z, Wang X, Cohen AM, Fluck J, Ruch P, Divoli A, Fundel K, Leaman R, Hakenberg J, Sun C, Liu Hh, Torres R, Krauthammer M, Lau WW, Liu H, Hsu CN, Schuemie M, Cohen KB, Hirschman L: Overview of BioCreative II gene normalization. Genome Biol. 2008, 9 (Suppl 2): S3-10.1186/gb-2008-9-s2-s3. · DOI · Paper page
  16. Smith L, Tanabe L, Ando R, Kuo C, Chung I, Hsu C, Lin Y, Klinger R, Friedrich C, Ganchev K, Torii M, Liu H, Haddow B, Struble C, Povinelli R, Vlachos A, Baumgartner W, Hunter L, Carpenter B, Tsai R, Dai H, Liu F, Chen Y, Sun C, Katrenko S, Adriaans P, Blaschke C, Torres R, Neves M, Nakov P, et al: Overview of BioCreative II gene mention recognition. Genome Biol. 2008, 9 (Suppl 2): S2-10.1186/gb-2008-9-s2-s2. · DOI · Paper page
  17. Chatr-aryamontri A, Kerrien S, Khadake J, Orchard S, Ceol A, Licata L, Castagnoli L, Costa S, Derow C, Huntley R, Aranda B, Leroy C, Thorneycroft D, Apweiler R, Cesareni G, Hermjakob H: MINT and IntAct contribute to the Second BioCreative Challenge: serving the text-mining community with high quality molecular interaction data. Genome Biol. 2008, 9 (Suppl 2): S5-10.1186/gb-2008-9-s2-s5. · DOI
  18. Noble W: What is a support vector machine?. Nat Biotechnol. 2006, 24: 1565-1567. 10.1038/nbt1206-1565. · DOI · Paper page
  19. Lafferty J, McCallum A, Pereira F: Conditional random fields: probabilistic models for segmenting and labeling sequence data. Proceedings of the ICML 01. 2001, [http://www.cis.upenn.edu/~pereira/papers/crf.pdf]
  20. Tsuruoka Y, Tsujii J: Bidirectional inference with the easiest-first strategy for tagging sequence data. Proceedings of HLT/EMNLP. 2005, [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/emnlp05bidir.pdf]

Cited by

See all 213

Metadata

Authors
Krallinger, M., Morgan, A., Smith, L., Leitner, F., Tanabe, L., Wilbur, J., Hirschman, L., & Valencia, A.
Venue
Genome Biology
Published
2008-09-01
Volume
9
Issue
S2
Pages
S1-S1
ISSN
  • 1465-6906 (ISSN-L)
  • 1474-760X
OpenAlex
W2101727078
Cited by
213
References
1
FWCI
9.55
Type
journal article
Language
en
Open access
gold
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Evaluation of text-mining systems for biology: overview of the Second BioCreative community challenge. https://app.cochat.ai/papers/10.1186/gb-2008-9-s2-s1
The paper
Krallinger, M., Morgan, A., Smith, L., Leitner, F., Tanabe, L., Wilbur, J., Hirschman, L., & Valencia, A. (2008). Evaluation of text-mining systems for biology: overview of the Second BioCreative community challenge. Genome Biology. https://doi.org/10.1186/gb-2008-9-s2-s1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.