The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
NCBI disease corpus: A resource for disease name recognition and concept normalization
Open accessAbstract
Information encoded in natural language in biomedical literature publications is only useful if efficient and reliable ways of accessing and analyzing that information are available. Natural language processing and text mining tools are therefore essential for extracting valuable information, however, the development of powerful, highly effective tools to automatically detect central biomedical concepts such as diseases is conditional on the availability of annotated corpora. This paper presents the disease name and concept annotations of the NCBI disease corpus, a collection of 793 PubMed abstracts fully annotated at the mention and concept level to serve as a research resource for the biomedical natural language processing community. Each PubMed abstract was manually annotated by two annotators with disease mentions and their corresponding concepts in Medical Subject Headings (MeSH®) or Online Mendelian Inheritance in Man (OMIM®). Manual curation was performed using PubTator, which allowed the use of pre-annotations as a pre-step to manual annotations. Fourteen annotators were randomly paired and differing annotations were discussed for reaching a consensus in two annotation phases. In this setting, a high inter-annotator agreement was observed. Finally, all results were checked against annotations of the rest of the corpus to assure corpus-wide consistency. The public release of the NCBI disease corpus contains 6892 disease mentions, which are mapped to 790 unique disease concepts. Of these, 88% link to a MeSH identifier, while the rest contain an OMIM identifier. We were able to link 91% of the mentions to a single disease concept, while the rest are described as a combination of concepts. In order to help researchers use the corpus to design and test disease identification methods, we have prepared the corpus as training, testing and development sets. To demonstrate its utility, we conducted a benchmarking experiment where we compared three different knowledge-based disease normalization methods with a best performance in F-measure of 63.7%. These results show that the NCBI disease corpus has the potential to significantly improve the state-of-the-art in disease name recognition and normalization research, by providing a high-quality gold standard thus enabling the development of machine-learning based approaches for such tasks. The NCBI disease corpus, guidelines and other associated resources are available at: http://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356
GENIA corpus—a semantically annotated corpus for bio-textmining
Kim, J.-D., Ohta, T., Tateisi, Y., & Tsujii, J. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg1023 · Cited by 1,232
A large-scale evaluation of computational protein function prediction
Radivojac, P., Clark, W. T., Oron, T. R., Schnoes, A. M., Wittkop, T., Sokolov, A., Graim, K., Funk, C., Verspoor, K., Ben-Hur, A., et al. (2013) · Nature Methods · doi:10.1038/nmeth.2340 · Cited by 1,028 · cited 1×
“Recent research efforts on large-scale community-wide assessment of protein function annotation [45] illustrate the importance of annotation and the use of the appropriate ontology for both annotation and evaluation of computational methods.”
BackgroundAgreement, the F-Measure, and Reliability in Information Retrieval
Hripcsak, G. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1733 · Cited by 949
Disease Ontology: a backbone for disease semantic integration
Schriml, L. M., Arze, C., Nadendla, S., Chang, Y.-W. W., Mazaitis, M., Felix, V., Feng, G., & Kibbe, W. A. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr972 · Cited by 929
Digital disease detection - harnessing the Web for public health surveillance
Brownstein, J. S., Freifeld, C. C., & Madoff, L. C. (2009) · New England Journal of Medicine · doi:10.1056/nejmp0900702 · Cited by 819
PubTator: a web-based text mining tool for assisting biocuration
Wei, C.-H., Kao, H.-Y., & Lu, Z. (2013) · Nucleic Acids Research · doi:10.1093/nar/gkt441 · Cited by 566
Self-citationBioInfer: a corpus for information extraction in the biomedical domain
Pyysalo, S., Ginter, F., Heimonen, J., Björne, J., Boberg, J., Järvinen, J., & Salakoski, T. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-50 · Cited by 485
DNorm: disease name normalization with pairwise learning to rank
Leaman, R., Islamaj Doğan, R., & Lu, Z. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt474 · Cited by 484
Self-citationBANNER: AN EXECUTABLE SURVEY OF ADVANCES IN BIOMEDICAL NAMED ENTITY RECOGNITION
LEAMAN, R., & GONZALEZ, G. (2007) · Biocomputing 2008 · doi:10.1142/9789812776136_0062 · Cited by 479
Self-citationRobinson, P., & Mundlos, S. (2010) · Clinical Genetics · doi:10.1111/j.1399-0004.2010.01436.x · Cited by 354
LINNAEUS: A species name identification system for biomedical literature
Gerner, M., Nenadic, G., & Bergman, C. M. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-85 · Cited by 335
Overview of BioCreative II gene normalization
Morgan, A. A., Lu, Z., Wang, X., Cohen, A. M., Fluck, J., Ruch, P., Divoli, A., Fundel, K., Leaman, R., Hakenberg, J., et al. (2008) · Genome Biology · doi:10.1186/gb-2008-9-s2-s3 · Cited by 327
Self-citationForty years of SNOMED: a literature review
Cornet, R., & de Keizer, N. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-s1-s2 · Cited by 299
Concept annotation in the CRAFT corpus
Bada, M., Eckert, M., Evans, D., Garcia, K., Shipley, K., Sitnikov, D., Baumgartner, W. A., Cohen, K. B., Verspoor, K., Blake, J. A., et al. (2012) · BMC Bioinformatics · doi:10.1186/1471-2105-13-161 · Cited by 272
Understanding PubMed(R) user search behavior through log analysis
Islamaj Dogan, R., Murray, G. C., Neveol, A., & Lu, Z. (2009) · Database · doi:10.1093/database/bap018 · Cited by 265
Self-citationGENETAG: a tagged corpus for gene/protein named entity recognition
Tanabe, L., Xie, N., Thom, L. H., Matten, W., & Wilbur, W. J. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-s1-s3 · Cited by 231
Getting Started in Text Mining
Cohen, K. B., & Hunter, L. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.0040020 · Cited by 230
Integrated Annotation for Biomedical Information Extraction
Seth Kulick, A., Ann Bies, A., Liberman, M., Mandel, M. A., McDonald, R., Palmer, M. S., Schein, A. I., Ungar, L., Winters, S., & White, P. (2004) · North American Chapter of the Association for Computational Linguistics · Cited by 186
The gene normalization task in BioCreative III
Lu, Z., Kao, H.-Y., Wei, C.-H., Huang, M., Liu, J., Kuo, C.-J., Hsu, C.-N., Tsai, R. T.-H., Dai, H.-J., Okazaki, N., et al. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s8-s2 · Cited by 181
Self-citationMEDIC: a practical disease vocabulary used at the Comparative Toxicogenomics Database
Davis, A. P., Wiegers, T. C., Rosenstein, M. C., & Mattingly, C. J. (2012) · Database · doi:10.1093/database/bar065 · Cited by 166
Overview of BioCreAtIvE task 1B: normalized gene lists
Hirschman, L., Colosimo, M., Morgan, A., & Yeh, A. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-s1-s11 · Cited by 164
Verspoor, K. M., Cohen, K. B., Lanfranchi, A., Warner, C., Johnson, H. L., Roeder, C., Choi, J. D., Funk, C., Malenkiy, Y., Eckert, M., et al. (2012) · BMC Bioinformatics · doi:10.1186/1471-2105-13-207 · Cited by 142
Using rule-based natural language processing to improve disease normalization in biomedical text
Kang, N., Singh, B., Afzal, Z., van Mulligen, E. M., & Kors, J. A. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001173 · Cited by 132
Assessment of disease named entity recognition on a corpus of annotated sentences
Yepes, A. J., Jiménez-Ruiz, E., Lee, V., Gaudan, S., Berlanga, R., & Rebholz‐Schuhmann, D. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-s3-s3 · Cited by 123
Construction of an annotated corpus to support biomedical information extraction
Thompson, P., Iqbal, S. A., McNaught, J., & Ananiadou, S. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-349 · Cited by 121
Lease, M., & Eugene Charniak, D. (2005) · Lecture notes in computer science · doi:10.1007/11562214_6 · Cited by 120
Natural Language Processing – IJCNLP 2005
Dale, R., Wong, K.-F., Su, J., et al. (2005) · Lecture notes in computer science · doi:10.1007/11562214 · Cited by 108
Wei, C.-H., Harris, B. R., Li, D., Berardini, T., Huala, E., Kao, H.-Y., & Lu, Z. (2012) · Database · doi:10.1093/database/bas041 · Cited by 96
Self-citationSemi-automatic semantic annotation of PubMed queries: A study on quality, efficiency, satisfaction
Névéol, A., Islamaj, R., & Lu, Z. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2010.11.001 · Cited by 93
Self-citationAnalysis of Biological Processes and Diseases Using Text Mining Approaches
Krallinger, M., Leitner, F., & Valencia, A. (2009) · Methods in molecular biology · doi:10.1007/978-1-60327-194-3_16 · Cited by 92
Unified Medical Language System term occurrences in clinical notes: a large-scale corpus analysis
Wu, S. T., Liu, H., Li, D., Tao, C., Musen, M. A., Chute, C. G., & Shah, N. H. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000744 · Cited by 90
An improved corpus of disease mentions in PubMed citations
Islamaj, R., & Lu, Z. (2012) · Cited by 65
Self-citationExtracting Rx information from clinical narrative
Mork, J. G., Bodenreider, O., Demner‐Fushman, D., Islamaj, R., Lang, F.-M., Lu, Z., Aurélie Névéol, A., Peters, L. B., Shooshan, S. E., & Aronson, A. R. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.003970 · Cited by 31
Self-citationA reappraisal of sentence and token splitting for life sciences documents.
Tomanek, K., Wermter, J., & Hahn, U. (2007) · PubMed · Cited by 30
An Inference Method for Disease Name Normalization.
Islamaj, R., & Lu, Z. (2012) · National Conference on Artificial Intelligence · Cited by 24
Self-citationExploring two biomedical text genres for disease recognition
Névéol, A., Kim, W. B., Wilbur, W. J., & Lu, Z. (2009) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.3115/1572364.1572385 · Cited by 23
Self-citationA context-blocks model for identifying clinical relationships in patient records
Islamaj, R., Névéol, A., & Lu, Z. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s3-s3 · Cited by 21
Self-citationLinking multiple disease-related resources through UMLS
Névéol, A., Li, J., & Lu, Z. (2012) · International Health Informatics Symposium · doi:10.1145/2110363.2110455 · Cited by 11
Self-citation(2001) · doi:10.1002/pds.4850 · Cited by 7
Inherited Neuromuscular Disease
Kichula, E. A., Ferguson, T. A., & Lynch, D. R. (2017) · Encyclopedia of Life Sciences · doi:10.1002/9780470015902.a0005513.pub2 · Cited by 2
Cited by
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
A Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169
BioCreative V CDR task corpus: a resource for chemical disease relation extraction
Li, J., Sun, Y., Johnson, R. J., Sciaky, D., Wei, C.-H., Leaman, R., Davis, A. P., Mattingly, C. J., Wiegers, T. C., & Lu, Z. (2016) · Database · doi:10.1093/database/baw068 · Cited by 870
Self-citation
Metadata
- Authors
- Doğan, R. I., Leaman, R., & Lu, Z.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2014-02
- Volume
- 47
- Pages
- 1-10
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2169099542
- Cited by
- 820
- References
- 41
- FWCI
- 10.95
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.