How to Train good Word Embeddings for Biomedical NLP
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
The quality of word embeddings depends on the input corpora, model architectures, and hyper-parameter settings.Using the state-of-the-art neural embedding tool word2vec and both intrinsic and extrinsic evaluations, we present a comprehensive study of how the quality of embeddings changes according to these features.Apart from identifying the most influential hyper-parameters, we also observe one that creates contradictory results between intrinsic and extrinsic evaluations.Furthermore, we find that bigger corpora do not necessarily produce better biomedical domain word embeddings.We make our evaluation tools and resources as well as the created state-of-the-art word embeddings available under open licenses from https://github.com/cambridgeltl/BioNLP-2016.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 2×
“Following the window approach architecture with word-level likelihood proposed by Collobert and Weston (2008), we apply a tagger built on a simple feed-forward neural network, with a window of five words, one hidden layer of 300 neurons and a hard sigmoid activation, leading to a Softmax output layer.”
BackgroundMethodNLTK: the Natural Language Toolkit
Loper, E., & Bird, S. (2002) · Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - · doi:10.3115/1118108.1118117 · Cited by 3,218
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
(2010) · Cited by 1,906
NLTK: The Natural Language Toolkit
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,844
Improving Distributional Similarity with Lessons Learned from Word Embeddings
Levy, O., Goldberg, Y., & Dagan, I. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00134 · Cited by 1,308
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
Hill, F., Reichart, R., & Korhonen, A. (2015) · Computational Linguistics · doi:10.1162/coli_a_00237 · Cited by 1,175
Self-citationIntroduction to the bio-entity recognition task at JNLPBA
Kim, J.-D., Ohta, T., Tsuruoka, Y., Tateisi, Y., & Collier, N. (2004) · Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications - JNLPBA '04 · doi:10.3115/1567594.1567610 · Cited by 573
Distributional Semantics Resources for Biomedical Text Processing
Pyysalo, S., Ginter, F., Moen, H., Salakoski, T., & Ananiadou, S. (2013) · Research Explorer (The University of Manchester) · Cited by 548
Self-citationOverview of BioCreative II gene mention recognition
Smith, L., Tanabe, L. K., Ando, R. J. N., Kuo, C.-J., Chung, I.-F., Hsu, C.-N., Lin, Y.-S., Klinger, R., Friedrich, C. M., Ganchev, K., et al. (2008) · Genome Biology · doi:10.1186/gb-2008-9-s2-s2 · Cited by 486
Semantic Similarity and Relatedness between Clinical Terms: An Experimental Study.
Pakhomov, S., McInnes, B. T., Adam, T. J., Liu, Y., Pedersen, T., & Melton, G. B. (2010) · PubMed · Cited by 226
Lapesa, G., & Evert, S. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00201 · Cited by 174
Domain and function: A dual-space model of semantic relations and compositions
Turney, P. D. (2012) · Cited by 138
Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
Chiu, B., Korhonen, A., & Pyysalo, S. (2016) · Workshop on Evaluating Vector-Space Representations for NLP · doi:10.18653/v1/w16-2501 · Cited by 130
Self-citationEvaluating distributed word representations for capturing semantics of biomedical concepts
TH, M., Sahu, S. K., & Anand, A. (2015) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/w15-3820 · Cited by 92
GPLSI: Supervised Sentiment Analysis in Twitter using Skipgrams
Fernández, J., Gutiérrez, Y., Gomez, J. M., & Martínez-Barco, P. (2014) · International Workshop on Semantic Evaluation (SemEval) · doi:10.3115/v1/s14-2048 · Cited by 28
Size (and Domain) Matters: Evaluating Semantic Word Space Representations for Biomedical Text
Stenetorp, P., Soyer, H., Pyysalo, S., Ananiadou, S., & Chikayama, T. (2012) · Cited by 22
Self-citation
Cited by
Peng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618
BioWordVec, improving biomedical word embeddings with subword information and MeSH
Zhang, Y., Chen, Q., Yang, Z., Lin, H., & Lu, Z. (2019) · Scientific Data · doi:10.1038/s41597-019-0055-0 · Cited by 455
A Neural Layered Model for Nested Named Entity Recognition
Ju, M., Miwa, M., & Ananiadou, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1131 · Cited by 309
A neural network multi-task learning approach to biomedical named entity recognition
Crichton, G., Pyysalo, S., Chiu, B., & Korhonen, A. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1776-8 · Cited by 229
“Those are however aimed at general domain work and can produce very high out-of-vocabulary rates when used on biomedical texts, thus for this work we used the embed-dings created in [30] which are created from biomedical texts.”
MethodSelf-citation
Metadata
- Authors
- Chiu, B., Crichton, G., Korhonen, A., & Pyysalo, S.
- Venue
- Proceedings of the 15th Workshop on Biomedical Natural Language Processing
- Published
- 2016
- Pages
- 166-174
- OpenAlex
- W2515248967
- Cited by
- 344
- References
- 20
- FWCI
- 25.23
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.