A Neural Candidate-Selector Architecture for Automatic Structured Clinical Text Annotation.
Open accessAbstract
We consider the task of automatically annotating free texts describing clinical trials with concepts from a controlled, structured medical vocabulary. Specifically, we aim to build a model to infer distinct sets of (ontological) concepts describing complementary clinically salient aspects of the underlying trials: the populations enrolled, the interventions administered and the outcomes measured, i.e., the PICO elements. This important practical problem poses a few key challenges. One issue is that the output space is vast, because the vocabulary comprises many unique concepts. Compounding this problem, annotated data in this domain is expensive to collect and hence sparse. Furthermore, the outputs (sets of concepts for each PICO element) are correlated: specific populations (e.g., diabetics) will render certain intervention concepts likely (insulin therapy) while effectively precluding others (radiation therapy). Such correlations should be exploited. We propose a novel neural model that addresses these challenges. We introduce a Candidate-Selector architecture in which the model considers setes of candidate concepts for PICO elements, and assesses their plausibility conditioned on the input text to be annotated. This relies on a 'candidate set' generator, which may be learned or relies on heuristics. A conditional discriminative neural model then jointly selects candidate concepts, given the input text. We compare the predictive performance of our approach to strong baselines, and show that it outperforms them. Finally, we perform a qualitative evaluation of the generated annotations by asking domain experts to assess their quality.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Publications in the neural and deep-learning domain described approaches such as early stopping, dropout, L2-regularisation, or weight decay.
“Publications in the neural and deep-learning domain described approaches such as early stopping, dropout, L2-regularisation, or weight decay. 59,96,106”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [59]
Variations in style between structured and unstructured abstracts and description lengths and detail can cause inconsistent results in data extraction.
“In particular, variations in style between structured and unstructured abstracts 65,78 and the description lengths and detail 59,79 can cause inconsistent results in the data extraction”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [59]
Another type of relaxation for standard metrics is a distance relaxation when normalising entities into concepts in MeSH or UMLS, to allow N hops between predicted and target concepts.
“Another type of relaxation for standard metrics is a distance relaxation when normalising entities into concepts in medical subject headings (MesH) or unified medical language system (UMLS), to allow N hops between predicted and target concepts. 59”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
A small number of publications went one step further to normalise to actual concepts and including data sources such as UMLS.
“and a small number of publications went one step further to normalise to actual concepts and including data sources such as UMLS (Unified Medical Language System). 35 , 39 , 59 , 73 , 85”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
Ten included publications described using PDF files as input.
“Ten included publications described using PDF files as input. 44 , 46 , 59 , 68 , 75 , 81 , 86 , 158 , 161 , 173”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
Texts and metadata from Cochrane, either provided in full or retrieved via PubMed, were used in five publications.
“Texts and metadata from Cochrane, either provided in full or retrieved via PubMed, were used in five publications. 57 , 59 , 68 , 75 , 86”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
Length of input text influences classification scores.
“Length of input text 59 and position of a sentence within a paragraph or abstract, e.g. up to 10% lower classification scores for certain sentence combinations in unstructured abstracts, were shown in several publications. 46 , 66 , 102”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
Publications in the neural and deep-learning domain described approaches such as early stopping, dropout, L2-regularisation, or weight decay.
“Publications in the neural and deep-learning domain described approaches such as early stopping, dropout, L2-regularisation, or weight decay. 59 , 96 , 106”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
Variations in style between structured and unstructured abstracts and the description lengths and detail can cause inconsistent results in data extraction, for example by not detecting information correctly or extracting unexpected information.
“In particular, variations in style between structured and unstructured abstracts 65 , 78 and the description lengths and detail 59 , 79 can cause inconsistent results in the data extraction, for example by not detecting information correctly or extracting unexpected information.”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
The size of the label space must be considered when comparing scores, as models that normalise to specific concepts rather than detecting entities tend to have lower precision, recall, and F1 scores.
“Another caveat mentioned by Refs. 59, 85 is that the size of the label space must be considered when comparing scores, as models that normalise to specific concepts rather than detecting entities tend to have lower precision, recall, and F1 scores.”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
For models like CNNs that assign specific concepts, unseen entities are not defined in the output label space.
“This also applies to models like CNNs that assign specific concepts, where unseen entities are not defined in the output label space. 59”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,882 · see citing works
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · see citing works
An overview of MetaMap: historical perspective and recent advances
Aronson, A. R., & Lang, F.-M. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.002733 · Cited by 1,519 · see citing works
Support vector machine learning for interdependent and structured output spaces
Tsochantaridis, I., Hofmann, T., Joachims, T., & Altun, Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015341 · Cited by 1,493 · see citing works
Frustratingly Easy Domain AdaptationPreprint
Daumé, H. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0907.1815 · Cited by 1,471 · see citing works
A Kernel Method for Multi-Labelled Classification
Elisseeff, A., & Weston, J. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0092 · Cited by 1,388 · see citing works
Convolutional Neural Networks for Sentence ClassificationPreprint
Kim, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5882 · Cited by 1,112 · see citing works
Multilabel classification via calibrated label ranking
Fürnkranz, J., Hüllermeier, E., Mencía, E. L., & Brinker, K. (2008) · Machine Learning · doi:10.1007/s10994-008-5064-8 · Cited by 914 · see citing works
Learning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879 · see citing works
(2016) · PubMed · Cited by 576 · see citing works
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317 · see citing works
Self-citationNatural Language Processing (Almost) from Scratch
CollobertRonan, WestonJason, BottouLéon, KarlenMichael, KavukcuogluKoray, & KuksaPavel (2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078186 · Cited by 309 · see citing works
Frontiers of biomedical text mining: current progress
Zweigenbaum, P., Demner‐Fushman, D., Yu, H. Q., & Cohen, K. B. (2007) · Briefings in Bioinformatics · doi:10.1093/bib/bbm045 · Cited by 282 · see citing works
Rationale-Augmented Convolutional Neural Networks for Text Classification
Zhang, Y., Marshall, I., & Wallace, B. C. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1076 · Cited by 249 · see citing works
Self-citationDeep Computational Phenotyping
Che, Z., Kale, D. C., Li, W., Bahadori, M. T., & Liu, Y. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2783365 · Cited by 247 · see citing works
Feature-aware Label Space Dimension Reduction for Multi-label Classification
Chen, Y.-N., & Lin, H.-T. (2012) · neural information processing systems · Cited by 217 · see citing works
ExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · see citing works
Zhang, Y., & Wallace, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03820 · Cited by 153 · see citing works
Self-citationEfficient Multi-label Classification with Many Labels
Bi, W., & Kwok, J. T.-Y. (2013) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 137 · see citing works
(2016) · PubMed · Cited by 134 · see citing works
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131 · see citing works
Linear dimensionality reduction for multi-label classification
Ji, S., & Ye, J. (2009) · Cited by 87 · see citing works
Machine learning to assist risk-of-bias assessments in systematic reviews
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv306 · Cited by 65 · see citing works
Automating Risk of Bias Assessment for Clinical Trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2431314 · Cited by 59 · see citing works
Self-citationElhadad, N., & Demner‐Fushman, D. (2016) · Yearbook of Medical Informatics · doi:10.15265/iy-2016-017 · Cited by 52 · see citing works
Automatic Summarization of Results from Clinical Trials
Summerscales, R. L., Argamon, S., Bai, S., Hupert, J., & Schwartz, A. (2011) · 2011 IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm.2011.72 · Cited by 49 · see citing works
Improving Medical Information Retrieval with PICO Element Detection.
Boudin, F., Shi, L., & Nie, J.-Y. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-12275-0_8 · Cited by 21 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- [9]Choi Edward Bahadori Mohammad Taha Sun Jimeng Kulas Joshua Schuetz Andy Stewart Walter 2016 RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism Advances in Neural Information Processing Systems 29 Lee DD Sugiyama M Luxburg UV Guyon I Garnett R Curran Associates, Inc. 3504 3512
- [16]Ioannis Katakis Grigorios Tsoumakas Ioannis Vlahavas 2008 Multilabel text classification for automated tag suggestion ECML PKDD discovery challenge 75 2008
- [22]Andrew McCallum 1999 Multi-label text classification with a mixture model trained by EM AAAI workshop on text learning 1 7
- [25]Perozzi Bryan Al-Rfou Rami Skiena Steven 2014 Deepwalk: Online learning of social representations Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining ACM 701 710
- [27]Srivastava Nitish Hinton Geoffrey E Krizhevsky Alex Sutskever Ilya Salakhutdinov Ruslan 2014 Dropout: a simple way to prevent neural networks from overfitting Journal of Machine Learning Research 15 1 (2014) 1929 1958
- [30]Tsoumakas Grigorios Katakis Ioannis Vlahavas Ioannis 2008 Effective and efficient multilabel classification in domains with large number of labels Proc ECML/PKDD Workshop on Mining Multidimensional Data 30 44
Cited by
Sorted by citations
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106 · see citing works
Self-citationA survey on clinical natural language processing in the United Kingdom from 2007 to 2022
Wu, H., Wang, M., Wu, J., Francis, F., Chang, Y.-H., Shavick, A., Dong, H., Poon, M. T. C., Fitzpatrick, N., Levine, A. P., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00730-6 · Cited by 101 · see citing works
Improving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62 · see citing works
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27 · see citing works
“Eight included publications described using PDF files as input.(29,30,34,38,48,55,59,60) 3.”
InfluentialSelf-citation
- Authors
- Singh, G., Marshall, I. J., Thomas, J., Shawe-Taylor, J., & Wallace, B. C.
- Affiliations
- University College London (GB)
- King's College London (GB)
- Northeastern University (US)
- Venue
- Proceedings of the 2017 ACM on Conference on Information and Knowledge Management
- Publisher
- ACM
- Published
- 2017-11-06
- Volume
- 2017
- Pages
- 1519-1528
- ISSN
- 2155-0751 (ISSN-L)
- PMC
- PMC5752318
- OpenAlex
- W2767908683
- Cited by
- 11
- References
- 29
- Citations by year
- 2025: 2 · 2024: 1 · 2023: 2 · 2022: 1 · 2021: 2 · 2020: 2 · 2019: 1
- FWCI
- 3.06
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Research Portal (King's College London) (submittedVersion, other-oa)
- Funders
- Cochrane 'Transform' (001)
- NLM (R01LM012086)
- MRC (UK) (MR/N015185/1)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- clinical text annotation, clinical trials, controlled medical vocabulary, concept selection, clinical natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
