Scaling drug indication curation through crowdsourcing
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Motivated by the high cost of human curation of biological databases, there is an increasing interest in using computational approaches to assist human curators and accelerate the manual curation process. Towards the goal of cataloging drug indications from FDA drug labels, we recently developed LabeledIn, a human-curated drug indication resource for 250 clinical drugs. Its development required over 40 h of human effort across 20 weeks, despite using well-defined annotation guidelines. In this study, we aim to investigate the feasibility of scaling drug indication annotation through a crowdsourcing technique where an unknown network of workers can be recruited through the technical environment of Amazon Mechanical Turk (MTurk). To translate the expert-curation task of cataloging indications into human intelligence tasks (HITs) suitable for the average workers on MTurk, we first simplify the complex task such that each HIT only involves a worker making a binary judgment of whether a highlighted disease, in context of a given drug label, is an indication. In addition, this study is novel in the crowdsourcing interface design where the annotation guidelines are encoded into user options. For evaluation, we assess the ability of our proposed method to achieve high-quality annotations in a time-efficient and cost-effective manner. We posted over 3000 HITs drawn from 706 drug labels on MTurk. Within 8 h of posting, we collected 18 775 judgments from 74 workers, and achieved an aggregated accuracy of 96% on 450 control HITs (where gold-standard answers are known), at a cost of $1.75 per drug label. On the basis of these results, we conclude that our crowdsourcing approach not only results in significant cost and time saving, but also leads to accuracy comparable to that of domain experts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Data-driven prediction of drug effects and interactions
Tatonetti, N. P., Ye, P. P., Daneshjou, R., & Altman, R. B. (2012) · Science Translational Medicine · doi:10.1126/scitranslmed.3003377 · Cited by 936
Understanding PubMed(R) user search behavior through log analysis
Islamaj Dogan, R., Murray, G. C., Neveol, A., & Lu, Z. (2009) · Database · doi:10.1093/database/bap018 · Cited by 265 · cited 1×
“Over the years, its definition has evolved to include many activities, such as analysis of search logs (14), editing wikis in biology (29), etc.”
BackgroundSelf-citationDevelopment and evaluation of an ensemble resource linking medications to their indications
Wei, W.-Q., Cronin, R. M., Xu, H., Lasko, T. A., Bastarache, L., & Denny, J. C. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001431 · Cited by 106 · cited 1×
“Thus our study has more resemblance to (23, 24) than (22).”
ResultZhai, H., Lingren, T., Deleger, L., Li, Q., Kaiser, M., Stoutenborough, L., & Solti, I. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2426 · Cited by 73 · cited 2×
“More specifically, we posted 3004 HITs corresponding to 706 new drug labels on the MTurk platform, and also injected 450 controls HITs drawn from previously annotated 87 drug labels to assess the turkers’ performances (35).”
BackgroundMethod
Cited by
Systematic integration of biomedical knowledge prioritizes drugs for repurposing
Himmelstein, D. S., Lizee, A., Hessler, C., Brueggeman, L., Chen, S. L., Hadley, D., Green, A., Khankhanian, P., & Baranzini, S. E. (2017) · eLife · doi:10.7554/elife.26726 · Cited by 666
“…( Maglott et al., 2011 ; Himmelstein et al., 2015h ; Himmelstein, 2016l ) LabeledIn CtD, CpD custom 1 RRID:SCR_015667 ( Khare et al., 2014 ; Khare et al., 2015 ; Himmelstein and Khare, 2015s ) PW, GpPW 1 RRID:SCR_006866 ( Schaefer et al., 2009 ; Pico and Himmelstein, 2015 ; Himmelstein and…”
BackgroundA survey of current trends in computational drug repositioning
Li, J., Zheng, S., Chen, B., Butte, A. J., Swamidass, S. J., & Lu, Z. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbv020 · Cited by 555
Self-citationMortensen, K., & Hughes, T. L. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4246-0 · Cited by 441
Information retrieval and text mining technologies for chemistry
Krallinger, M., Rabal, O., Lourenço, A., Oyarzabal, J., & Valencia, A. (2017) · Chemical Reviews · doi:10.1021/acs.chemrev.6b00851 · Cited by 281
Extraction and analysis of signatures from the Gene Expression Omnibus by the crowd
Wang, Z., Monteiro, C. D., Jagodnik, K. M., Fernandez, N. F., Gundersen, G. W., Rouillard, A. D., Jenkins, S. L., Feldmann, A. S., Hu, K. S., McDermott, M. G., et al. (2016) · Nature Communications · doi:10.1038/ncomms12846 · Cited by 256
Metadata
- Authors
- Khare, R., Burger, J. D., Aberdeen, J. S., Tresner-Kirsch, D. W., Corrales, T. J., Hirchman, L., & Lu, Z.
- Venue
- Database
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-01-01
- Volume
- 2015
- ISSN
- 1758-0463 (ISSN-L)
- OpenAlex
- W2013130010
- Cited by
- 39
- References
- 5
- FWCI
- 2.83
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.