Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Objectives: Identifying all published reports of randomized controlled trials (RCTs) is an important aim, but it requires extensive manual effort to separate RCTs from non-RCTs, even using current machine learning (ML) approaches. We aimed to make this process more efficient via a hybrid approach using both crowdsourcing and ML. Methods: We trained a classifier to discriminate between citations that describe RCTs and those that do not. We then adopted a simple strategy of automatically excluding citations deemed very unlikely to be RCTs by the classifier and deferring to crowdworkers otherwise. Results: Combining ML and crowdsourcing provides a highly sensitive RCT identification strategy (our estimates suggest 95%–99% recall) with substantially less effort (we observed a reduction of around 60%–80%) than relying on manual screening alone. Conclusions: Hybrid crowd-ML strategies warrant further exploration for biomedical curation/annotation tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning for identifying RCTs has already been deployed in Cochrane; Cochrane authors may access this technology via the Cochrane Register of Studies [24].”
From Toward systematic review automation: a practical guide to using machine learning tools in research synthesis (2019) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These include automated classification of articles by study method4,7,8”
From Trialstreamer: A living, automatically updated database of clinical trial reports (2020) · cites this paper as [8]
Experimental — accuracy not yet measuredIn the citing paper
“Some not-for-profit research institutions appear to be highly interested in the development and promotion of SR automatization tools that are made freely available, which may in part be compensating for the funding disadvantage [26, 75]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [75]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 1×
“This is partly why systematic reviews are timeconsuming and expensive to conduct, a problem exacerbated by the rapid growth of the published evidence base.(5) To increase the efficiency of reviewing, Cochrane (a global network of health science researchers) has invested effort into systematically identify RCTs and is…”
BackgroundCHALMERS, I. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26345.x · Cited by 561
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Self-citationSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
Self-citationRetrieving randomized controlled trials from medline : a comparison of 38 published search filters
McKibbon, K. A., Wilczynski, N. L., Haynes, R. B., & for the Hedges Team (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2008.00827.x · Cited by 126
Cohen, A. M., Smalheiser, N. R., McDonagh, M. S., Yu, C., Adams, C. E., Davis, J. M., & Yu, P. S. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu025 · Cited by 78 · cited 2×
“Cohen et al.2 report high areas under the curve for this task, but given the low prevalence of RCTs, any particular threshold used for classification could still yield a substantial number of missed trials or produce an abundance of false positives.”
BackgroundSelf-citationCombining Crowd and Expert Labels Using Decision Theoretic Active Learning
Nguyen, A., Wallace, B., & Lease, M. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13225 · Cited by 73
Self-citationWho Should Label What? Instance Allocation in Multiple Expert Active Learning
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2011) · Proceedings of the 2011 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972818.16 · Cited by 63
Self-citationWieland, L. S., Robinson, K. A., & Dickersin, K. (2012) · BMJ · doi:10.1136/bmj.d7501 · Cited by 34
Cited by
A Systematic Review on Supervised and Unsupervised Machine Learning Algorithms for Data Science
Alloghani, M., Al-Jumeily, D., Mustafina, J., Hussain, A., & Aljaaf, A. J. (2020) · Unsupervised and Semi-Supervised Learning · doi:10.1007/978-3-030-22475-2_1 · Cited by 702
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Self-citationLymphadenectomy for the management of endometrial cancer
Frost, J. A., Webster, K. E., Bryant, A., & Morrison, J. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007585.pub4 · Cited by 430
Living systematic reviews: 2. Combining human and machine effort
Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilt, I., Synnot, A., Turner, T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.011 · Cited by 355
“At the time of writing, there are good classifiers available for RCTs [5], and members of the author team are currently developing classifiers for systematic reviews and economic evaluations; but there are no high-performing classifiers to identify, for example, diagnostic test accuracy studies because training data a…”
BackgroundInfluentialSelf-citationInterventions for increasing fruit and vegetable consumption in children aged five years and under
Hodder, R. K., O'Brien, K. M., Tzelepis, F., Wyse, R. J., & Wolfenden, L. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008552.pub7 · Cited by 311
Thomas, J., McDonald, S., Noel-Storr, A., Shemilt, I., Elliott, J., Mavergames, C., & Marshall, I. J. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.11.003 · Cited by 253
“To date, most automation approaches operate at the level of individual reviews [8,9], rather than addressing structural deficiencies in research curation [10].”
BackgroundSelf-citationNye, B., Li, J. J., Patel, R., Yang, Y., Marshall, I., Nenkova, A., & Wallace, B. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1019 · Cited by 224
Self-citationIntraoperative imaging technology to maximise extent of resection for glioma
Jenkinson, M. D., Barone, D. G., Bryant, A., Vale, L. D., Bulbeck, H., Lawrie, T. A., Hart, M. G., & Watts, C. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012788.pub2 · Cited by 176
Anticoagulation for the long-term treatment of venous thromboembolism in people with cancer
Kahale, L. A., Hakoum, M. B., Tsolakian, I. G., Alturki, F., Matar, C. F., Terrenato, I., Sperati, F., Barba, M., Yosuico, V. E. D., Schünemann, H. J., et al. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006650.pub5 · Cited by 170
Gonçalves-Bradley, D. C., J Maria, A. R., Ricci-Cabello, I., Villanueva, G., Fønhus, M. S., Glenton, C., Lewin, S., Henschke, N., Buckley, B. S., Mehl, G. L., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012927.pub2 · Cited by 145
Biologics for chronic rhinosinusitis
Chong, L., Piromchai, P., Sharp, S., Snidvongs, K., Webster, K. E., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013513.pub3 · Cited by 124
Development of a search filter to identify reports of controlled clinical trials within CINAHL Plus
Glanville, J., Dooley, G., Wisniewski, S., Foxlee, R., & Noel‐Storr, A. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12251 · Cited by 121
Self-citationAkl, E. A., Gunukula, S., Barba, M., Yosuico, V. E. D., van Doormaal, F. F., Kuipers, S., Middeldorp, S., Dickinson, H. O., Bryant, A., & Schünemann, H. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006652.pub3 · Cited by 118
Steps in undertaking meta-analysis and addressing heterogeneity in meta-analysis
Gandhi, A. P., Shamim, M. A., & Padhi, B. K. (2023) · The Evidence · doi:10.61505/evidence.2023.1.1.7 · Cited by 115
Decision-support tools via mobile devices to improve quality of care in primary healthcare settings
Agarwal, S., Glenton, C., Tamrat, T., Henschke, N., Maayan, N., Fønhus, M. S., Mehl, G. L., & Lewin, S. A. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012944.pub2 · Cited by 105
Parenteral anticoagulation in ambulatory patients with cancer
Akl, E. A., Kahale, L. A., Hakoum, M. B., Matar, C. F., Sperati, F., Barba, M., Yosuico, V. E., Terrenato, I., Synnot, A., & Schünemann, H. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006652.pub5 · Cited by 102
A survey on clinical natural language processing in the United Kingdom from 2007 to 2022
Wu, H., Wang, M., Wu, J., Francis, F., Chang, Y.-H., Shavick, A., Dong, H., Poon, M. T. C., Fitzpatrick, N., Levine, A. P., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00730-6 · Cited by 101
Interventions for increasing fruit and vegetable consumption in children aged five years and under
Hodder, R. K., O’Brien, K. M., Stacey, F. G., Wyse, R., Clinton‐McHarg, T., Tzelepis, F., James, E. L., Bartlem, K. M., Nathan, N., Sutherland, R., et al. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008552.pub5 · Cited by 98
Interventions for increasing fruit and vegetable consumption in children aged five years and under
Hodder, R. K., O’Brien, K. M., Stacey, F. G., Tzelepis, F., Wyse, R., Bartlem, K. M., Sutherland, R., James, E. L., Barnes, C., & Wolfenden, L. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008552.pub6 · Cited by 91
Biologics for chronic rhinosinusitis
Chong, L., Piromchai, P., Sharp, S., Snidvongs, K., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013513.pub2 · Cited by 89
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
“Some not-for-profit research institutions appear to be highly interested in the development and promotion of SR automatization tools that are made freely available, which may in part be compensating for the funding disadvantage [26, 75].”
BackgroundTrialstreamer: A living, automatically updated database of clinical trial reports
Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa163 · Cited by 83
Self-citationSystemic pharmacological interventions for Ménière’s disease
Webster, K. E., Galbraith, K., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., & Burton, M. J. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015171.pub2 · Cited by 78
Interventions for increasing fruit and vegetable consumption in children aged five years and under
Hodder, R. K., Stacey, F. G., O’Brien, K. M., Wyse, R., Clinton‐McHarg, T., Tzelepis, F., James, E. L., Bartlem, K. M., Nathan, N., Sutherland, R., et al. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008552.pub4 · Cited by 73
Interventions for the treatment of persistent post-COVID-19 olfactory dysfunction
O’Byrne, L. M., Webster, K. E., MacKeith, S. A. C., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013876.pub2 · Cited by 68
Which are the most sensitive search filters to identify randomized controlled trials in MEDLINE?
Glanville, J., Kotas, E., Featherstone, R. M., & Dooley, G. (2020) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2020.912 · Cited by 66
Santos, Á. O. D., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 64
Intratympanic corticosteroids for Ménière’s disease
Webster, K. E., Lee, A. Y. K., Galbraith, K., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015245.pub2 · Cited by 63
Interventions for increasing fruit and vegetable consumption in children aged five years and under
Hodder, R. K., Stacey, F. G., Wyse, R., O’Brien, K. M., Clinton‐McHarg, T., Tzelepis, F., Nathan, N., James, E. L., Bartlem, K. M., Sutherland, R., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008552.pub3 · Cited by 57
Nama, N., Hennawy, M., Barrowman, N., O’Hearn, K., Sampson, M., & McNally, J. D. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01632-6 · Cited by 53
Intraoperative imaging technology to maximise extent of resection for glioma
Jenkinson, M. D., Barone, D. G., Hart, M. G., Bryant, A., Lawrie, T. A., & Watts, C. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012788 · Cited by 52
Intratympanic gentamicin for Ménière’s disease
Webster, K. E., Galbraith, K., Lee, A. Y. K., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015246.pub2 · Cited by 50
Crowdsourcing the Citation Screening Process for Systematic Reviews: Validation Study
Nama, N., Sampson, M., Barrowman, N., Sandarage, R., Menon, K., Macartney, G., Murto, K., Vaccani, J.-P., Katz, S., Zemek, R., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/12953 · Cited by 50
Counotte, M. J., Egli-Gany, D., Riesen, M., Abraha, M., Porgo, T. V., Wang, J., & Low, N. N. (2018) · F1000Research · doi:10.12688/f1000research.13704.1 · Cited by 49
Surgical interventions for Ménière’s disease
Lee, A. Y. K., Webster, K. E., George, B., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015249.pub2 · Cited by 48
Citizen Science: Crowdsourcing for Research
Lichten, C. A., Ioppolo, B., d’Angelo, C., Simmons, R. K., & Jones, M. M. (2018) · Cited by 48
Lifestyle and dietary interventions for Ménière’s disease
Webster, K. E., George, B., Lee, A. Y. K., Galbraith, K., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Murdin, L. J., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015244.pub2 · Cited by 46
Interventions for the treatment of persistent post-COVID-19 olfactory dysfunction
O’Byrne, L. M., Webster, K. E., MacKeith, S. A. C., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013876.pub3 · Cited by 45
Ventilation tubes (grommets) for otitis media with effusion (OME) in children
MacKeith, S. A. C., Mulvaney, C. A., Galbraith, K., Webster, K. E., Connolly, R., Paing, A., Marom, T., Daniel, M., Venekamp, R. P., Rovers, M. M., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015215.pub2 · Cited by 44
Interventions for the prevention of persistent post-COVID-19 olfactory dysfunction
Webster, K. E., O’Byrne, L. M., MacKeith, S. A. C., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013877.pub2 · Cited by 41
Machine Learning Methods for Systematic Reviews:: A Rapid Scoping Review.
Roth, S. C., & Wermer‐Colan, A. (2023) · PubMed Central · doi:10.32481/djph.2023.11.008 · Cited by 41
Kahale, L. A., Hakoum, M. B., Tsolakian, I. G., Matar, C. F., Barba, M., Yosuico, V. E. D., Terrenato, I., Sperati, F., Schünemann, H. J., & Akl, E. A. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006466.pub6 · Cited by 38
Screening PubMed abstracts: is class imbalance always a challenge to machine learning?
Lanera, C., Berchialla, P., Sharma, A., Minto, C., Gregori, D., & Baldi, I. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1245-8 · Cited by 38
Scotti, K. L., Young, S. M. N., Gainey, M. A., & Lan, H. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70046 · Cited by 37
Interventions for increasing fruit and vegetable consumption in children aged five years and under
Hodder, R. K., O’Brien, K. M., Wyse, R., Tzelepis, F., Yoong, S. L., Stacey, F. G., & Wolfenden, L. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008552.pub8 · Cited by 37
Interventions for improving health literacy in migrants
Baumeister, A., Aldin, A., Chakraverty, D., Hübner, C., Adams, A., Monsef, I., Skoetz, N., Kalbe, E., & Woopen, C. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013303.pub2 · Cited by 36
Magee, L., Goldsmith, L. P., Chaudhry, U. A. R., Donin, A. S., Wahlich, C., Stovold, E. M., Nightingale, C. M., Rudnicka, A. R., & Owen, C. G. (2022) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2022.3172 · Cited by 35
Hazlewood, G. S., Whittle, S., Kamso, M. M., Akl, E. A., Wells, G. A., Tugwell, P., Thomas, M., Lee, C., Ejaredar, M., Choudhary, D., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013562 · Cited by 35
Interventions for the prevention of persistent post-COVID-19 olfactory dysfunction
Webster, K. E., MacKeith, S. A. C., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013877 · Cited by 35
Adenoidectomy for otitis media with effusion (OME) in children
MacKeith, S. A. C., Mulvaney, C. A., Galbraith, K., Webster, K. E., Paing, A., Connolly, R., Marom, T., Daniel, M., Venekamp, R. P., & Schilder, A. G. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015252.pub2 · Cited by 34
Sivapalarajah, S., Krishnakumar, M., Bickerstaffe, H. C., Chan, Y., Clarkson, J., Hampden‐Martin, A., Mirza, A., Tanti, M., Marson, A. G., Pirmohamed, M., et al. (2018) · Epilepsia · doi:10.1111/epi.13994 · Cited by 32
Decision-support tools via mobile devices to improve quality of care in primary healthcare settings
Agarwal, S., Tamrat, T., Glenton, C., Lewin, S. A., Henschke, N., Maayan, N., Fønhus, M. S., & Mehl, G. L. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012944 · Cited by 31
Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation
Robinson, A., Thorne, W., Wu, B. P., Pandor, A., Essat, M., Stevenson, M., & Song, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.06610 · Cited by 31
Antibiotics for otitis media with effusion (OME) in children
Mulvaney, C. A., Galbraith, K., Webster, K. E., Rana, M., Connolly, R., Marom, T., Daniel, M., Venekamp, R. P., Schilder, A. G., & MacKeith, S. A. C. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015254.pub2 · Cited by 30
Technological advances in preclinical meta-research
Bannach‐Brown, A., Hair, K., Bahor, Z., Soliman, N., Macleod, M. R., & Liao, J. (2021) · BMJ Open Science · doi:10.1136/bmjos-2020-100131 · Cited by 28
Towards a Hybrid Human-Computer Scientific Information Extraction Pipeline
Tchoua, R., Chard, K., Audus, D. J., Ward, L. T., Lequieu, J. P., Pablo, J., & Foster, I. T. (2017) · IEEE International Conference on eScience · doi:10.1109/escience.2017.23 · Cited by 28
Martin, P. B., Surian, D., Bashir, R., Bourgeois, F. T., & Dunn, A. G. (2019) · JAMIA Open · doi:10.1093/jamiaopen/ooy062 · Cited by 27
Prospective approaches to accumulating evidence
Thomas, J., Askie, L. M., Berlin, J. A., Elliott, J. H., Ghersi, D., Simmonds, M., Takwoingi, Y., Tierney, J. F., & Higgins, J. P. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch22 · Cited by 26
Self-citationPositive pressure therapy for Ménière’s disease
Webster, K. E., George, B., Galbraith, K., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015248.pub2 · Cited by 25
Understanding the Impact of Text Highlighting in Crowdsourcing Tasks
Ramírez, J., Báez, M., Casati, F., & Benatallah, B. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5268 · Cited by 25
Interventions for the prevention of persistent post-COVID-19 olfactory dysfunction
Webster, K. E., O’Byrne, L. M., MacKeith, S. A. C., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013877.pub3 · Cited by 24
Crowd-based Multi-Predicate Screening of Papers in Literature Reviews
Krivosheev, E., Casati, F., & Benatallah, B. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186036 · Cited by 23
Karagiannis, T., Andreadis, P., Manolopoulos, A., Malandris, K., Avgerinos, I., Karagianni, A., & Τσάπας, Α. (2019) · Diabetic Medicine · doi:10.1111/dme.13939 · Cited by 22
Autoinflation for otitis media with effusion (OME) in children
Webster, K. E., Mulvaney, C. A., Galbraith, K., Rana, M., Marom, T., Daniel, M., Venekamp, R. P., Schilder, A. G., & MacKeith, S. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015253.pub2 · Cited by 21
Methotrexate for juvenile idiopathic arthritis
Tan, J., Renton, W. D., Whittle, S., Takken, T., Johnston, R. V., Tiller, G., Munro, J. E., & Buchbinder, R. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003129.pub2 · Cited by 21
Why Cochrane should prioritise sharing data
Shokraneh, F., Adams, C. E., Clarke, M., Amato, L., Bastian, H., Beller, E., Brassey, J., Buchbinder, R., Davoli, M., Del Mar, C., et al. (2018) · BMJ · doi:10.1136/bmj.k3229 · Cited by 21
Kahale, L. A., Matar, C. F., Tsolakian, I. G., Hakoum, M. B., Yosuico, V. E. D., Terrenato, I., Sperati, F., Barba, M., Hicks, L. K., Schünemann, H. J., et al. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd014739 · Cited by 21
Interventions for the treatment of persistent post-COVID-19 olfactory dysfunction
Webster, K. E., MacKeith, S. A. C., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013876 · Cited by 21
Biologics for chronic rhinosinusitis
Chong, L., Piromchai, P., Sharp, S., Snidvongs, K., Philpott, C. M., Hopkins, C. L., & Burton, M. J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013513 · Cited by 20
Topical and oral steroids for otitis media with effusion (OME) in children
Mulvaney, C. A., Galbraith, K., Webster, K. E., Rana, M., Connolly, R., Tudor-Green, B., Marom, T., Daniel, M., Venekamp, R. P., Schilder, A. G., et al. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015255.pub2 · Cited by 18
Schneider, J., Hoang, L., Kansara, Y., Cohen, A. M., & Smalheiser, N. R. (2022) · JAMIA Open · doi:10.1093/jamiaopen/ooac015 · Cited by 18
Self-citationLiving evidence and adaptive policy: perfect partners?
Turner, T., Lavis, J. N., Grimshaw, J., Green, S., & Elliott, J. H. (2023) · Health Research Policy and Systems · doi:10.1186/s12961-023-01085-4 · Cited by 18
Stem cell injections for osteoarthritis of the knee
Whittle, S., Johnston, R. V., McDonald, S. J., Worthley, D. L., Campbell, T. M., & Buchbinder, R. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013342 · Cited by 18
Combining Crowd and Machines for Multi-predicate Item Screening
Krivosheev, E., Casati, F., Báez, M., & Benatallah, B. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274366 · Cited by 18
Working with a new kind of team: harnessing the wisdom of the crowd in trial identification
Noel-Storr, A. H. (2019) · EFSA Journal · doi:10.2903/j.efsa.2019.e170715 · Cited by 17
Self-citationGonçalves, D. C., Buckley, B. S., Fønhus, M. S., Glenton, C., Henschke, N., Lewin, S. A., Maayan, N., Mehl, G. L., Tamrat, T., & Shepperd, S. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012927 · Cited by 16
Cohen, A. M., Schneider, J., Fu, Y., McDonagh, M. S., Das, P., Holt, A. W., & Smalheiser, N. R. (2021) · medRxiv · doi:10.1101/2021.07.13.21260468 · Cited by 16
Self-citationResistance training for fatigue in people with cancer
Ernst, M., Wagner, C., Oeser, A., Messer, S., Wender, A., Cryns, N., Bröckelmann, P. J., Holtkamp, U., Baumann, F. T., Wiskemann, J., et al. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015518 · Cited by 16
Machine learning from crowds: A systematic review of its applications
Rodrigo, E. G., Aledo, J. A., & Gámez, J. A. (2018) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1288 · Cited by 16
Pianta, M. J., Makrai, E., Verspoor, K. M., Cohn, T. A., & Downie, L. E. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.07.015 · Cited by 15
Non-pharmacological interventions for prophylaxis of vestibular migraine
Webster, K. E., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., & Burton, M. J. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015321 · Cited by 15
Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study
Noel-Storr, A. H., Redmond, P., Lamé, G., Liberati, E. G., Kelly, S., Miller, L., Dooley, G., Paterson, A., & Burt, J. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01271-4 · Cited by 15
Self-citationSystemic pharmacological interventions for Ménière’s disease
Webster, K. E., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., & Burton, M. J. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015171 · Cited by 14
Non-pharmacological interventions for persistent postural-perceptual dizziness (PPPD)
Webster, K. E., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., & Burton, M. J. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015333 · Cited by 14
Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks
Ramírez, J., Báez, M., Casati, F., & Benatallah, B. (2019) · BMC Research Notes · doi:10.1186/s13104-019-4858-z · Cited by 13
O’Hearn, K., MacDonald, C., Tsampalieros, A. K., Kadota, L., Sandarage, R. V., Jayawarden, S. K., Datko, M., Reynolds, J. M., Bui, T., Sultan, S. A., et al. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01335-5 · Cited by 13
CrowdRev: A platform for Crowd-based Screening of Literature ReviewsPreprint
Ramírez, J., Krivosheev, E., Báez, M., Casati, F., & Benatallah, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.12376 · Cited by 13
Uthman, O. A., Court, R. A., Enderby, J., Al-Khudairy, L., Nduka, C. U., Mistry, H., Meléndez‐Torres, G. J., Taylor‐Phillips, S., & Clarke, A. (2022) · Health Technology Assessment · doi:10.3310/udir6682 · Cited by 12
Gonçalves, D. C., Buckley, B. S., Fønhus, M. S., Glenton, C., Henschke, N., Lewin, S. A., Maayan, N., Mehl, G. L., Tamrat, T., & Shepperd, S. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012928 · Cited by 11
Lee, C., Thomas, M., Ejaredar, M., Kassam, A., Whittle, S., Buchbinder, R., Tugwell, P., Wells, G. A., Pardo, J. P., & Hazlewood, G. S. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.03.019 · Cited by 11
Publication Type Tagging using Transformer Models and Multi-Label ClassificationPreprint
Menke, J. D., Kilicoglu, H., & Smalheiser, N. R. (2025) · medRxiv · doi:10.1101/2025.03.06.25323516 · Cited by 11
Self-citationEvaluating the Accuracy of scite, a Smart Citation Index
Bakker, C., Theis-Mahon, N., & Brown, S. J. (2023) · Hypothesis: Research Journal for Health Information Professionals · doi:10.18060/26528 · Cited by 10
Vaccines for preventing infections in adults with haematological malignancies
Zorger, A.-M., Hirsch, C., Baumann, M., Feldmann, M., Bröckelmann, P. J., Mellinghoff, S. C., Monsef, I., Skoetz, N., & Kreuzberger, N. (2025) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015530.pub2 · Cited by 10
Cardiovascular training for fatigue in people with cancer
Wagner, C., Ernst, M., Cryns, N., Oeser, A., Messer, S., Wender, A., Wiskemann, J., Baumann, F. T., Monsef, I., Bröckelmann, P. J., et al. (2025) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015517 · Cited by 10
Pharmacological interventions for persistent postural-perceptual dizziness (PPPD)
Webster, K. E., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., & Burton, M. J. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015188 · Cited by 10
Kim, J., Kim, J., Lee, A., & Kim, J. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283342 · Cited by 10
Adenoidectomy for otitis media with effusion (OME) in children
MacKeith, S. A. C., Mulvaney, C. A., Galbraith, K., Marom, T., Daniel, M., Venekamp, R. P., & Schilder, A. G. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015252 · Cited by 10
Intratympanic corticosteroids for Ménière’s disease
Webster, K. E., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Ray, J., van Vugt, V. A., Westerberg, B. D., & Burton, M. J. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015245 · Cited by 10
Lifestyle and dietary interventions for Ménière’s disease
Webster, K. E., Harrington-Benton, N. A., Judd, O., Kaski, D. N., Maarsingh, O. R., MacKeith, S. A. C., Murdin, L. J., Ray, J., van Vugt, V. A., & Burton, M. J. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015244 · Cited by 10
Identifying Diagnostic Test Accuracy Publications using a Deep Model.
Singh, G., Marshall, I. J., Thomas, J., & Wallace, B. (2017) · UCL Discovery (University College London) · Cited by 10
Self-citation
Metadata
- Authors
- Wallace, B. C., Noel-Storr, A., Marshall, I. J., Cohen, A. M., Smalheiser, N. R., & Thomas, J.
- Affiliations
- Northeastern University (US)
- University of Oxford (GB)
- King's College London (GB)
- Oregon Health & Science University (US)
- University of Illinois Chicago (US)
- University College London (GB)
- Corresponding author
- Wallace, Byron C
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-05-25
- Volume
- 24
- Issue
- 6
- Pages
- 1165-1168
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC5975623
- OpenAlex
- W2619694921
- Semantic Scholar
- 92e5852ecc8076460b86c83c06e9374e38b054d8
- Cited by
- 186
- References
- 10
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 7 · 2023: 21 · 2022: 31 · 2021: 43 · 2020: 11 · 2019: 20 · 2018: 22 · 2017: 8
- FWCI
- 14.78
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Research Portal (King's College London) (publishedVersion, cc-by-nc)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- National Institutes of Health
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- randomized controlled trials, citation classification, crowdsourcing, hybrid machine learning, literature screening, citation annotation
- MeSH terms
- Machine Learning, Natural Language Processing, Review Literature as Topic, ROC Curve, Randomized Controlled Trials as Topic, Databases, Bibliographic, Information Storage and Retrieval, Biomedical Research, Support Vector Machine, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.