Using text mining for study identification in systematic reviews: a systematic review of current approaches
Open accessThe DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way for inclusion in systematic reviews both complex and time consuming. Text mining has been offered as a potential solution: through automating some of the screening process, reviewer time can be saved. The evidence base around the use of text mining for screening has not yet been pulled together systematically; this systematic review fills that research gap. Focusing mainly on non-technical issues, the review aims to increase awareness of the potential of these technologies and promote further collaborative research between the computer science and systematic review communities. METHODS: Five research questions led our review: what is the state of the evidence base; how has workload reduction been evaluated; what are the purposes of semi-automation and how effective are they; how have key contextual problems of applying text mining to the systematic review field been addressed; and what challenges to implementation have emerged? We answered these questions using standard systematic review methods: systematic and exhaustive searching, quality-assured data extraction and a narrative synthesis to synthesise findings. RESULTS: The evidence base is active and diverse; there is almost no replication between studies or collaboration between research teams and, whilst it is difficult to establish any overall conclusions about best approaches, it is clear that efficiencies and reductions in workload are potentially achievable. On the whole, most suggested that a saving in workload of between 30% and 70% might be possible, though sometimes the saving in workload is accompanied by the loss of 5% of relevant studies (i.e. a 95% recall). CONCLUSIONS: Using text mining to prioritise the order in which items are screened should be considered safe and ready for use in 'live' reviews. The use of text mining as a 'second screener' may also be used cautiously. The use of text mining to eliminate studies automatically should be considered promising, but not yet fully proven. In highly technical/clinical areas, it may be used with a high degree of confidence; but more developmental and evaluative work is needed in other disciplines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They can be calibrated to achieve a given level of recall, thus enabling reviewers to implement screening rules, such as eliminating records or replacing double with single screening.”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, there is a risk of erroneously excluding relevant studies because of uncertainty about when it is safe to stop screening; the balance between efficiency gains and risk tolerance will be review-specific52 53”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Because there are fewer examples of relevant than irrelevant publications to train on, the class imbalance causes the classifier to miss relevant publications [32].”
From Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records (2023) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As mentioned above, these algorithms/software are more and more used for article screening in systematic reviews [53, 55, 56]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent advances in natural language processing, text mining and machine learning have produced new algorithms that can accurately mimic human endeavour in systematic review activity, faster and more cheaply [5–8].”
From Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR) (2018) · cites this paper as [6]
Experimental — accuracy not yet measured
34 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · cited 1×
“For example, if you care about both the TPs and the TNs, you’d use ROC-AUC, but if you only care about the TPs, you might prefer PR_AUC [78].”
BackgroundProceedings of the 19th International Joint Conference on Artificial Intelligence
Pujol, J. M., Delgado, J., Ramon Sangüesa, A., & Flache, A. (2005) · International Joint Conference on Artificial Intelligence · Cited by 5,771
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“Some online learning algorithms adapt their models quickly to new coming data and can be adapted to deal with slight concept drift [76].”
Background2009 International Joint Conference on Artificial Intelligence
(2009) · doi:10.1109/jcai15434.2009 · Cited by 2,800
Systematic Reviews: Rationale for systematic reviews
Mulrow, C. D. (1994) · BMJ · doi:10.1136/bmj.309.6954.597 · Cited by 1,677
An Introduction to Systematic Reviews
Gough, D., Oliver, S., & Thomas, J. (2017) · doi:10.4135/9781036234942 · Cited by 1,239
Self-citationSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
(1976) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1976.4309452 · Cited by 1,078
Clarifying differences between review designs and methods
Gough, D., Thomas, J., & Oliver, S. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-28 · Cited by 1,063 · cited 1×
“Whilst they are often associated with medical research and randomised controlled trials, they can be used to address any research question using any relevant type of research [3].”
BackgroundSelf-citation(1976) · Computer-Aided Design · doi:10.1016/0010-4485(76)90048-8 · Cited by 1,046
Finding What Works in Health Care: Standards for Systematic Reviews
Eden, J., Levit, L. A., Berg, A. O., & Morton, S. C. (2013) · Cited by 948
A brief history of research synthesis
Chalmers, I., Hedges, L. V., & Cooper, H. (2002) · Evaluation & the Health Professions · doi:10.1177/0163278702025001003 · Cited by 552 · cited 1×
“A critical feature of a systematic review is the application of scientific methods to uncover and minimise bias and error in the selection and treatment of studies [4,5].”
BackgroundDeploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532 · cited 4×
“The bespoke systems for systematic reviews which were used in evaluations in this review are: Abstrackr [49,50], EPPI-Reviewer [31,57], GAPScreener [51] and Revis [64].”
BackgroundMethodInfluentialSystematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1×
“The use of automation to assist in study selection is possibly the most advanced of all the areas where automation in systematic reviews is being developed; but others range from writing sections of the report, formulating the review question and automated data extraction and quality assessment [90-93].”
BackgroundProceedings of the Fourteenth International Conference on Machine Learning
Fisher, D. (1997) · Cited by 422
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 7×
“Cohen et al. also reported good results for a weighted model, in which they modified their voting perceptron classifier to incorporate a false negative learning rate (FNLR) [36].”
BackgroundMethodInfluentialText Mining for Biology And Biomedicine
Ananiadou, S., & McNaught, J. (2005) · Cited by 369
Self-citationSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 10×
“as a means of overcoming hasty generalisation using an active learning approach [11].”
BackgroundMethodInfluentialInterventions based on the Theory of Mind cognitive model for autism spectrum disorder (ASD)
Fletcher-Watson, S., McConnell, F., Manola, E., & McConachie, H. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd008785.pub2 · Cited by 228 · cited 1×
“For example, 4 out of the 31 Cochrane Collaboration systematic reviews published in March 2014 had over 10,000 items to screen [26-29].”
Watt, A., Cameron, A., Sturm, L., Lathlean, T., Babidge, W., Blamey, S., Facey, K., Hailey, D., Norderhaug, I., & Maddern, G. (2008) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462308080185 · Cited by 226 · cited 1×
“The need to complete reviews to tight timescales has led (particularly in health technology assessments and other rapid reviews) to the adoption of highly pragmatic (and relatively specific) strategies to searching in order to limit the number of studies to screen—even though relevant research is probably missed becau…”
BackgroundProceedings of the 23rd International Conference on Computational Linguistics: Posters
Joshi, A. K., Huang, C., & Jurafsky, D. (2010) · Cited by 215
Applications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212 · cited 2×
“There are two ways of using text mining that are particularly promising for assisting with screening in systematic reviews: one aims to prioritise the list of items for manual screening so that the studies at the top of the list are those that are most likely to be relevant; the second method uses the manually assigne…”
BackgroundMethodSelf-citationReducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190 · cited 12×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialSelf-citationExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · cited 1×
“The use of automation to assist in study selection is possibly the most advanced of all the areas where automation in systematic reviews is being developed; but others range from writing sections of the report, formulating the review question and automated data extraction and quality assessment [90-93].”
BackgroundAn Optimized Cost-Sensitive SVM for Imbalanced Data Learning
Cao, P., Zhao, D., & Zai͏̈ane, O. R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37456-2_24 · Cited by 187
Cohen, K. B., Johnson, H. L., Verspoor, K., Roeder, C., & Hunter, L. E. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-492 · Cited by 174 · cited 1×
“Cohen and colleagues have investigated more widely the implications for text mining of processing abstracts as opposed to full-text articles, and moreover mention a second issue, to do with problems that may arise for systems in going from the processing of abstracts to the processing of full text, but note that there…”
BackgroundShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170 · cited 10×
“The second approach is to use a ranking or prioritisation system and then exclude items that fall below some threshold or criterion, or that lie within a ‘negative prediction zone’ [31,52-57]; seven studies used this approach.”
BackgroundMethodInfluentialSelf-citationEstimating Time to Conduct a Meta-analysis From Number of Citations Retrieved
Allen, I. E. (1999) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.282.7.634 · Cited by 163 · cited 1×
“Given that an experienced reviewer can take between 30 seconds and several minutes to evaluate a citation [11], the work involved in screening 10,000 citations is considerable (and the screening burden in some reviews is considerably higher than this) (see also [12]).”
BackgroundSystematic Research Synthesis to Inform Policy, Practice and Democratic Debate
Gough, D., & Elbourne, D. (2002) · Social Policy and Society · doi:10.1017/s147474640200307x · Cited by 161 · cited 1×
“Systematic reviews are a widely used method to bring together the findings from multiple studies in a reliable way and are often used to inform policy and practice, such as guideline development [1,2].”
MethodMartin, A., Saunders, D. H., Shenkin, S. D., & Sproule, J. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009728.pub2 · Cited by 149 · cited 1×
“For example, 4 out of the 31 Cochrane Collaboration systematic reviews published in March 2014 had over 10,000 items to screen [26-29].”
BackgroundSupporting Systematic Reviews Using Text Mining
Ananiadou, S., Rea, B., Okazaki, N., Procter, R., & Thomas, J. (2009) · Social Science Computer Review · doi:10.1177/0894439309332293 · Cited by 144 · cited 1×
“In the context of finding research for inclusion in a review, we are interested in automated techniques of discovering whether a given study (described by a title and abstract) is relevant to our review [19,20].”
BackgroundSelf-citationTikka, C., Verbeek, J. H. A. M., Saarto, A., Lavoie, M., Pahwa, M., & Ijaz, S. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009573.pub2 · Cited by 143 · cited 1×
“For example, 4 out of the 31 Cochrane Collaboration systematic reviews published in March 2014 had over 10,000 items to screen [26-29].”
BackgroundThe automation of systematic reviews
Tsafnat, G., Dunn, A., Glasziou, P., & Coiera, E. (2013) · BMJ · doi:10.1136/bmj.f139 · Cited by 131
Automatic text classification to support systematic reviews in medicine
García Adeva, J. J., Pikatza Atxa, J. M., Ubeda Carrillo, M., & Ansuategi Zengotitabengoa, E. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.047 · Cited by 125 · cited 5×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialActive learning for biomedical citation screening
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835829 · Cited by 123 · cited 7×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialA new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100 · cited 6×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialVibration response imaging : protocol for a systematic review
Berry, M., & Camporota (2013) · Research Bank (Australian Catholic University) · doi:10.1186/2046 · Cited by 99
Searching for studies: A guide to information retrieval for Campbell systematic reviews
MacDonald, H., Comer, C., Foster, M., Labelle, P. R., Marsalis, S., Nyhan, K., Premji, Z., Rogers, M., Splenda, R., Stansfield, C., et al. (2024) · Campbell Systematic Reviews · doi:10.1002/cl2.1433 · Cited by 98
An Actual Survey of Dimensionality Reduction
Sarveniazi, A. (2014) · American Journal of Computational Mathematics · doi:10.4236/ajcm.2014.42006 · Cited by 95 · cited 1×
“2 [82].”
Blake, C. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.11.001 · Cited by 91 · cited 1×
“For example, regarding use of an abstract to determine what claims are being made, Blake found that, in biomedicine, fewer than 8% of the scientific claims made in full-text articles were to be found in their abstracts, which would certainly motivate the need to process full text [86].”
BackgroundCross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88 · cited 6×
“In contrast, whilst the review update might appear to be the more straightforward situation, since there are preexisting citation decisions on which to ‘learn’, some of the earliest work included in our review—by Cohen— shows that review updates face many challenges of their own [35,66,68,69].”
BackgroundMethodInfluentialAMIA 2007 Symposium Proceedings
Leegon, J., Hoot, N. R., Aronsky, D., & Storkey, A. (2006) · Cited by 85
Optimizing feature representation for automated systematic review work prioritization.
Cohen, A. M. (2008) · PubMed Central · Cited by 84
Precision of healthcare systematic review searches in a cross‐sectional sample
Sampson, M., Tetzlaff, J., & Urquhart, C. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.42 · Cited by 80 · cited 1×
“9% across 94 health-related systematic reviews [72].”
Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70 · cited 8×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialUsing Visual Text Mining to Support the Study Selection Activity in Systematic Literature Reviews
Felizardo, K. R., Salleh, N., Martins, R. M., Mendes, E., MacDonell, S. G., & Maldonado, J. C. (2011) · 2011 International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.16 · Cited by 68 · cited 4×
“However, the vast majority were conducted on review datasets that were well below 5,000 items, with the smallest datasets being only 57 items (20 in the training set, 37 in the test set; [64,65]).”
BackgroundInfluentialDevices for preventing percutaneous exposure injuries caused by needles in healthcare personnel
Lavoie, M., Verbeek, J. H. A. M., & Pahwa, M. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009740.pub2 · Cited by 65 · cited 1×
“For example, 4 out of the 31 Cochrane Collaboration systematic reviews published in March 2014 had over 10,000 items to screen [26-29].”
BackgroundWho Should Label What? Instance Allocation in Multiple Expert Active Learning
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2011) · Proceedings of the 2011 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972818.16 · Cited by 63 · cited 6×
“Wallace (2011) [58] used U19—a weighted metric in which recall is 19 times as important as cost.”
BackgroundMethodInfluentialToward modernizing the systematic review pipeline in genetics: efficient updating via data mining
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A. (2012) · Genetics in Medicine · doi:10.1038/gim.2012.7 · Cited by 62 · cited 8×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialA new iterative method to reduce workload in systematic review process
Jonnalagadda, S., & Petitti, D. (2013) · International Journal of Computational Biology and Drug Design · doi:10.1504/ijcbdd.2013.052198 · Cited by 61 · cited 9×
“policy demand 100% recall and informatics approaches such as ours are not able to guarantee 100% recall, the approaches may be doomed’ ([23] p.”
BackgroundMethodInfluentialA visual analysis approach to validate the selection review of primary studies in systematic reviews
Felizardo, K. R., Andery, G. F., Paulovich, F. V., Minghim, R., & Maldonado, J. C. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.04.003 · Cited by 60 · cited 7×
“The results suggest that humans can screen faster with VDM aids than without, although the accuracy of the human screeners does not appear to change substantially [13,14,63-65].”
BackgroundMethodInfluentialTowards Automating the Initial Screening Phase of a Systematic Review
Tanja, B., & Dina, D.-F. (2010) · Studies in Health Technology and Informatics · doi:10.3233/978-1-60750-588-4-146 · Cited by 60 · cited 6×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialAutomating Risk of Bias Assessment for Clinical Trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2431314 · Cited by 59
Cohen, A. M., Ambert, K., & McDonagh, M. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-33 · Cited by 57 · cited 5×
“In contrast, whilst the review update might appear to be the more straightforward situation, since there are preexisting citation decisions on which to ‘learn’, some of the earliest work included in our review—by Cohen— shows that review updates face many challenges of their own [35,66,68,69].”
BackgroundMethodInfluentialTruyens, M., & Eecke, P. V. (2014) · Computer law & security review · doi:10.1016/j.clsr.2014.01.009 · Cited by 57 · cited 1×
“Open Access publications are not subject to the same constraints as subscriptionbased content; however, there is growing concern amongst researchers and funding bodies that opportunities are being lost to advance knowledge and boost innovation and growth due to restrictive copyright and licencing regimes that are unsu…”
BackgroundYu, W., Clyne, M., Dolan, S. M., Yesupriya, A., Wulf, A., Liu, T., Khoury, M. J., & Gwinn, M. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-205 · Cited by 52 · cited 4×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialExploiting the systematic review protocol for classification of medical abstracts
Frunza, O., Inkpen, D. Z., Matwin, S., Klement, W., & O’Blenis, P. (2010) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2010.10.005 · Cited by 51 · cited 10×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodResultInfluentialBekhuis, T., Tseytlin, E., Mitchell, K. J., & Demner-Fushman, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086277 · Cited by 45 · cited 9×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialAn effective general purpose approach for automated biomedical document classification.
(2006) · Europe PMC (PubMed Central) · Cited by 44
Thomas, J. (2013) · OA Evidence-Based Medicine · doi:10.13172/2053-2636-1-2-1109 · Cited by 44 · cited 1×
“Whilst the logic behind applying text mining to the screening stage of systematic reviews has intuitive appeal, there are obvious concerns that might be raised by the systematic review community [21].”
BackgroundSelf-citationReichman, J. H., & Okediji, R. L. (2012) · Minnesota law review · doi:10.24926/265535.1308 · Cited by 44 · cited 1×
“Open Access publications are not subject to the same constraints as subscriptionbased content; however, there is growing concern amongst researchers and funding bodies that opportunities are being lost to advance knowledge and boost innovation and growth due to restrictive copyright and licencing regimes that are unsu…”
BackgroundCohen, A. M., Ambert, K. H., & McDonagh, M. S. (2010) · PubMed Central · Cited by 41
Linked data approach for selection process automation in systematic reviews
Tomassetti, F., Rizzo, G., Vetro, A., Ardito, L., Torchiano, M., & Morisio, M. (2011) · 15th Annual Conference on Evaluation & Assessment in Software Engineering (EASE 2011) · doi:10.1049/ic.2011.0004 · Cited by 41 · cited 3×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodDalal, S. R., Shekelle, P. G., Hempel, S., Newberry, S. J., Motala, A., & Shetty, K. D. (2013) · Medical Decision Making · doi:10.1177/0272989x12457243 · Cited by 32 · cited 4×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialCohen, A. M. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.008177 · Cited by 32 · cited 3×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodGarcía, V., Mollineda, R. A., & Sánchez, J. S. (2014) · Knowledge-Based Systems · doi:10.1016/j.knosys.2014.01.021 · Cited by 31 · cited 1×
“See also [79].”
Automating risk of bias assessment for clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2014) · Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics · doi:10.1145/2649387.2649406 · Cited by 28 · cited 1×
“The use of automation to assist in study selection is possibly the most advanced of all the areas where automation in systematic reviews is being developed; but others range from writing sections of the report, formulating the review question and automated data extraction and quality assessment [90-93].”
BackgroundModeling annotation time to reduce workload in comparative effectiveness reviews
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2010) · Proceedings of the 1st ACM International Health Informatics Symposium · doi:10.1145/1882992.1882999 · Cited by 25 · cited 9×
“The only example that was identified of this type was by Wallace and colleagues [49].”
BackgroundMethodInfluentialMatwin, S., & Sazonova, V. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001072 · Cited by 24
Combining relevancy and methodological quality into a single ranking for evidence-based medicine
Choi, S., Ryu, B., Yoo, S., & Choi, J. (2012) · Information Sciences · doi:10.1016/j.ins.2012.05.027 · Cited by 23 · cited 4×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialFelizardo, K. R., Souza, S. R. S., & Maldonado, J. C. (2013) · International Workshop on Replication in Empirical Software Engineering Research (RESER) · doi:10.1109/reser.2013.9 · Cited by 22 · cited 6×
“The results suggest that humans can screen faster with VDM aids than without, although the accuracy of the human screeners does not appear to change substantially [13,14,63-65].”
BackgroundInfluentialKim, S., & Choi, J. (2012) · Healthcare Informatics Research · doi:10.4258/hir.2012.18.1.18 · Cited by 18 · cited 4×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialClassifying Biomedical Abstracts Using Committees of Classifiers and Collective Ranking Techniques
Kouznetsov, A., Matwin, S., Inkpen, D. Z., Razavi, A. H., Frunza, O., Sehatkar, M., Seaward, L., & O’Blenis, P. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-01818-3_29 · Cited by 17 · cited 4×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialSampson, M., Barrowman, N. J., Moher, D., Clifford, T., Platt, R. W., Morrison, A., Klassen, T. P., & Zhang, L. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-7 · Cited by 16
Brunton, Caird, J., Sutcliffe, K., Rees, R. W., Stokes, G., Oliver, S. R., Stansfield, C., Llewellyn, A., Simmonds, M. C., & Thomas, J. (2015) · UCL Discovery (University College London) · Cited by 16
Self-citationHarden, A., Peersman, G., Oliver, S. R., & Oakley, A. (1999) · Health Education Journal · doi:10.1177/001789699905800310 · Cited by 15 · cited 1×
“In practice, this means adopting a multi-layered approach to searching which combines: extensive Boolean searches of electronic bibliographic databases, specialised registers and websites; with individual approaches to authors and key informants; and the following of ‘citation trails’ (identifying which papers are cit…”
BackgroundRazavi, A. H., Matwin, S., Inkpen, D. Z., & Kouznetsov, A. (2009) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2009.49 · Cited by 14 · cited 7×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialCombining relevance assignment with quality of the evidence to support guideline development.
Marcelo Fiszman, B., Bray, B. E., Shin, D., Halil Kilicoglu, B., Bennett, G. C., Bodenreider, O., & Rindflesch, T. C. (2010) · PubMed · Cited by 14
Performance of SVM and Bayesian classifiers on the systematic review classification task
Matwin, S., Kouznetsov, A., Inkpen, D. Z., Frunza, O., & O’Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.009555 · Cited by 13 · cited 3×
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodSemantic processing to support clinical guideline development.
Marcelo Fiszman, B., Ortíz, E., Bray, B. E., & Rindflesch, T. C. (2008) · PubMed Central · Cited by 13
AUTOMATIC SUMMARIZATION OF CLINICAL ABSTRACTS FOR EVIDENCE-BASED MEDICINE
Summerscales, R. (2013) · Digital Commons - Andrews University (Andrews University) · Cited by 10
Text classification on imbalanced data: Application to systematic reviews automation
Ma, Y. (2007) · uO Research (University of Ottawa) · doi:10.20381/ruor-12126 · Cited by 10
Kouznetsov, A., & Japkowicz, N. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13059-5_33 · Cited by 6 · cited 7×
“The second approach is to use a ranking or prioritisation system and then exclude items that fall below some threshold or criterion, or that lie within a ‘negative prediction zone’ [31,52-57]; seven studies used this approach.”
BackgroundMethodInfluential
Research works printed but not linked: 12
- Lefebvre C Manheimer E Glanville J Higgins J Green S Searching for studies (chapter 6) Cochrane Handbook for Systematic Reviews of Interventions Version 510 [updated March 2011] 2011 Oxford The Cochrane Collaboration
- Gomersall A Cooper C Database selection bias and its affect on systematic reviews: a United Kingdom perspective Joint Colloquium of the Cochrane and Campbell Collaborations 2010 Keystone, Colorado The Campbell Collaboration
- Malheiros V Hohn E Pinho R Mendonca M A visual text mining approach for systematic reviews Empirical Software Engineering and Measurement, 2007 ESEM 2007 First International Symposium on: 2007 2007 2007 Piscataway IEEE 245 54
- Hearst M Untangling Text Data Mining Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL 1999): 1999 1999 3 10
- Thomas J Brunton J Graziosi S EPPI-Reviewer 4.0: Software for Research Synthesis 2010 London EPPI-Centre Software, Social Science Research Unit, Institute of Education
- Sun Y Yang Y Zhang H Zhang W Wang Q Towards evidence-based ontology for supporting Systematic Literature Review Proceedings of the EASE Conference 2012: 2012 2012 2012 Ciudad Real Spain IET
- Martinez D Karimi S Cavedon L Baldwin T Facilitating biomedical systematic reviews using ranked text retrieval and classification Proceedings of the 13th Australasian Document Computing Symposium: 2008; Hobart Australia 2008 53
- Thomas J O’Mara A How can we find relevant research more quickly? NCRM MethodsNews 2011 UK NCRM 3
- Felizardo K, Maldonado J, Minghim R, MacDonell S, Mendes E: An extension of the systematic literature review process with visual text mining: a case study on software engineering. 16. Unpublished. Downloadable from: (p.164) http://www.teses.usp.br/teses/disponiveis/55/55134/tde-18072012–102032/publico/Thesis.pdf
- Sasaki Y Automatic text classification University of Manchester: presentation 2008
- Brinker K Incorporating diversity in active learning with support vector machines Proceedings of the 20th International Conference on Machine Learning: 2003 2003 Palo Alto AAAI Press 59 66
- Settles B Active Learning Literature Survey. Computer Sciences Technical Report 1648 2009 Wisconsin University of Wisconsin–Madison
Cited by
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationRayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874
Self-citationPage, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Self-citationThe PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
Self-citation
Metadata
- Authors
- O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S.
- Affiliations
- University of London (GB)
- University of Manchester (GB)
- Toyota Technological Institute (JP)
- Corresponding author
- Thomas, James
- Venue
- Systematic Reviews
- Published
- 2015-01-14
- Volume
- 4
- Issue
- 1
- Pages
- 5-5
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC4320539
- OpenAlex
- W4313371821
- Cited by
- 681
- References
- 82
- Citations by year
- 2026: 60 · 2025: 72 · 2024: 87 · 2023: 80 · 2022: 56 · 2021: 102 · 2020: 64 · 2019: 54 · 2018: 44 · 2017: 27 · 2016: 23 · 2015: 9
- FWCI
- 5.65
- Type
- journal article
- Language
- en
- Update notices
- erratum (2015-04-28)
- Open access
- gold
- doi.org
- Licence
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Medical Research Council (MR/J005037/1, MR/L01078X/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- text mining, systematic review screening, study identification, automated screening, memory retrieval, interdisciplinary collaboration
- MeSH terms
- Humans, Publications, Databases, Factual, Information Storage and Retrieval, Computational Biology, Evidence-Based Medicine, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
