The DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.
Using text mining for study identification in systematic reviews: a systematic review of current approaches
Open accessAbstract
BACKGROUND: The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way for inclusion in systematic reviews both complex and time consuming. Text mining has been offered as a potential solution: through automating some of the screening process, reviewer time can be saved. The evidence base around the use of text mining for screening has not yet been pulled together systematically; this systematic review fills that research gap. Focusing mainly on non-technical issues, the review aims to increase awareness of the potential of these technologies and promote further collaborative research between the computer science and systematic review communities. METHODS: Five research questions led our review: what is the state of the evidence base; how has workload reduction been evaluated; what are the purposes of semi-automation and how effective are they; how have key contextual problems of applying text mining to the systematic review field been addressed; and what challenges to implementation have emerged? We answered these questions using standard systematic review methods: systematic and exhaustive searching, quality-assured data extraction and a narrative synthesis to synthesise findings. RESULTS: The evidence base is active and diverse; there is almost no replication between studies or collaboration between research teams and, whilst it is difficult to establish any overall conclusions about best approaches, it is clear that efficiencies and reductions in workload are potentially achievable. On the whole, most suggested that a saving in workload of between 30% and 70% might be possible, though sometimes the saving in workload is accompanied by the loss of 5% of relevant studies (i.e. a 95% recall). CONCLUSIONS: Using text mining to prioritise the order in which items are screened should be considered safe and ready for use in 'live' reviews. The use of text mining as a 'second screener' may also be used cautiously. The use of text mining to eliminate studies automatically should be considered promising, but not yet fully proven. In highly technical/clinical areas, it may be used with a high degree of confidence; but more developmental and evaluative work is needed in other disciplines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They can be calibrated to achieve a given level of recall, thus enabling reviewers to implement screening rules, such as eliminating records or replacing double with single screening.”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, there is a risk of erroneously excluding relevant studies because of uncertainty about when it is safe to stop screening; the balance between efficiency gains and risk tolerance will be review-specific52 53”
From PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews (2021) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Because there are fewer examples of relevant than irrelevant publications to train on, the class imbalance causes the classifier to miss relevant publications [32].”
From Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records (2023) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As mentioned above, these algorithms/software are more and more used for article screening in systematic reviews [53, 55, 56]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent advances in natural language processing, text mining and machine learning have produced new algorithms that can accurately mimic human endeavour in systematic review activity, faster and more cheaply [5–8].”
From Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR) (2018) · cites this paper as [6]
Experimental — accuracy not yet measured
34 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · cited 1×
“For example, if you care about both the TPs and the TNs, you’d use ROC-AUC, but if you only care about the TPs, you might prefer PR_AUC [78].”
BackgroundProceedings of the 19th International Joint Conference on Artificial Intelligence
Pujol, J. M., Delgado, J., Ramon Sangüesa, A., & Flache, A. (2005) · International Joint Conference on Artificial Intelligence · Cited by 5,771
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · cited 1×
“Some online learning algorithms adapt their models quickly to new coming data and can be adapted to deal with slight concept drift [76].”
Background2009 International Joint Conference on Artificial Intelligence
(2009) · doi:10.1109/jcai15434.2009 · Cited by 2,800
Systematic Reviews: Rationale for systematic reviews
Mulrow, C. D. (1994) · BMJ · doi:10.1136/bmj.309.6954.597 · Cited by 1,677
An Introduction to Systematic Reviews
Gough, D., Oliver, S., & Thomas, J. (2017) · doi:10.4135/9781036234942 · Cited by 1,239
Self-citationSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
(1976) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1976.4309452 · Cited by 1,078
Clarifying differences between review designs and methods
Gough, D., Thomas, J., & Oliver, S. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-28 · Cited by 1,063 · cited 1×
“Whilst they are often associated with medical research and randomised controlled trials, they can be used to address any research question using any relevant type of research [3].”
BackgroundSelf-citation
Research works printed but not linked: 12
- Lefebvre C, Manheimer E, Glanville J: Searching for studies (chapter 6). In Cochrane Handbook for Systematic Reviews of Interventions Version 510 [updated March 2011]. Edited by: Higgins J, Green S. Oxford: The Cochrane Collaboration; 2011.
- Gomersall A, Cooper C: Database selection bias and its affect on systematic reviews: a United Kingdom perspective. In Joint Colloquium of the Cochrane and Campbell Collaborations. Keystone, Colorado: The Campbell Collaboration; 2010.
- Malheiros V, Hohn E, Pinho R, Mendonca M: A visual text mining approach for systematic reviews. In Empirical Software Engineering and Measurement, 2007 ESEM 2007 First International Symposium on: 2007 2007. Piscataway: IEEE; 2007:245–54.
- Hearst M: Untangling Text Data Mining.Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL 1999): 1999 1999, 3–10.
- Thomas J, Brunton J, Graziosi S: EPPI-Reviewer 4.0: Software for Research Synthesis. London: EPPI-Centre Software, Social Science Research Unit, Institute of Education; 2010.
- Sun Y, Yang Y, Zhang H, Zhang W, Wang Q: Towards evidence-based ontology for supporting Systematic Literature Review. In Proceedings of the EASE Conference 2012: 2012 2012. Ciudad Real Spain: IET; 2012.
- Martinez D, Karimi S, Cavedon L, Baldwin T: Facilitating biomedical systematic reviews using ranked text retrieval and classification.Proceedings of the 13th Australasian Document Computing Symposium: 2008; Hobart Australia 2008, 53.
- Thomas J, O’Mara A: How can we find relevant research more quickly? In NCRM MethodsNews. UK: NCRM; 2011:3.
- Felizardo K, Maldonado J, Minghim R, MacDonell S, Mendes E: An extension of the systematic literature review process with visual text mining: a case study on software engineering. 16. Unpublished. Downloadable from: (p.164) http://www.teses.usp.br/teses/disponiveis/55/55134/tde-18072012–102032/publico/Thesis.pdf
- Sasaki Y: Automatic text classification.University of Manchester: presentation 2008.
- Brinker K: Incorporating diversity in active learning with support vector machines. In Proceedings of the 20th International Conference on Machine Learning: 2003. Palo Alto: AAAI Press; 2003:59–66.
- Settles B: Active Learning Literature Survey. Computer Sciences Technical Report 1648. Wisconsin: University of Wisconsin–Madison; 2009.
Cited by
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationRayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874
Self-citationPage, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Self-citationThe PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
Self-citation
Metadata
- Authors
- O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S.
- Affiliations
- University of London (GB)
- University of Manchester (GB)
- Toyota Technological Institute (JP)
- Corresponding author
- Thomas, James
- Venue
- Systematic Reviews
- Published
- 2015-01-14
- Volume
- 4
- Issue
- 1
- Pages
- 5-5
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC4320539
- OpenAlex
- W4313371821
- Cited by
- 681
- References
- 82
- Citations by year
- 2026: 60 · 2025: 72 · 2024: 87 · 2023: 80 · 2022: 56 · 2021: 102 · 2020: 64 · 2019: 54 · 2018: 44 · 2017: 27 · 2016: 23 · 2015: 9
- FWCI
- 5.65
- Type
- journal article
- Language
- en
- Update notices
- erratum (2015-04-28)
- Open access
- gold
- doi.org
- Licence
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Medical Research Council (MR/J005037/1, MR/L01078X/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- text mining, systematic review screening, study identification, automated screening, memory retrieval, interdisciplinary collaboration
- MeSH terms
- Humans, Publications, Databases, Factual, Information Storage and Retrieval, Computational Biology, Evidence-Based Medicine, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using text mining for study identification in systematic reviews: a systematic review of current approaches. https://app.cochat.ai/papers/10.1186/2046-4053-4-5
- The paper
- O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews. https://doi.org/10.1186/2046-4053-4-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
