Title and Abstract Screening and Evaluation in Systematic Reviews (TASER): a pilot randomised controlled trial of title and abstract screening by medical students
Open accessAbstract
BACKGROUND: The production of high quality systematic reviews requires rigorous methods that are time-consuming and resource intensive. Citation screening is a key step in the systematic review process. An opportunity to improve the efficiency of systematic review production involves the use of non-expert groups and new technologies for citation screening. We performed a pilot study of citation screening by medical students using four screening methods and compared students' performance to experienced review authors. METHODS: The aims of this pilot randomised controlled trial were to provide preliminary data on the accuracy of title and abstract screening by medical students, and on the effect of screening modality on screening accuracy and efficiency. Medical students were randomly allocated to title and abstract screening using one of the four modalities and required to screen 650 citations from a single systematic review update. The four screening modalities were a reference management software program (EndNote), Paper, a web-based systematic review workflow platform (ReGroup) and a mobile screening application (Screen2Go). Screening sensitivity and specificity were analysed in a complete case analysis using a chi-squared test and Kruskal-Wallis rank sum test according to screening modality and compared to a final set of included citations selected by expert review authors. RESULTS: Sensitivity of medical students' screening decisions ranged from 46.7% to 66.7%, with students using the web-based platform performing significantly better than the paper-based group. Specificity ranged from 93.2% to 97.4% with the lowest specificity seen with the web-based platform. There was no significant difference in performance between the other three modalities. CONCLUSIONS: Medical students are a feasible population to engage in citation screening. Future studies should investigate the effect of incentive systems, training and support and analytical methods on screening performance. SYSTEMATIC REVIEW REGISTRATION: Cochrane Database CD001048.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Human citation screening is imperfect with relevant studies wrongly excluded.
“It is noteworthy, however, that human citation screening is imperfect with relevant studies wrongly excluded [25].”
From Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers (2015) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320 · see citing works
Use of Ranks in One-Criterion Variance Analysis
Kruskal, W. H., & Wallis, W. A. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483441 · Cited by 13,912 · see citing works
The proof and measurement of association between two things
Spearman, C. (1904) · The American Journal of Psychology · doi:10.2307/1412159 · Cited by 8,039 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1. The Cochrane Collaboration
Higgins, J., Green, S. R., & Higgins, J. (2008) · Cited by 5,544 · see citing works
Multiple comparisons among means
Dunn, O. J. (1961) · Journal of the American Statistical Association · doi:10.1080/01621459.1961.10482090 · Cited by 5,006 · see citing works
Intention-to-treat concept: A review
Gupta, S. (2011) · Perspectives in Clinical Research · doi:10.4103/2229-3485.83221 · Cited by 1,921 · see citing works
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · see citing works
The proof and measurement of association between two things. By C. Spearman, 1904.
Spearman, C. (1987) · PubMed · Cited by 1,023 · see citing works
Finding What Works in Health Care: Standards for Systematic Reviews
Eden, J., Levit, L. A., Berg, A. O., & Morton, S. C. (2013) · Cited by 943 · see citing works
The Use of Maximum Likelihood Estimates in $\chi^2$ Tests for Goodness of Fit
Chernoff, H., & Lehmann, E. L. (1954) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728726 · Cited by 559 · see citing works
Edwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., & Wentz, R. (2002) · Statistics in Medicine · doi:10.1002/sim.1190 · Cited by 340 · see citing works
Hypothermia for traumatic head injury
Sydenham, E., Roberts, I., & Alderson, P. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001048.pub4 · Cited by 143 · see citing works
Horton, J., Vandermeer, B., Hartling, L., Tjosvold, L., Klassen, T. P., & Buscemi, N. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.04.007 · Cited by 81 · see citing works
Flagged first, then by citations
1 of the 14 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Clavisi O Turner T Thomas J Cavedon L Development of a Web Based Software Tool to Improve Efficiency and User Experience of Systematic Reviews 2012 Auckland, New Zealand: Proceedings of the 20th Cochrane Colloquium
- Huckvale K van Valthoven M Car J Screen2Go: A Pilot Smartphone App for Citation Screening 2011 Madrid, Spain: Proceedings of the 19th Cochrane Colloquium
Cited by
Sorted by citations
Rathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184 · see citing works
Social media use in disaster recovery: A systematic literature review
Ogie, R., James, S., Moore, A. R., Dilworth, T.-L., Amirghasemi, M., & Whittaker, J. (2022) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2022.102783 · Cited by 170 · see citing works
Noel-Storr, A. H., Dooley, G., Elliott, J., Steele, E. J., Shemilt, I., Mavergames, C., Wisniewski, S., McDonald, S. J., Murano, M., Glanville, J., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.006 · Cited by 135 · see citing works
Self-citationDoes Exercise Alter Gut Microbial Composition? A Systematic Review
Mitchell, C. M., Davy, B. M., Hulver, M. W., Neilson, A. P., Bennett, B. J., & Davy, K. P. (2018) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000001760 · Cited by 117 · see citing works
Robson, R. C., Pham, B., Hwee, J., Thomas, S. M., Rios, P., Page, M. J., & Tricco, A. C. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.10.003 · Cited by 86 · see citing works
Rathbone, J., Albarqouni, L., Bakhit, M., Beller, E., Byambasuren, O., Hoffmann, T., Scott, A. M., & Glasziou, P. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0629-x · Cited by 64 · see citing works
Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow
Pham, B., Jovanovic, J., Bagheri, E., Antony, J., Ashoor, H., Nguyen, T. T., Rios, P., Robson, R., Thomas, S. M., Watt, J., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01700-x · Cited by 63 · see citing works
Affengruber, L., van der Maten, M. M., Spiero, I., Nußbaumer-Streit, B., Mahmić-Kaknjo, M., Ellen, M., Gooßen, K., Kantorová, L., Hooft, L., Riva, N., et al. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02320-4 · Cited by 57 · see citing works
Prioritization and sequential exclusion of articles in systematic reviews
Saif‐Ur‐Rahman, K., Hasan, M., Hossain, S., Anwar, I., Hirakawa, Y., & Yatsuya, H. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1229 · Cited by 48 · see citing works
Haby, M. M., Barreto, J. O. M., Kim, J. Y. H., Peiris, S., Mansilla, C., Torres, M., Guerrero‐Magaña, D. E., & Revéiz, L. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1664 · Cited by 47 · see citing works
Unraveling heterogeneity in farmer's adoption of mobile phone technologies: A systematic review
Aparo, N. O., Odongo, W., & Steur, H. D. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122048 · Cited by 41 · see citing works
Evaluation of text mining to reduce screening workload for injury-focused systematic reviews
Giummarra, M. J., Lau, G., & Gabbe, B. J. (2019) · Injury Prevention · doi:10.1136/injuryprev-2019-043247 · Cited by 38 · see citing works
Madlabana, C. Z., Mashamba-Thompson, T. P., & Petersen, I. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01294-w · Cited by 37 · see citing works
2022 Systematic Review of Evidence-Based Guidelines for Prehospital Care
Martin‐Gill, C., Brown, K., Cash, R. E., Haupt, R. M., Potts, B. T., Richards, C. T., Patterson, P. D., & for the Prehospital Guidelines Consortium (2022) · Prehospital Emergency Care · doi:10.1080/10903127.2022.2143603 · Cited by 28 · see citing works
Systematic Review Methodology for the Fatigue in Emergency Medical Services Project
Patterson, P. D., Higgins, J. S., Weiss, P. M., Lang, E., & Martin‐Gill, C. (2018) · Prehospital Emergency Care · doi:10.1080/10903127.2017.1380096 · Cited by 23 · see citing works
Buhos: A web-based systematic literature review management software
Bustos, C., Malverde, M. ́. M., Salcedo, P., & Mújica, A. D. (2018) · SoftwareX · doi:10.1016/j.softx.2018.10.004 · Cited by 21 · see citing works
Crowd-based Multi-Predicate Screening of Papers in Literature Reviews
Krivosheev, E., Casati, F., & Benatallah, B. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186036 · Cited by 19 · see citing works
Crowdsourcing Paper Screening in Systematic Literature Reviews
Krivosheev, E., Casati, F., Caforio, V., & Benatallah, B. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13302 · Cited by 16 · see citing works
O’Hearn, K., MacDonald, C., Tsampalieros, A. K., Kadota, L., Sandarage, R. V., Jayawarden, S. K., Datko, M., Reynolds, J. M., Bui, T., Sultan, S. A., et al. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01335-5 · Cited by 12 · see citing works
Geidl, W., Schlesinger, S., Mino, E., Miranda, L., Ryan, A., Bartsch, K. M., Janz, L., & Pfeifer, K. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-028653 · Cited by 11 · see citing works
2024 Systematic Review of Evidence-Based Guidelines for Prehospital Care
Martin‐Gill, C., Patterson, P. D., Richards, C. T., Misra, A. J., Potts, B. T., Cash, R. E., & for the Prehospital Guidelines Consortium (2024) · Prehospital Emergency Care · doi:10.1080/10903127.2024.2412299 · Cited by 9 · see citing works
Ayele, K., Demisew, M., & Gemede, H. F. (2025) · BMC Nutrition · doi:10.1186/s40795-025-01194-z · Cited by 9 · see citing works
Carter, B. U. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0782-x · Cited by 8 · see citing works
Gopaul, U., Charalambous, M., Thilarajah, S., Kwah, L. K., Chapman, S., Bayley, M. T., & Demers, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02147-4 · Cited by 6 · see citing works
Gopaul, U., Charalambous, M., Thilarajah, S., Kwah, L. K., Chapman, S., Bayley, M. T., & Demers, M. (2022) · Research Square · doi:10.21203/rs.3.rs-1685943/v1 · Cited by 2 · see citing works
Li, D., Smets, A., & Ranaivoson, H. R. (2026) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2026.103741 · Cited by 1 · see citing works
A SCOPING REVIEW OF SUICIDE IDEATION IN MALAYSIA AND ITS ECONOMIC COSTS .
Mat, N., Samsudin, S., & Applanaidu, S. D. (2021) · International Journal of Social Science and Economic Research · doi:10.46609/ijsser.2021.v06i07.016
Murtadho, N., & Dawood, M. (2022) · Jurnal Pendidikan Teori Penelitian dan Pengembangan · doi:10.17977/2502-471x.2506 · Cited by 1 · see citing works
Ahern, E., Adedeji, T., Whiston, A., Dillon, S., & Lynn, F. A. (2025) · HRB Open Research · doi:10.12688/hrbopenres.14129.1
Systematic Review: A Comprehensive Guide
Kamal, M. A., Alaswad, B. M., Shehata, I. M., Viswanath, O., & Koushik, S. S. (2025) · Springer eBooks · doi:10.1007/978-3-031-92538-2_14
Gopaul, U., Charalambous, M., Thilarajah, S., Kwah, L. K., Chapman, S., Bayley, M., & Demers, M. (2023) · Figshare · doi:10.6084/m9.figshare.22612433
Gopaul, U., Charalambous, M., Thilarajah, S., Kwah, L. K., Chapman, S., Bayley, M., & Demers, M. (2023) · Figshare · doi:10.6084/m9.figshare.22612427
Pham, B., Jovanovic, J., Ebrahim Bagheri, A., Antony, J., Ashoor, H. M., Nguyen, T. T., Rios, P., Robson, R. C., Thomas, S. M., Watt, J., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14684514
Gopaul, U., Charalambous, M., Thilarajah, S., Kwah, L. K., Chapman, S., Bayley, M., & Demers, M. (2023) · Figshare · doi:10.6084/m9.figshare.22612424
Gopaul, U., Charalambous, M., Thilarajah, S., Kwah, L. K., Chapman, S., Bayley, M., & Demers, M. (2023) · Figshare · doi:10.6084/m9.figshare.22612436
O'Hearn, K., MacDonald, C., Tsampalieros, A., Kadota, L., Sandarage, R., Jayawarden, S. K., Datko, M., Reynolds, J. M., Bui, T., Sultan, S. A., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14936470
Gopaul, U., Charalambous, M., Thilarajah, S., Kwah, L. K., Chapman, S., Bayley, M., & Demers, M. (2023) · Figshare · doi:10.6084/m9.figshare.22612430
- Authors
- Ng, L., Pitt, V., Huckvale, K., Clavisi, O., Turner, T., Gruen, R., & Elliott, J. H.
- Affiliations
- Monash University (AU)
- National Trauma Research Institute (AU)
- Imperial College London (GB)
- World Vision (US)
- The Alfred Hospital (AU)
- Corresponding author
- Elliott, Julian H
- Venue
- Systematic Reviews
- Published
- 2014-10-21
- Volume
- 3
- Issue
- 1
- Pages
- 121-121
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC4217707
- OpenAlex
- W2149579726
- Cited by
- 38
- References
- 14
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 6 · 2022: 7 · 2021: 6 · 2020: 1 · 2019: 2 · 2018: 6 · 2017: 2 · 2015: 1
- FWCI
- 0.27
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- ANU Open Research (Australian National University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- citation screening, title and abstract screening, medical students, screening accuracy, screening efficiency, reference management software, sensitivity and specificity
- MeSH terms
- Database Management Systems, Humans, Information Science, Judgment, Pilot Projects, Review Literature as Topic, Sensitivity and Specificity, Students, Medical, Time and Motion Studies, Databases, Bibliographic, Computers, Handheld, Mobile Applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
