CoChat
Sign Up
Cited by 21Meta-analysis and systematic reviewsscientometrics and bibliometrics researchInformation Retrieval and Search Behavior

An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews

Open access
Kusa, Wojciech, Lipani, Aldo, Knoth, Petr, Hanbury, Allan·2023·Intelligent Systems with Applications
DOI 10.1016/j.iswa.2023.200193
Open this paper in CoChat — free account

Abstract

Citation screening is an essential and time-consuming step of the systematic literature review process in medicine. Multiple previous studies have proposed various automation techniques to assist manual annotators in this tedious task. The most widely used measure for the evaluation of automated citation screening techniques is Work Saved over Sampling (WSS). In this work, we analyse this measure and examine its drawbacks. We subsequently propose to normalise WSS which enables citation screening performance comparisons across different systematic reviews. We analytically show that normalised WSS is equivalent to the True Negative Rate (TNR). Finally, we provide benchmark scores for fifteen systematic review datasets with [email protected]% recall measure and compare the measure with Precision and AUC.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.iswa.2023.200193

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Normalised Work Saved over Sampling (nWSS) metric facilitates the comparison of paper screening performance across various datasets.

    “To address this, they introduced the Normalised Work Saved over Sampling (nWSS) metric (Kusa et al. 2023), which facilitates the comparison of paper screening performance across various datasets.”

    From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Kusa et al. 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. References Bannach-Brown, A., Przyby la, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019). Machine learning algorithms for system-atic review: reducing workload in a preclinical review of animal studies and 300 reducing human screening error. Systematic Reviews 2019 8:1, 8, 1–12. URL: https://systematicreviewsjournal.biomedcentral.com/articles/10. 1186/s13643-019-0942-7. doi:10.1186/S13643-019-0942-7. · DOI · Paper page
  2. Busin, L., & Mizzaro, S. (2013). Axiometrics: An axiomatic approach to infor-mation retrieval effectiveness metrics. In Proceedings of the 2013 Conference 305 on the Theory of Information Retrieval (pp. 22–29).
  3. Cohen, A. M. (2008). Optimizing Feature Representation for Automated Systematic Review Work Prioritization. AMIA Annual Symposium Pro-ceedings, 2008, 121. URL: /pmc/articles/PMC2656096//pmc/articles/ 18 PMC2656096/?report=abstracthttps://www.ncbi.nlm.nih.gov/pmc/ 310 articles/PMC2656096/.
  4. Cohen, A. M. (2011). Letter: Performance of support-vector-machine-based classification on 15 systematic review topics evaluated with the WSS@95 mea-sure. Journal of the American Medical Informatics Association : JAMIA, 18, 104. URL: /pmc/articles/PMC3005879/https://www.ncbi.nlm.nih.gov/ 315 pmc/articles/PMC3005879/. doi:10.1136/JAMIA.2010.008177. · DOI
  5. J¨ arvelin, K., & Kek¨ al¨ ainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20, 422–446. doi:10.1145/582415.582418. · DOI · Paper page
  6. Jiang, Y., Metz, C. E., & Nishikawa, R. M. (1996). A receiver operating char-350 acteristic partial area index for highly sensitive diagnostic tests. Radiology, 201, 745–750. Kanoulas, E., Li, D., Azzopardi, L., & Spijker, R. (2017). CLEF 2017 technologically assisted reviews in empirical medicine overview. CEUR Workshop Proceedings, 1866, 1– 355 29. URL: https://pureportal.strath.ac.uk/en/publications/ clef-2017-technologically-assisted-reviews-in-empirical-medicine-. Kanoulas, E., Li, D., Azzopardi, L., & Spijker, R. (2018). CLEF 2018 technologically assisted reviews in empirical medicine overview. CEUR Workshop Proceedings, 2125. 360 URL: https://pureportal.strath.ac.uk/en/publications/ clef-2018-technologically-assisted-reviews-in-empirical-medicine-.
  7. Kontonatsios, G., Spencer, S., Matthew, P., & Korkontzelos, I. (2020). Using a neural network-based feature extraction method to facilitate citation screen-ing for systematic reviews. Expert Systems with Applications: X , 6, 100030. 365 doi:10.1016/j.eswax.2020.100030. 20 Kusa, W., Hanbury, A., & Knoth, P. (2022). Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study. In Advances in Information Retrieval, 44th European Conference on IR Research, ECIR 2022. URL: https://arxiv.org/abs/2201.07534v1. 370 Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O’Blenis, P. (2010). A new algorithm for reducing the workload of experts in performing systematic reviews. Journal of the American Medical Informatics Association, 17, 446– 453. doi:10.1136/JAMIA.2010.004325. · DOI
  8. Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014). Reducing systematic review workload through certainty-based screening. Journal of Biomedical Informatics, 51, 242–253. doi:10.1016/J.JBI.2014.06.005. · DOI · Paper page
  9. Scells, H., Zuccon, G., & Koopman, B. (2019). Automatic boolean query refinement for systematic review literature search. The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019, 11, 1646–1656. URL: https://doi.org/10.1145/3308558.3313544. doi:10. 390 1145/3308558.3313544. · DOI
  10. Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2010). Active learning for biomedical citation screening. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, (pp. 400 173–181). doi:10.1145/1835804.1835829. 22 Declaration of interests ☒ The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. ☐ The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Wojciech Kusa: Conceptualization, Formal analysis, Methodology, Writing - Original Draft Aldo Lipani: Formal analysis, Validation, Writing - Review & Editing, Supervision Petr Knoth: Validation, Writing - Review & Editing, Supervision Allan Hanbury: Validation, Writing - Review & Editing, Supervision · DOI · Paper page

Cited by

Sorted by citations

Authors
Kusa, W., Lipani, A., Knoth, P., & Hanbury, A.
Affiliations
  • TU Wien (AT)
  • University College London (GB)
  • The Open University (GB)
Corresponding author
Kusa, Wojciech
ORCID
Venue
Intelligent Systems with Applications
Publisher
Elsevier BV
Published
2023-05
Volume
18
Pages
200193
ISSN
2667-3053 (ISSN-L)
DOI
10.1016/j.iswa.2023.200193
OpenAlex
W4319962920
Cited by
21
References
6
Citations by year
2026: 6 · 2025: 5 · 2024: 4 · 2023: 6
FWCI
1.31
Type
journal article
Language
en
Open access
Licence
Locations
  • Intelligent Systems with Applications (publishedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion, cc-by)
  • Open Research Online (The Open University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Horizon 2020 Framework Programme (H2020-EU.1.3.1, 860721)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
test specificity, systematic reviews, benchmarking, memory retrieval, precision, area under the curve, performance indicators

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review