Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.

Advancing Multilingual Abstract Classification: A Comparative Analysis of Feature Extraction Models in Systematic Reviews

Boer, Rob den, Hofstee, Laura, Leenaars, Cathalijn, Bagheri, Ayoub, Teijema, Jelle Jasper, van de Schoot, Rens·2024
DOI 10.31234/osf.io/zht9d
Open this paper in CoChat — free account

Abstract

Screening prioritization via active learning plays a vital role in streamlining the systematic review process. However, most existing software implementations are primarily designed for English texts, limiting their effectiveness in handling multilingual abstracts. This study aims to evaluate eight feature extraction models explicitly engineered for multilingual tasks, comparing their performance with two commonly used models, TFiDF and SBERT. Using a simulation-based approach, the models are assessed across four datasets, considering two classification algorithms and two prior knowledge conditions. The results highlight the superiority of models specifically designed for multilingual tasks, such as mLongT5, LaBSE, and MUSE, in classifying multilingual abstracts. These models exhibit faster identification of relevant papers, outperforming traditional models like TFiDF, which struggles to handle non-English papers. The choice of classification algorithm and prior knowledge conditions showed minimal impact on performance. This finding challenges the common reliance on TFiDF in active learning-based tools and emphasizes the need for more effective multilingual models. The study also reveals variations in model performance based on the language composition of datasets. Thus, the generalizability of findings may be limited to the specific dataset used. This emphasizes the importance of consciously addressing language diversity in systematic reviews. By adopting more effective multilingual models, researchers can enhance the efficiency and accuracy of automated abstract screening, overcoming language barriers in scientific research. Future research should explore diverse datasets and evaluate the trade-off between model performance and computational resources. Overall, this study contributes valuable insights for researchers and developers in selecting suitable feature extraction models for multilingual abstract screening tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.31234/osf.io/zht9d

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Recall plots per model for a simulation using the logistic classifier, Non-English 1 dataset, and prior knowledge combination 1. The proportion of Labeled Records on the x-axis recall on the y-axis. The black diagonal line represents nonautomated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not explicitly designed for multilingual texts, while Panels C to J depict multilingual models.
Figure 1 Recall plots per model for a simulation using the logistic classifier, Non-English 1 dataset, and prior knowledge combination 1. The proportion of Labeled Records on the x-axis recall on the y-axis. The black diagonal line represents nonautomated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not explicitly designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Figure 3 Cumulative Average Time to Discovery of target papers across different datasets. The y-axis represents the Average Time to Discovery over all simulations. Each bar in the plot consists of the stacked paper Average Time to Discovery values for the corresponding dataset. The legend provides color-coded information for each target paper, identified by its record ID.
Figure 3 Cumulative Average Time to Discovery of target papers across different datasets. The y-axis represents the Average Time to Discovery over all simulations. Each bar in the plot consists of the stacked paper Average Time to Discovery values for the corresponding dataset. The legend provides color-coded information for each target paper, identified by its record ID.CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Figure 5 Cumulative Average Time to Discovery of the target papers across various datasets and prior knowledge condition combinations. The x-axis represents the combinations of datasets and prior knowledge conditions, while the y-axis signifies the average time to discovery over all simulations. Each bar on the chart consists of stacked segments representing the Average Time to Discovery (ATD) for the five target papers specific to a dataset and prior knowledge condition combination. The legend provides color-coded information for each target paper, identified by its record ID.
Figure 5 Cumulative Average Time to Discovery of the target papers across various datasets and prior knowledge condition combinations. The x-axis represents the combinations of datasets and prior knowledge conditions, while the y-axis signifies the average time to discovery over all simulations. Each bar on the chart consists of stacked segments representing the Average Time to Discovery (ATD) for the five target papers specific to a dataset and prior knowledge condition combination. The legend provides color-coded information for each target paper, identified by its record ID.CC-BY · From the PDF
Figure A1: Recall plots per model, logistic classifier, Multiple dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A1: Recall plots per model, logistic classifier, Multiple dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A2: Recall plots per model, logistic classifier, English dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A2: Recall plots per model, logistic classifier, English dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A3: Recall plots per model, logistic classifier, Non-English 1 dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A3: Recall plots per model, logistic classifier, Non-English 1 dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A4: Recall plots per model, logistic classifier, Non-English 2 dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A4: Recall plots per model, logistic classifier, Non-English 2 dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A5: Recall plots per model, svm classifier, Multiple dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A5: Recall plots per model, svm classifier, Multiple dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A6: Recall plots per model, svm classifier, English dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A6: Recall plots per model, svm classifier, English dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A7: Recall plots per model, svm classifier, Non-English 1 dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A7: Recall plots per model, svm classifier, Non-English 1 dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A8: Recall plots per model, svm classifier, Non-English 2 dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A8: Recall plots per model, svm classifier, Non-English 2 dataset, prior knowledge combination 1. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A9: Recall plots per model, logistic classifier, Multiple dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A9: Recall plots per model, logistic classifier, Multiple dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A10: Recall plots per model, logistic classifier, English dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A10: Recall plots per model, logistic classifier, English dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A11: Recall plots per model, logistic classifier, Non-English 1 dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A11: Recall plots per model, logistic classifier, Non-English 1 dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A2: Recall plots per model, logistic classifier, Non-English 2 dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A2: Recall plots per model, logistic classifier, Non-English 2 dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A13: Recall plots per model, svm classifier, Multiple dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A13: Recall plots per model, svm classifier, Multiple dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A14: Recall plots per model, svm classifier, English dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A14: Recall plots per model, svm classifier, English dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A15: Recall plots per model, svm classifier, Non-English 1 dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A15: Recall plots per model, svm classifier, Non-English 1 dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF
Figure A16: Recall plots per model, svm classifier, Non-English 2 dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.
Figure A16: Recall plots per model, svm classifier, Non-English 2 dataset, prior knowledge combination 2. Proportion of Labeled Records on x-axis, recall on y-axis. The black diagonal line represents a non-automated screening progress where the relevant papers are uniformly distributed in the dataset. The recall plots illustrate the discovery points of target papers, annotated with record IDs, while the legend indicates the language of each target paper. Panels A and B represent models not specifically designed for multilingual texts, while Panels C to J depict multilingual models.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Research works printed but not linked: 14

  1. Lefebvre, C., E. Manheimer, and J. Glanville, Searching for Studies, in Cochrane Handbook for Systematic Reviews of Interventions, J.P.T. Higgins and S. Green, Editors. 2008, John Wiley & Sons, Ltd: Chichester, UK. p. 95-150.
  2. Stone, J., et al., Stratification by quality induced selection bias in a meta-analysis of clinical trials. Journal of Clinical Epidemiology, 2019. 107: p. 51-59.
  3. Higgins, J.P., et al., Cochrane handbook for systematic reviews of interventions. 2019: John Wiley & Sons.
  4. Pieper, D. and L. Puljak, Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility criteria if necessary. Journal of Clinical Epidemiology, 2021. 132: p. 146-147.
  5. Klassen, T.P., M.L. Lawson, and D. Moher, Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. Journal of clinical epidemiology, 2005. 58(8): p. 769-776. e2.
  6. Teijema, J.J., et al., Simulation-based Active Learning for Systematic Reviews: A Systematic Review of the Literature. 2023.
  7. Wallace, B.C., et al., Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. Proceedings of the ACM International Health Informatics Symposium (IHI), 2012: p. 819-824.
  8. Artetxe, M. and H. Schwenk, Massively multilingual sentence embeddings for zero-shot cross- lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 2019. 7: p. 597-610.
  9. Ainslie, J., et al., ETC: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.08483, 2020. · arXiv
  10. Zaheer, M., et al., Big bird: Transformers for longer sequences. Advances in neural information processing systems, 2020. 33: p. 17283-17297.
  11. Beltagy, I., M.E. Peters, and A. Cohan, Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. · arXiv
  12. Tay, Y., et al., Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020. · arXiv
  13. Kaplan, J., et al., Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. · arXiv
  14. Dobrescu, A., et al., Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review. Journal of Clinical Epidemiology, 2021. 137: p. 209-217.

Cited by

Metadata

Authors
Boer, R. D., Hofstee, L., Leenaars, C., Bagheri, A., Teijema, J. J., & van de Schoot, R.
Publisher
Center for Open Science
Published
2024-01-26
DOI
10.31234/osf.io/zht9d
Cited by
1
References
13
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode (unspecified)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.