Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 12

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Addressing the challenges of reconstructing systematic reviews datasets: a case study and a noisy label filter procedure

Neeleman, Rutger, Leenaars, Cathalijn H. C., Oud, Matthijs, Weijdema, Felix, van de Schoot, Rens·2024·Systematic Reviews
DOI 10.1186/s13643-024-02472-wPMC PMC10874047
Open this paper in CoChat — free account

Abstract

Systematic reviews and meta-analyses typically require significant time and effort. Machine learning models have the potential to enhance screening efficiency in these processes. To effectively evaluate such models, fully labeled datasets—detailing all records screened by humans and their labeling decisions—are imperative. This paper presents the creation of a comprehensive dataset for a systematic review of treatments for Borderline Personality Disorder, as reported by Oud et al. (2018) for running a simulation study. The authors adhered to the PRISMA guidelines and published both the search query and the list of included records, but the complete dataset with all labels was not disclosed. We replicated their search and, facing the absence of initial screening data, introduced a Noisy Label Filter (NLF) procedure using active learning to validate noisy labels. Following the NLF application, no further relevant records were found. A simulation study employing the reconstructed dataset demonstrated that active learning could reduce screening time by 82.30% compared to random reading. The paper discusses potential causes for discrepancies, provides recommendations, and introduces a decision tree to assist in reconstructing datasets for the purpose of running simulation studies.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-024-02472-w

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Schematic representation of record types and labeling notations
Fig. 1 Schematic representation of record types and labeling notationsCC BY · PubMed Central
Fig. 2 Lapse of screening process in application NLF procedure. The yellow line represents the recall of the four relevant records during the NLF procedure. Notably, the four relevant records were later classified as irrelevant, see Table 2
Fig. 2 Lapse of screening process in application NLF procedure. The yellow line represents the recall of the four relevant records during the NLF procedure. Notably, the four relevant records were later classified as irrelevant, see Table 2CC BY · PubMed Central
Fig. 3 Recall curve of the simulation study. The x-axis represents the proportion of labeled records, and the y-axis represents the proportion of relevant records. The black line represents the average process of random screening, while the colored lines represent the performance of different active learning models, with blue for logistic regression, orange for Naive Bayes, green for Random Forest, and red for support vector machine
Fig. 3 Recall curve of the simulation study. The x-axis represents the proportion of labeled records, and the y-axis represents the proportion of relevant records. The black line represents the average process of random screening, while the colored lines represent the performance of different active learning models, with blue for logistic regression, orange for Naive Bayes, green for Random Forest, and red for support vector machineCC BY · PubMed Central
Fig. 4 Simulation decision tree
Fig. 4 Simulation decision treeCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 23

  1. Akhter, S., Pauyo, T., & Khan, M. What is the difference between a systematic review and a meta-analysis? In V. Musahl, J. Karlsson, M. T. Hirschmann, O. R. Ayeni, R. G. Marx, J. L. Koh, & N. Nakamura (Eds.), Basic Methods Handbook for Clinical Orthopaedic Research: A Practical Guide and Case Based Research Approach (pp. 331–342). 2019; Springer. https://doi.org/10.1007/978-3-662-58254-1_37 · DOI
  2. Bateman, A. W., & Fonagy, P. Psychotherapy for severe personality disorder. Article did not do justice to available research data. BMJ (Clinical Research Ed.). 1999;319(7211):709–710; author reply 710–711. · DOI
  3. Biocic M, Fidahic M, Puljak L. Reproducibility of search strategies of non-Cochrane systematic reviews published in anaesthesiology journals is suboptimal: Primary methodological study. Br J Anaesth. 2019;122(6):e79–81. https://doi.org/10.1016/j.bja.2019.02.014. · DOI
  4. Bloodgood, M., & Vijay-Shanker, K. A method for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. ArXiv Preprint ArXiv:1409.5165. 2014. · arXiv
  5. Cassidy LC, Leenaars CHC, Rincon AV, Pfefferle D. Comprehensive search filters for retrieving publications on nonhuman primates for literature reviews (filterNHP). Am J Primatol. 2021;83(7): e23287. https://doi.org/10.1002/ajp.23287. · DOI
  6. Franceschini F, Maisano D, Mastrogiacomo L. The museum of errors/horrors in Scopus. J Informet. 2016;10(1):174–82. https://doi.org/10.1016/j.joi.2015.11.006. · DOI
  7. Gmel G, Venzin V, Marmet K, Danko G, Labhart F. A quasi-randomized group trial of a brief alcohol intervention on risky single occasion drinking among secondary school students. Int J Public Health. 2012;57(6):935–44. https://doi.org/10.1007/s00038-012-0419-0. · DOI
  8. Ipeirotis PG, Provost F, Sheng VS, Wang J. Repeated labeling using multiple noisy labelers. Data Min Knowl Disc. 2014;28(2):402–41. https://doi.org/10.1007/s10618-013-0306-1. · DOI
  9. Linehan MM, Heard HL, Armstrong HE. Naturalistic follow-up of a behavioral treatment for chronically parasuicidal borderline patients. Arch Gen Psychiatry. 1993;50(12):971–4. https://doi.org/10.1001/archpsyc.1993.01820240055007. · DOI · Paper page
  10. López-Nicolás R, López-López JA, Rubio-Aparicio M, Sánchez-Meca J. A meta-review of transparency and reproducibility-related reporting practices in published meta-analyses on clinical psychological interventions (2000–2020). Behav Res Methods. 2022;54(1):334–49. https://doi.org/10.3758/s13428-021-01644-z. · DOI
  11. Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Acad Med. 2011;86(8):1049–54. https://doi.org/10.1097/ACM.0b013e31822221e7. · DOI · Paper page
  12. McMain SF, Guimond T, Streiner DL, Cardish RJ, Links PS. Dialectical behavior therapy compared with general psychiatric management for borderline personality disorder: Clinical outcomes and functioning over a 2-year follow-up. Am J Psychiatry. 2012;169(6):650–61. https://doi.org/10.1176/appi.ajp.2012.11091416. · DOI
  13. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ. 2009;339: b2535. https://doi.org/10.1136/bmj.b2535. · DOI · Paper page
  14. Neeleman, R., Oud, M., Weijdema, F., Leenaars, C., & Schoot, R. van de. Scripts, data and output to reproduce ‘Addressing the Challenges of Reconstructing Systematic Reviews Datasets: A Case Study and a Noisy Label Filter Procedure’. (2022). https://doi.org/10.17605/OSF.IO/PJR97 · DOI
  15. Oud M, Arntz A, Hermens ML, Verhoef R, Kendall T. Specialized psychotherapies for adults with borderline personality disorder: A systematic review and meta-analysis. Aust N Z J Psychiatry. 2018;52(10):949–61. https://doi.org/10.1177/0004867418791257. · DOI · Paper page
  16. Peterson MH, Barnason S, Donnelly B, Hill K, Miley H, Riggs L, Whiteman K. Choosing the best evidence to guide clinical practice: application of AACN levels of evidence. Crit Care Nurse. 2014;34(2):58–68. https://doi.org/10.4037/ccn2014411. · DOI
  17. Teijema, J., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. Simulation-based active learning for systematic reviews: a systematic review of the literature. 2023. https://doi.org/10.31234/osf.io/67zmt · DOI · Paper page
  18. Van den Bosch LMC. Efficacy of dialectical behaviour therapy in the treatment of female borderline patients with and without substance abuse problems: Result of a Dutch study. Dialectische gedragstherapie bij Nederlandse vrouwen met een borderline persoonlijkheidsstoornis, met en zonder verslavingsproblemen. 2005;47(3):127–37.
  19. Van den Bosch LMC, Koeter MWJ, Stijnen T, Verheul R, Van den Brink W. Sustained efficacy of dialectical behaviour therapy for borderline personality disorder. Behav Res Ther. 2005;43(9):1231–41. https://doi.org/10.1016/j.brat.2004.09.008. · DOI
  20. van der Mierden S, Hooijmans CR, Tillema AH, Rehn S, Bleich A, Leenaars CH. Laboratory animals search filter for different literature databases: PubMed, Embase, Web of Science and PsycINFO. Lab Anim. 2022;56(3):279–86. https://doi.org/10.1177/00236772211045485. · DOI
  21. Verheul R, van den Bosch LMC, Koeter MWJ, de Ridder MAJ, Stijnen T, van den Brink W. Dialectical behaviour therapy for women with borderline personality disorder: 12-month, randomised clinical trial in The Netherlands. Br J Psychiatry. 2003;182(2):135–40. https://doi.org/10.1192/bjp.182.2.135. · DOI
  22. Vlachos A. A stopping criterion for active learning. Comput Speech Lang. 2008;22(3):295–312. · DOI · Paper page
  23. Yang, E., Lewis, D. D., & Frieder, O. Heuristic stopping rules for technology-assisted review. Proceedings of the 21st ACM Symposium on Document Engineering 2021:1–10. · DOI

Cited by

Metadata

Authors
Neeleman, R., Leenaars, C. H. C., Oud, M., Weijdema, F., & van de Schoot, R.
ORCID
Neeleman, Rutger
Venue
Systematic Reviews
Publisher
Springer Science and Business Media LLC
Published
2024-02-17
Volume
13
Issue
1
ISSN
2046-4053
DOI
10.1186/s13643-024-02472-w
PMC
PMC10874047
Cited by
12
References
12
Language
en
Licence
Funders
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (406.22.GO.048)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.