The efficiency of machine learning-assisted platform for article screening in systematic reviews in orthopaedics

Abstract

Purpose With the development of machine learning and artificial intelligence, various platforms were developed to aid in the time-consuming process of article screening in systematic reviews. We aim to analyze the efficiency of a machine learning-assisted platform as an end-user to aid in the screening of the articles for selection into systematic review in orthopaedic surgery. Methods We included three previously published systematic reviews in the field of orthopaedics of increasing levels of difficulty in the structure of the research question to assess the efficiency of a platform with active-learning technology for article screening. We compared the efficiency of the platform compared to the traditional screening and also across the various scenarios tested. We performed five iterations for each review analyzed. The outcome parameters analyzed were the work saved at 95% recall (WSS-95), work saved at 100% recall (WSS-100), and relevant records found after screening the first 30% of the total records (RRF-30). Results The machine learning-assisted screening significantly improved the rate of identifying the relevant records compared to the traditional screening method (p<0.001). The WSS-95 for the easy, intermediate, and advanced screening scenarios were 78%, 59%, and 38%, respectively. The WSS-100 for the easy, intermediate, and advanced screening scenarios were 75%, 48%, and 7%, respectively. The RRF-30 for the easy, intermediate, and advanced screening scenarios were 97%, 86%, and 64%, respectively. We noted a significant reduction (p<0.001) in the efficiency with the increasing level of difficulty of the screening scenarios. Conclusion The machine learning platform is significantly better than the traditional method as an assistive technology to aid in article screening. However, the efficiency of the platform significantly decreases as the complexity of the research question increases.