Finding better active learners for faster literature reviews
Open accessThe DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
Literature reviews can be time-consuming and tedious to complete. By cataloging and refactoring three state-of-the-art active learning techniques from evidence-based medicine and legal electronic discovery, this paper finds and implements FASTREAD, a faster technique for studying a large corpus of documents, combining and parametrizing the most efficient active learning algorithms. This paper assesses FASTREAD using datasets generated from existing SE literature reviews (Hall, Wahono, Radjenović, Kitchenham et al.). Compared to manual methods, FASTREAD lets researchers find 95% relevant studies after reviewing an order of magnitude fewer papers. Compared to other state-of-the-art automatic methods, FASTREAD reviews 20–50% fewer studies while finding same number of relevant primary studies in a systematic literature review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In another study [11] that did use active learning, the Software dataset was used for simulation and a WSS@95 value of 91% was reached”
From Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records (2023) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Cases with high percentages of inclusions have been underrepresented in previous benchmarking analyses of active learning software 9 -11,33 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [33]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Therefore, researchers may either stop too early and risk missing essential records or continue for too long and incur unnecessary additional reading [54].”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [54]
The cited source says what the claim attributes to it.
In the citing paper
“Such recall plots offer insight in model performance throughout the entire screening process [11, 13].”
The cited source says what the claim attributes to it.
In the citing paper
“ready-to-use software tools implementing such models (Abstrackr [22], Colandr [23], FASTREAD [11], Rayyan [24], and RobotAnalyst [25]) currently use the same classification technique to predict relevance of publications, namely support vector machines (SVM).”
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Distributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934 · cited 1×
“2003), paragraph vectors (Le and Mikolov 2014) require further exploration and are scheduled in our future works.”
Background(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 5×
“Jalali and Wohlin (Jalali and Wohlin 2012; Wohlin 2014) applied backward snowballing to search for primary studies in SE SLRs and found comparably good result as database search.”
ResultInfluentialTraining linear SVMs in linear time
Joachims, T. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150429 · Cited by 1,920 · cited 17×
“The algorithm utilizes uncertainty sampling to build the classifier, where unlabeled examples closest to the SVM decision plane are sampled for query.”
MethodInfluentialSystematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230
Settles, B. (2012) · Cited by 1,183
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 3×
“studies (Hall et al. 2012; Wahono 2015; Radjenović et al. 2013) and one dataset provided by Kitchenham et al.”
BackgroundMethodA systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094 · cited 1×
“In the SE research community, linear manual review, which requires reviewer to review every candidate study, is still the standard approach for primary study selection [31].”
BackgroundO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 3×
“To facilitate citation screening (primary study selection) in systematic review, many groups of researchers have investigated different types of machine learning algorithms and evaluation mechanisms [49,50].”
BackgroundSystematic literature studies: database searches vs. backward snowballing
Jalali, S., & Wohlin, C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372257 · Cited by 659 · cited 8×
“Jalali and Wohlin (Jalali and Wohlin 2012; Wohlin 2014) applied backward snowballing to search for primary studies in SE SLRs and found comparably good result as database search.”
MethodResultInfluential(2004) · doi:10.5555/998675.999432 · Cited by 656
A systematic review of machine learning techniques for software fault prediction
Malhotra, R. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2014.11.023 · Cited by 620 · cited 1×
“There can be variances on the real implementations [60,38,51,59,23], but all are based on the same guideline [30].”
BackgroundSettles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.2200/s00429ed1v01y201207aim018 · Cited by 605
Identifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 557
Software fault prediction metrics: A systematic literature review
Radjenović, D., Heričko, M., Torkar, R., & Živkovič, A. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.02.009 · Cited by 538
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532 · cited 2×
“al knowledge and equally expensive reviewers. Details on these two approaches are provided in Section 4. There are also existing tools to support study selection in systematic reviews, e.g. Abstrakr1 [59], EPPI-Reviewer2 [54], Rayaan3 [45]. Useful features can be found in these tools such as a) Rayaan and EPPI-Reviewe…”
BackgroundMethodEvidence-based software engineering for practitioners
Dyba, T., Kitchenham, B. A., & Jorgensen, M. (2005) · IEEE Software · doi:10.1109/ms.2005.6 · Cited by 463
Evidence-based software engineering
Kitchenham, B. A., Dyba, T., & Jorgensen, M. (2004) · Proceedings. 26th International Conference on Software Engineering · doi:10.1109/icse.2004.1317449 · Cited by 455 · cited 3×
“Given the inherent relevance relationship between a study and its citations, it is of high probability for the citations of (used in backward snowballing) and the studies cite (used in forward snowballing) a known “relevant” study to also be “relevant” (Kitchenham et al. 2004).”
BackgroundMethodInfluentialReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 4×
“…mining into citation screening as well as the proposed performance metrics of Cohen has been widely acknowledged [49], most of Cohen’s work focused on supervised learning which does not utilize unlabeled data and relies on random sampling to obtain the sufficiently large training set [10,7,9,8].”
InfluentialO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 9×
“– Review those active learning methods [10, 11, 45, 58, 60] and find that in evidence-based medicine and legal electronic discovery, there are three widely recognized state-of-the-art active learning methods [10, 38, 60].”
BackgroundMethodInfluentialA Visual Text Mining approach for Systematic Reviews
Malheiros, V., Hohn, E., Pinho, R., Mendonca, M., & Maldonado, J. C. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.21 · Cited by 329
Evaluation and Measurement of Software Process Improvement—A Systematic Literature Review
Unterkalmsteiner, M., Gorschek, T., Islam, A. K. M. M., Cheng, C. K., Permadi, R. B., & Feldt, R. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.26 · Cited by 291
Tuning for software analytics: Is it really necessary?
Fu, W., Menzies, T., & Shen, X. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.017 · Cited by 210
Self-citationReducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
Validity threats in empirical software engineering research - An initial survey
FELDT, R. H., & Magazinius, A. (2010) · Chalmers Publication Library (Chalmers University of Technology) · Cited by 188
Using Software Dependencies and Churn Metrics to Predict Field Failures: An Empirical Case Study
Nagappan, N., & Ball, T. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.13 · Cited by 176 · cited 3×
“All these techniques (such as Quasi-Gold Standard based search [63, 64], visual text mining [15,17,18,34], and snowballing [25,61]) are compatible with works in this paper and a better performance is expected when applied together.”
BackgroundMethodShemilt, I., Khan, N., Park, S., & Thomas, J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0315-4 · Cited by 168 · cited 3×
“Similar conclusion was derived by Shemilt et al. in 2016 [56].”
BackgroundMethodResultEvaluation of machine-learning protocols for technology-assisted review in electronic discovery
Cormack, G. V., & Grossman, M. R. (2014) · Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval · doi:10.1145/2600428.2609601 · Cited by 149 · cited 5×
“2014) (Certainty plus Weighting), and Cormack’14 (Cormack and Grossman 2014) (continuous active learning).”
BackgroundMethodInfluentialUsing GQM and TAM to evaluate StArt – a tool that supports Systematic Review
Hernandes, E., Zamboni, A., Fabbri, S., & Di Thommazo, A. (2012) · CLEI Electronic Journal · doi:10.19153/cleiej.15.1.2 · Cited by 146 · cited 7×
“2012; Molléri and Benitti 2015), data extraction and validation (Molléri and Benitti 2015; Hernandes et al. 2012; Fernández-Sáez et al. 2010; Bowes et al. 2012), data synthesis (Molléri and Benitti 2015; Hernandes et al.”
BackgroundMethodInfluentialAutomatic text classification to support systematic reviews in medicine
García Adeva, J. J., Pikatza Atxa, J. M., Ubeda Carrillo, M., & Ansuategi Zengotitabengoa, E. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.047 · Cited by 125
Active learning for biomedical citation screening
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835829 · Cited by 123 · cited 3×
“– Review those active learning methods [10, 11, 45, 58, 60] and find that in evidence-based medicine and legal electronic discovery, there are three widely recognized state-of-the-art active learning methods [10, 38, 60].”
BackgroundMethodWahono (2007) · doi:10.3923/jse.2007.1.12 · Cited by 113 · cited 2×
“e experiments running on three software engineering SLR Finding Better Active Learners for Faster Literature Reviews 19 datasets created with information from Hall, Wahono, Radjenovi´c et al. studies [23,48,56] and one dataset provided by Kitchenham [31]. Therefore, such conclusions may not be applicable to datasets o…”
MethodUsing Forward Snowballing to update Systematic Reviews in Software Engineering
Felizardo, K. R., Mendes, E., Kalinowski, M., Souza, É. F., & Vijaykumar, N. L. (2016) · Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962630 · Cited by 102 · cited 1×
“[16] and Wohlin [62] applied forward snowballing to update SE SLRs and greatly reduced the number studies need to be reviewed comparing to”
BackgroundIdentifying Barriers to the Systematic Literature Review Process
Carver, J. C., Hassler, E., Hernandes, E., & Kraft, N. A. (2013) · 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.28 · Cited by 90 · cited 3×
“Previously we have analyzed the costs of SLRs (Hassler et al. 2014; Carver et al. 2013).”
BackgroundSelf-citationTools to Support Systematic Literature Reviews in Software Engineering: A Mapping Study
Marshall, C., & Brereton, P. (2013) · 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.32 · Cited by 87
University of Wisconsin at Madison
Tölle, W., Yasner, J., & Pieper, M. (1993) · Springer eBooks · doi:10.1007/978-3-642-77393-8_43 · Cited by 80
A Machine Learning Approach for Semi-Automated Search and Selection in Literature Studies
Ros, R., Bjarnason, E., & Runeson, P. (2017) · Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3084226.3084243 · Cited by 80 · cited 1×
“jobs to multiple reviewers or to resolving disagreements). Hence, we planned to introduce machine learning to assist primary study selection in SE SLRs but before this paper is published, Ros et al. [50] has achieved this in June 2017. While Ros’17 [50] provided a wide range of techniques to support both search and se…”
MethodCombining Crowd and Expert Labels Using Decision Theoretic Active Learning
Nguyen, A., Wallace, B., & Lease, M. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13225 · Cited by 73 · cited 2×
“cused on other aspects of machine learning assisted citation screening such as introducing external expert knowledge [60], allocating review tasks to multiple experts [61] or to crowdsourcing workers [42], and building a tool called abstrackr to provide overall support [59]. Wallace’s work on this topic is of exemplar…”
BackgroundInfluentialSecond-generation systematic literature studies using snowballing
Wohlin, C. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916006 · Cited by 69 · cited 1×
“owballing to search for primary studies in SE SLRs and found comparably good Finding Better Active Learners for Faster Literature Reviews 5 result as database search. Felizardo et al. [17] and Wohlin [64] applied forward snowballing to update SE SLRs and greatly reduced the number studies need to be reviewed comparing…”
BackgroundWho Should Label What? Instance Allocation in Multiple Expert Active Learning
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2011) · Proceedings of the 2011 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972818.16 · Cited by 63
Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review
Cormack, G. V., & Grossman, M. R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06868 · Cited by 62 · cited 10×
“The algorithm stops random sampling as long as ONE “relevant” study is retrieved, as suggested in Cormack’15 [10, 11].”
BackgroundMethodInfluentialA visual analysis approach to validate the selection review of primary studies in systematic reviews
Felizardo, K. R., Andery, G. F., Paulovich, F. V., Minghim, R., & Maldonado, J. C. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.04.003 · Cited by 60 · cited 1×
“All these techniques (such as Quasi-Gold Standard based search [63, 64], visual text mining [15,17,18,34], and snowballing [25,61]) are compatible with works in this paper and a better performance is expected when applied together.”
MethodTREC 2015 Total Recall Track Overview.
Roegiest, A., Cormack, G. V., Clarke, C. L. A., & Grossman, M. R. (2015) · Text REtrieval Conference · Cited by 58
Tools to support systematic reviews in software engineering: a feature analysis
Marshall, C., Brereton, P., & Kitchenham, B. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601270 · Cited by 56
Olorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54
Hassler, E., Carver, J. C., Kraft, N. A., & Hale, D. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601274 · Cited by 53 · cited 1×
“Previously we have analyzed the costs of SLRs (Hassler et al. 2014; Carver et al. 2013).”
BackgroundSelf-citationModernizing the systematic review process to inform comparative effectiveness: tools and methods
Wallace, B. C., Dahabreh, I. J., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2013) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.13.17 · Cited by 51 · cited 1×
“Wallace et al. conducted a series of studies with machine learning techniques, especially active learning [66,64,65,63,61,62,45].”
BackgroundMarshall, C., Brereton, P., & Kitchenham, B. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745827 · Cited by 46
An effective general purpose approach for automated biomedical document classification.
(2006) · Europe PMC (PubMed Central) · Cited by 44
SLR-TOOL - A Tool for Performing Systematic Literature Reviews
(2010) · Proceedings of the 5th International Conference on Software and Data Technologies · doi:10.5220/0003003601570166 · Cited by 42 · cited 2×
“These tools aim at providing support for protocol development [44,21,26], automated search [44,26], primary study selection [44,26,21,4], quality assessment [21,4,44], data extraction and validation [44,26,21,4], data synthesis [44, 26,21,4], and report write up [44,26,21,4].”
BackgroundMethodOlorisade, B. K., de Quincey, E., Brereton, P., & Andras, P. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2915982 · Cited by 42 · cited 1×
“pe that other researchers will explore similar work on different datasets). Existing active learning techniques in citation screening have been criticized by Olorisade et al. for being not replicable [43,44]. To this end, we have published all our code at https://github.com/fastread/src and all our data at https://doi…”
BackgroundInfluentialCohen, A. M., Ambert, K. H., & McDonagh, M. S. (2010) · PubMed Central · Cited by 41
Liu, J., Timsina, P., & El-Gayar, O. (2016) · Information Systems Frontiers · doi:10.1007/s10796-016-9724-0 · Cited by 41
SLuRp: a tool to help large complex systematic literature reviews deliver valid and rigorous results
Bowes, D., Hall, T., & Beecham, S. (2012) · Proceedings of the 2nd international workshop on Evidential assessment of software technologies · doi:10.1145/2372233.2372243 · Cited by 39 · cited 7×
“Usually, reviewers need to evaluate thousands of studies trying to find dozens of them that are relevant to the research questions based on their title, abstract, or full text (Bowes et al. 2012).”
BackgroundMethodInfluentialFelizardo, K. R., Nakagawa, E. Y., Feitosa, D., Minghim, R., & Maldonado, J. C. (2010) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2010.5 · Cited by 35 · cited 1×
“All these techniques (such as Quasi-Gold Standard based search [69,70], visual text mining [20,17,19,37], and snowballing [67,28]) are compatible and a better performance is expected when applied together.”
MethodIdentification of SLR tool needs – results of a community workshop
Hassler, E., Carver, J. C., Hale, D., & Al-Zubidy, A. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2015.10.011 · Cited by 35
A visual analysis approach to update systematic reviews
Felizardo, K. R., Nakagawa, E. Y., MacDonell, S. G., & Maldonado, J. C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601252 · Cited by 33
Cohen, A. M. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.008177 · Cited by 32 · cited 3×
“Each treatment is evaluated by “work saved over sampling at 95% recall” (WSS@95) [8].”
BackgroundSESRA: a web-based automated tool to support the systematic literature review process
Molléri, J. S., & Benitti, F. B. V. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745825 · Cited by 25
An empirical assessment of a systematic search process for systematic reviews
Zhang, H., Babar, M. A., Bai, X., Li, J., & Huang, L. (2011) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1049/ic.2011.0007 · Cited by 24 · cited 1×
“here are already tools to support other parts of the SLR process, as well as different techniques to facilitate primary study selection. All these techniques (such as Quasi-Gold Standard based search [65,66], visual text mining [16,18,19,36], and snowballing [27,63]) are compatible with works in this paper and a bette…”
MethodA Systematic Review of Software Product Lines Applied to Mobile Middleware
Bezerra, Y. M., Pereira, T. A. B., & da Silveira, G. E. (2009) · International Conference on Information Technology: New Generations · doi:10.1109/itng.2009.82 · Cited by 14
Krishna, R., Yu, Z., Agrawal, A., Dominguez, M., & Wolf, D. (2016) · doi:10.1145/2896825.2896836 · Cited by 12 · cited 2×
“Electronic Discovery (e-discovery) is a part of civil litigation where one party (the producing party), offers up materials which are pertinent to a legal case (Krishna et al. 2016).”
BackgroundMethodSelf-citationTuneR: a framework for tuning software engineering tools with hands-on instructions in R
Borg, M. (2016) · Journal of Software Evolution and Process · doi:10.1002/smr.1784 · Cited by 10
Research works printed but not linked: 3
- 22. Grossman, M.R., Cormack, G.V.: The grossman-cormack glossary of technology-assisted review with foreword by john m. facciola, u.s. magistrate judge. Federal Courts Law Review 7 (1), 1-34 (2013)
- 48. Radjenovi´ c, D., Heriˇ cko, M., Torkar, R., ˇ Zivkoviˇ c, A.: Software fault prediction metrics: A systematic literature review. Information and Software Technology 55 (8), 1397-1418 (2013)
- 57. Wallace, B.C., Dahabreh, I.J., Moran, K.H., Brodley, C.E., Trikalinos, T.A.: Active literature discovery for scoping evidence reviews: How many needles are there. In: KDD workshop on data mining for healthcare (KDD-DMH) (2013)
Cited by
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“Existing literature includes various evaluations of SLR tools that focus on individual phases of the SLR process [21, 139, 140].”
Artificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170
“FASTREAD [34] is a conceptual active learning approach also using SVM as the underlying classifier, which can be “instanti-ated” into 32 different learning models depending on: 1) when to start training, 2) which document to query next, 3) when to stop training, and 4) how to balance the training data.”
BackgroundMethodInfluential(2023) · doi:10.21203/rs.3.rs-2856011/v1 · Cited by 145
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146
“Therefore, researchers may either stop too early and risk missing essential records or continue for too long and incur unnecessary additional reading [54].”
BackgroundMethod
Metadata
- Authors
- Yu, Z., Kraft, N. A., & Menzies, T.
- Affiliations
- North Carolina State University (US)
- Corresponding author
- Menzies, Tim
- Venue
- Empirical Software Engineering
- Published
- 2018-03-07
- Volume
- 23
- Issue
- 6
- Pages
- 3161-3186
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W2785836523
- Cited by
- 83
- References
- 68
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 16 · 2023: 13 · 2022: 10 · 2021: 10 · 2020: 9 · 2019: 7 · 2018: 2
- FWCI
- 5.34
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Empirical Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Clemson University
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- active learning (machine learning), systematic reviews, evidence-based medicine, software engineering literature
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.