The DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index.
A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction
Open accessAbstract
Context: Software defect prediction (SDP) is an important challenge in the field of software engineering, hence much research work has been conducted, most notably through the use of machine learning algorithms. However, class-imbalance typified by few defective components and many non-defective ones is a common occurrence causing difficulties for these methods. Imbalanced learning aims to deal with this problem and has recently been deployed by some researchers, unfortunately with inconsistent results. Objective: We conduct a comprehensive experiment to explore (a) the basic characteristics of this problem; (b) the effect of imbalanced learning and its interactions with (i) data imbalance, (ii) type of classifier, (iii) input metrics and (iv) imbalanced learning method. Method: We systematically evaluate 27 data sets, 7 classifiers, 7 types of input metrics and 17 imbalanced learning methods (including doing nothing) using an experimental design that enables exploration of interactions between these factors and individual imbalanced learning algorithms. This yields 27 × 7 × 7 × 17 = 22491 results. The Matthews correlation coefficient (MCC) is used as an unbiased performance measure (unlike the more widely used F1 and AUC measures). Results: (a) we found a large majority (87 percent) of 106 public domain data sets exhibit moderate or low level of imbalance (imbalance ratio <; 10; median = 3.94); (b) anything other than low levels of imbalance clearly harm the performance of traditional learning for SDP; (c) imbalanced learning is more effective on the data sets with moderate or higher imbalance, however negative results are always possible; (d) type of classifier has most impact on the improvement in classification performance followed by the imbalanced learning method itself. Type of input metrics is not influential. (e) only 52% of the combinations of Imbalanced Learner and Classifier have a significant positive effect. Conclusion: This paper offers two practical guidelines. First, imbalanced learning should only be considered for moderate or highly imbalanced SDP data sets. Second, the appropriate combination of imbalanced method and classifier needs to be carefully chosen to ameliorate the imbalanced learning problem for SDP. In contrast, the indiscriminate application of imbalanced learning can be harmful.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 1×
“Lastly, our experimental program is based onWEKA [61].”
MethodBreiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Structural Holes: The Social Structure of Competition
Lazega, E., & Burt, R. S. (1995) · Revue Française de Sociologie · doi:10.2307/3322456 · Cited by 7,853
A metrics suite for object oriented design
Chidamber, S. R., & Kemerer, C. F. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.295895 · Cited by 5,669
Top 10 algorithms in data mining
Wu, X., Kumar, V., Ross Quinlan, J., Ghosh, J., Yang, Q., Motoda, H., McLachlan, G. J., Ng, A., Liu, B., Yu, P. S., et al. (2008) · Knowledge and Information Systems · doi:10.1007/s10115-007-0114-2 · Cited by 5,667 · cited 1×
“AdaBoost [16] is the most commonly used boosting method, and was identified as one of the top ten most influential data mining algorithms [29].”
MethodEvaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,290
A study of the behavior of several methods for balancing machine learning training data
Batista, G. E. A. P. A., Prati, R. C., & Monard, M. C. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007735 · Cited by 4,203
Cohen, W. W. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50023-2 · Cited by 3,787
Galar, M., Fernandez, A., Barrenechea, E., Bustince, H., & Herrera, F. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2011.2161285 · Cited by 2,794
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
Instance-based learning algorithms
Aha, D. W., Kibler, D., & Albert, M. K. (1991) · Machine Learning · doi:10.1007/bf00153759 · Cited by 2,163
Assessing the accuracy of prediction algorithms for classification: an overview
Baldi, P., Brunak, S., Chauvin, Y., Andersen, C. A. F., & Nielsen, H. (2000) · Bioinformatics · doi:10.1093/bioinformatics/16.5.412 · Cited by 2,111
Editorial: special issue on learning from imbalanced data sets
Chawla, N. V., Japkowicz, N., & Kotcz, A. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007733 · Cited by 2,063
Introduction to Robust Estimation and Hypothesis Testing
(2012) · Elsevier eBooks · doi:10.1016/c2010-0-67044-1 · Cited by 2,055
RUSBoost: A Hybrid Approach to Alleviating Class Imbalance
Seiffert, C., Khoshgoftaar, T. M., Van Hulse, J., & Napolitano, A. (2010) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/tsmca.2009.2029559 · Cited by 1,867
CLASSIFICATION OF IMBALANCED DATA: A REVIEW
SUN, Y., WONG, A. K. C., & KAMEL, M. S. (2009) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001409007326 · Cited by 1,692
SMOTEBoost: Improving Prediction of the Minority Class in Boosting
Chawla, N. V., Lazarevic, A., Hall, L. O., & Bowyer, K. W. (2003) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-39804-2_12 · Cited by 1,668
López, V., Fernández, A., García, S., Palade, V., & Herrera, F. (2013) · Information Sciences · doi:10.1016/j.ins.2013.07.007 · Cited by 1,590
Dominance statistics: Ordinal analyses to answer ordinal questions.
Cliff, N. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.114.3.494 · Cited by 1,485
MetaCost: a general method for making classifiers cost-sensitive
Domingos, P. (1999) · Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/312129.312220 · Cited by 1,303
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 3×
“In order to be as comprehensive as possible, we apply a total of 17 different imbalanced learning methods (16 plus a null method—see Table 5) to seven traditional classifiers chosen to be representative of commonly used approaches [1] (see Table 6).”
BackgroundMethodA Survey of Predictive Modeling on Imbalanced Domains
Branco, P., Torgo, L., & Ribeiro, R. P. (2017) · ACM Computing Surveys · doi:10.1145/2907070 · Cited by 1,129 · cited 2×
“For a fuller discussion of the merits and demerits of various classification performance metrics see [10], [21], [52].”
BackgroundThe Case against Accuracy Estimation for Comparing Induction Algorithms
Provost, F. J., Fawcett, T. E., & Kohavi, R. (1998) · Cited by 1,031
Measuring classifier performance: a coherent alternative to the area under the ROC curve
Hand, D. J. (2009) · Machine Learning · doi:10.1007/s10994-009-5119-5 · Cited by 1,003
Structural Holes: The Social Structure of Competition.
Andrews, S. B., & Burt, R. S. (1995) · Administrative Science Quarterly · doi:10.2307/2393644 · Cited by 988
Predicting Defects for Eclipse
Zimmermann, T., Premraj, R., & Zeller, A. (2007) · Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007) · doi:10.1109/promise.2007.10 · Cited by 753
A large-scale empirical study of just-in-time quality assurance
Kamei, Y., Shihab, E., Adams, B., Hassan, A. E., Mockus, A., Sinha, A., & Ubayashi, N. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.70 · Cited by 715
Moser, R., Pedrycz, W., & Succi, G. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368114 · Cited by 701
Automatically learning semantic features for defect prediction
Wang, S., Liu, T., & Tan, L. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884804 · Cited by 689
The PROMISE Repository of Empirical Software Engineering Data
Boetticher, G. D. (2007) · Medical Entomology and Zoology · Cited by 682
Evaluating defect prediction approaches: a benchmark and an extensive comparison
D’Ambros, M., Lanza, M., & Robbes, R. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9173-9 · Cited by 626
A systematic review of machine learning techniques for software fault prediction
Malhotra, R. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2014.11.023 · Cited by 620
Shin, Y., Meneely, A., Williams, L., & Osborne, J. A. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.81 · Cited by 607
Predicting defects using network analysis on dependency graphs
Zimmermann, T., & Nagappan, N. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368161 · Cited by 580
Using Class Imbalance Learning for Software Defect Prediction
Wang, S., & Yao, X. (2013) · IEEE Transactions on Reliability · doi:10.1109/tr.2013.2259203 · Cited by 571
Data Quality: Some Comments on the NASA Software Defect Datasets
Shepperd, M., Song, Q., Sun, Z., & Mair, C. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.11 · Cited by 570
Self-citationDiversity analysis on imbalanced data sets by using ensemble models
Wang, S., & Yao, X. (2009) · 2009 IEEE Symposium on Computational Intelligence and Data Mining · doi:10.1109/cidm.2009.4938667 · Cited by 548
Software fault prediction metrics: A systematic literature review
Radjenović, D., Heričko, M., Torkar, R., & Živkovič, A. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.02.009 · Cited by 538
An instance-weighting method to induce cost-sensitive trees
Ting, K. M. (2002) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2002.1000348 · Cited by 504
Ramentol, E., Caballero, Y., Bello, R., & Herrera, F. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0465-6 · Cited by 489
ReLink: recovering links between bugs and changes
Wu, R., Zhang, H., Kim, S., & Cheung, S.-C. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025120 · Cited by 412
A cost-sensitive decision tree approach for fraud detection
Sahin, Y., Bulkan, S., & Duman, E. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.05.021 · Cited by 381
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364
Researcher Bias: The Use of Machine Learning in Software Defect Prediction
Shepperd, M., Bowes, D., & Hall, T. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2322358 · Cited by 359
Self-citationConcept learning and the problem of small disjuncts
Holte, R. C., Acker, L., & Porter, B. (1989) · Cited by 351
Catal, C., & Diri, B. (2009) · Information Sciences · doi:10.1016/j.ins.2008.12.001 · Cited by 343
An investigation on the feasibility of cross-project defect prediction
He, Z., Shu, F., Yang, Y., Li, M., & Wang, Q. (2012) · Automated Software Engineering · doi:10.1007/s10515-011-0090-3 · Cited by 333
Software defect prediction using Bayesian networks
Okutan, A., & Yıldız, O. T. (2014) · Empirical Software Engineering · doi:10.1007/s10664-012-9218-8 · Cited by 330
SPECIAL ISSUE ON LEARNING FROM IMBALANCED DATA SETS
Chawla, N. V., Japkowicz, N., & Kołcz, A. (2004) · Cited by 325
Classifier ensembles: Select real-world applications
Oza, N. C., & Tumer, K. (2008) · Information Fusion · doi:10.1016/j.inffus.2007.07.002 · Cited by 313
An empirical study on software defect prediction with a simplified metric set
He, P., Li, B., Liu, X., Chen, J., & Ma, Y. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2014.11.006 · Cited by 309
New Applications of Ensembles of Classifiers
Barandela, R., Valdovinos, R. M., & S�nchez, J. S. (2003) · Pattern Analysis & Applications · doi:10.1007/s10044-003-0192-z · Cited by 280
Menzies, T., Dekhtyar, A., Distefano, J., & Greenwald, J. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70721 · Cited by 277
Cost-Sensitive Learning with Neural Networks.
Kukar, M., & Kononenko, I. (1998) · Cited by 273
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · Cited by 256
Precision-Recall-Gain Curves: PR Analysis Done Right
Flach, P. A., & Kull, M. (2015) · Bristol Research (University of Bristol) · doi:10.5555/2969239.2969333 · Cited by 256
A weighted voting framework for classifiers ensembles
Kuncheva, L. I., & Rodríguez, J. J. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-012-0586-6 · Cited by 254
How, and why, process metrics are better
Rahman, F., & Devanbu, P. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606589 · Cited by 248
Reducing Features to Improve Code Change-Based Bug Prediction
Shivaji, S., Whitehead, E. J., Akella, R., & Kim, S. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.43 · Cited by 247
Cross-project defect prediction using a connectivity-based unsupervised classifier
Zhang, F., Zheng, Q., Zou, Y., & Hassan, A. E. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884839 · Cited by 245
Class Imbalance Learning Methods for Support Vector Machines
Batuwita, R., & Palade, V. (2013) · Imbalanced Learning · doi:10.1002/9781118646106.ch5 · Cited by 227
The Effects of Over and Under Sampling on Fault-prone Module Detection
Kamei, Y., Monden, A., Matsumoto, S., Kakimoto, T., & Matsumoto, K. (2007) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2007.28 · Cited by 192
Graph-based analysis and prediction for software evolution
Bhattacharya, P., Iliofotou, M., Neamtiu, I., & Faloutsos, M. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227173 · Cited by 188
Using Coding-Based Ensemble Learning to Improve Software Defect Prediction
Sun, Z., Song, Q., & Zhu, X. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2012.2226152 · Cited by 187
Self-citationImplications of ceiling effects in defect predictors
Menzies, T., Turhan, B., Bener, A., Gay, G., Cukic, B., & Jiang, Y. (2008) · Proceedings of the 4th international workshop on Predictor models in software engineering · doi:10.1145/1370788.1370801 · Cited by 185
How, and Why, Process Metrics are better
Rahman, F., & Dévanbu, P. (2013) · Cited by 184
Seiffert, C., Khoshgoftaar, T. M., Hulse, J. V., & Folleco, A. (2011) · Information Sciences · doi:10.1016/j.ins.2010.12.016 · Cited by 176
Giger, E., D'Ambros, M., Pinzger, M., & Gall, H. C. (2012) · Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement · doi:10.1145/2372251.2372285 · Cited by 164
Applying Novel Resampling Strategies To Software Defect Prediction
Pelayo, L., & Dick, S. (2007) · NAFIPS 2007 - 2007 Annual Meeting of the North American Fuzzy Information Processing Society · doi:10.1109/nafips.2007.383813 · Cited by 158
Graph-based analysis and prediction for software evolution
Bhattacharya, P., Iliofotou, M., Neamtiu, I., & Faloutsos, M. (2012) · doi:10.5555/2337223.2337273 · Cited by 152
Towards Effective Bug Triage with Software Data Reduction Techniques
Xuan, J., Jiang, H., Hu, Y., Ren, Z., Zou, W., Luo, Z., & Wu, X. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2324590 · Cited by 150
Warrens, M. J. (2008) · Psychometrika · doi:10.1007/s11336-008-9070-3 · Cited by 150
Improving Software-Quality Predictions With Data Sampling and Boosting
Seiffert, C., Khoshgoftaar, T. M., & Hulse, J. V. (2009) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2009.2027131 · Cited by 113
Foundations of Imbalanced Learning
Weiss, G. M. (2013) · Wiley eBooks · doi:10.1002/9781118646106.ch2 · Cited by 109
Validation of network measures as indicators of defective modules in software systems
Tosun, A., Turhan, B., & Bener, A. (2009) · doi:10.1145/1540438.1540446 · Cited by 93
(2007) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2007.80 · Cited by 91
A Genetic Algorithm to Configure Support Vector Machines for Predicting Fault-Prone Components
Martino, S. D., Ferrucci, F., Gravino, C., & Sarro, F. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21843-9_20 · Cited by 85
Application of Random Forest in Predicting Fault-Prone Classes
Kaur, A., & Malhotra, R. (2008) · International Conference on Advanced Computer Theory and Engineering · doi:10.1109/icacte.2008.204 · Cited by 85
Network Versus Code Metrics to Predict Defects: A Replication Study
Premraj, R., & Herzig, K. (2011) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.30 · Cited by 82
Ordinal Analysis of Behavioral Data
Long, J. D., Du, F., & Cliff, N. (2003) · Wiley eBooks · doi:10.1002/0471264385.wei0225 · Cited by 58
Cost-Sensitive Classification with k-Nearest Neighbors
Qin, Z., Wang, A. T., Chengqi Zhang, B., & Zhang, S. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39787-5_10 · Cited by 56
Predicting Faults in High Assurance Software
Seliya, N., Khoshgoftaar, T. M., & Hulse, J. V. (2010) · IEEE International Symposium on High Assurance Systems Engineering (HASE) · doi:10.1109/hase.2010.29 · Cited by 54
Imbalanced evolving self-organizing learning
Cai, Q., He, H., & Man, H. (2014) · Neurocomputing · doi:10.1016/j.neucom.2013.11.010 · Cited by 37
Cost-Sensitive Learning of SVM for Ranking
Xu, J., Cao, Y., Li, H., & Huang, Y. (2006) · Lecture notes in computer science · doi:10.1007/11871842_86 · Cited by 33
Cited by
Resampling imbalanced data for network intrusion detection datasets
Bagui, S., & Li, K. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00390-x · Cited by 287
Predicting Design Impactful Changes in Modern Code Review: A Large-Scale Empirical Study
Uchoa, A., Barbosa, C., Coutinho, D., Oizumi, W., Assuncao, W. K. G., Vergilio, S. R., Pereira, J. A., Oliveira, A., & Garcia, A. (2021) · 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr52588.2021.00059 · Cited by 42
“Secondly, before training the models, we dealt with imbalanced data, a common issue with software engineering data [51].”
Metadata
- Authors
- Song, Q., Guo, Y., & Shepperd, M.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2019-12-01
- Volume
- 45
- Issue
- 12
- Pages
- 1253-1269
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2805001156
- Cited by
- 295
- References
- 90
- FWCI
- 44.59
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.