CoChat

A Survey on Automated Log Analysis for Reliability Engineering

Open access

The DOI matches its Crossref record; 130 of the 130 references it cites were found in a registry or an index.

Abstract

Logs are semi-structured text generated by logging statements in software source code. In recent decades, software logs have become imperative in the reliability assurance mechanism of many software systems, because they are often the only data available that record software runtime information. As modern software is evolving into a large scale, the volume of logs has increased rapidly. To enable effective and efficient usage of modern software logs in reliability engineering, a number of studies have been conducted on automated log analysis. This survey presents a detailed overview of automated log analysis research, including how to automate and assist the writing of logging statements, how to compress logs, how to parse logs into structured event templates, and how to employ logs to detect anomalies, predict failures, and facilitate diagnosis. Additionally, we survey work that releases open-source toolkits and datasets. Based on the discussion of the recent advances, we present several promising future directions toward real-world and next-generation automated log analysis.

Overview

Integrity signals

Registry

Crossref · 10.1145/3460345

Cites

130 of 130 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 47

  1. Anunay Amar and Peter C. Rigby . 2019. Mining historical test logs to predict bugs and localize faults in the test logs . In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering (ICSE’19) . 140--151. Anunay Amar and Peter C. Rigby. 2019. Mining historical test logs to predict bugs and localize faults in the test logs. In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). 140--151.
  2. Raju Balakrishnan and Ramendra K. Sahoo . 2006. Lossless compression for large scale cluster logs . In Proceedings 20th IEEE International Parallel & Distributed Processing Symposium (IPDPS’06) . 7. Raju Balakrishnan and Ramendra K. Sahoo. 2006. Lossless compression for large scale cluster logs. In Proceedings 20th IEEE International Parallel & Distributed Processing Symposium (IPDPS’06). 7.
  3. Ivan Beschastnikh , Yuriy Brun , Sigurd Schneider , Michael Sloan , and Michael D. Ernst . 2011. Leveraging existing instrumentation to automatically infer invariant-constrained models . In Proceedings of the 19th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’11) . 267--277. Ivan Beschastnikh, Yuriy Brun, Sigurd Schneider, Michael Sloan, and Michael D. Ernst. 2011. Leveraging existing instrumentation to automatically infer invariant-constrained models. In Proceedings of the 19th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’11). 267--277.
  4. CFDR. 2020. Computer failure data repository. Retrieved from https://www.usenix.org/cfdr. CFDR. 2020. Computer failure data repository. Retrieved from https://www.usenix.org/cfdr.
  5. Edward Chuah , Shyh-hao Kuo, Paul Hiew , William-Chandra Tjhi , Gary Lee , John Hammond , Marek T. Michalewicz , Terence Hung , and James C. Browne . 2010. Diagnosing the root-causes of failures from cluster log files . In Proceedings of the International Conference on High Performance Computing (HPC’10) . 1--10. Edward Chuah, Shyh-hao Kuo, Paul Hiew, William-Chandra Tjhi, Gary Lee, John Hammond, Marek T. Michalewicz, Terence Hung, and James C. Browne. 2010. Diagnosing the root-causes of failures from cluster log files. In Proceedings of the International Conference on High Performance Computing (HPC’10). 1--10.
  6. Sebastian Deorowicz and Szymon Grabowski . 2008 . Sub-atomic field processing for improved web log compression . In Proceedings of the IEEE International Conference on “Modern Problems of Radio Engineering, Telecommunications and Computer Science” (TCSET’08) . 551--556. Sebastian Deorowicz and Szymon Grabowski. 2008. Sub-atomic field processing for improved web log compression. In Proceedings of the IEEE International Conference on “Modern Problems of Radio Engineering, Telecommunications and Computer Science” (TCSET’08). 551--556.
  7. Rui Ding , Hucheng Zhou , Jian-Guang Lou , Hongyu Zhang , Qingwei Lin , Qiang Fu , Dongmei Zhang , and Tao Xie . 2015 . Log2: A cost-aware logging mechanism for performance diagnosis . In Proceedings of the USENIX Annual Technical Conference (ATC). 139--150 . Rui Ding, Hucheng Zhou, Jian-Guang Lou, Hongyu Zhang, Qingwei Lin, Qiang Fu, Dongmei Zhang, and Tao Xie. 2015. Log2: A cost-aware logging mechanism for performance diagnosis. In Proceedings of the USENIX Annual Technical Conference (ATC). 139--150.
  8. Stephen E. Hansen and E. Todd Atkins . 1993. Automated system monitoring and notification with swatch . In Proceedings of the 7th USENIX Large Installation System Administration Conference (LISA’93) , Vol. 93 . 145--152. Stephen E. Hansen and E. Todd Atkins. 1993. Automated system monitoring and notification with swatch. In Proceedings of the 7th USENIX Large Installation System Administration Conference (LISA’93), Vol. 93. 145--152.
  9. Pinjia He , Zhuangbin Chen , Shilin He , and Michael R. Lyu . 2018. Characterizing the natural language descriptions in software logging statements . In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18) . 178--189. Pinjia He, Zhuangbin Chen, Shilin He, and Michael R. Lyu. 2018. Characterizing the natural language descriptions in software logging statements. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE’18). 178--189.
  10. Pinjia He , Jieming Zhu , Pengcheng Xu , Zibin Zheng , and Michael R . Lyu . 2018 . A directed acyclic graph approach to online log parsing. Retrieved from https://arXiv:1806.04356. Pinjia He, Jieming Zhu, Pengcheng Xu, Zibin Zheng, and Michael R. Lyu. 2018. A directed acyclic graph approach to online log parsing. Retrieved from https://arXiv:1806.04356.
  11. Shilin He , Jieming Zhu , Pinjia He , and Michael R. Lyu . 2016. Experience report: System log analysis for anomaly detection . In Proceedings of the 27th IEEE International Symposium on Software Reliability Engineering (ISSRE’16) . 207--218. Shilin He, Jieming Zhu, Pinjia He, and Michael R. Lyu. 2016. Experience report: System log analysis for anomaly detection. In Proceedings of the 27th IEEE International Symposium on Software Reliability Engineering (ISSRE’16). 207--218.
  12. Shilin He , Jieming Zhu , Pinjia He , and Michael R . Lyu . 2020 . Loghub : A Large Collection of System Log Datasets towards Automated Log Analytics. Retrieved from https://arXiv:2008.06448. Shilin He, Jieming Zhu, Pinjia He, and Michael R. Lyu. 2020. Loghub: A Large Collection of System Log Datasets towards Automated Log Analytics. Retrieved from https://arXiv:2008.06448.
  13. 10.5555/1525908.1525912 · doi:10.5555/1525908.1525912
  14. Zhen Ming Jiang , Ahmed E. Hassan , Parminder Flora , and Gilbert Hamann . 2008 . Abstracting execution logs to execution events for enterprise applications . In Proceedings of the 8th International Conference on Quality Software (QSIC). Zhen Ming Jiang, Ahmed E. Hassan, Parminder Flora, and Gilbert Hamann. 2008. Abstracting execution logs to execution events for enterprise applications. In Proceedings of the 8th International Conference on Quality Software (QSIC).
  15. Suhas Kabinna , Cor-Paul Bezemer , Weiyi Shang , and Ahmed E. Hassan . 2016. Logging library migrations: A case study for the apache software foundation projects . In Proceedings of the 13th International Conference on Mining Software Repositories (MSR’16) . 154--164. Suhas Kabinna, Cor-Paul Bezemer, Weiyi Shang, and Ahmed E. Hassan. 2016. Logging library migrations: A case study for the apache software foundation projects. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR’16). 154--164.
  16. Suhas Kabinna , Cor-Paul Bezemer , Weiyi Shang , Mark D. Syer , and Ahmed E . Hassan . 2018 . Examining the stability of logging statements. Empir. Softw. Eng . (2018), 290--333. Suhas Kabinna, Cor-Paul Bezemer, Weiyi Shang, Mark D. Syer, and Ahmed E. Hassan. 2018. Examining the stability of logging statements. Empir. Softw. Eng. (2018), 290--333.
  17. Soila P. Kavulya , Kaustubh Joshi , Felicita Di Giandomenico, and Priya Narasimhan . 2012 . Failure diagnosis of complex systems. In Resilience Assessment and Evaluation of Computing Systems. Springer , 239--261. Soila P. Kavulya, Kaustubh Joshi, Felicita Di Giandomenico, and Priya Narasimhan. 2012. Failure diagnosis of complex systems. In Resilience Assessment and Evaluation of Computing Systems. Springer, 239--261.
  18. Suleman Khan , Abdullah Gani , Ainuddin Wahid Abdul Wahab , Mustapha Aminu Bagiwa, Muhammad Shiraz, Samee U. Khan, Rajkumar Buyya, and Albert Y. Zomaya. 2016 . Cloud log forensics: Foundations, state of the art, and future directions. ACM Comput. Surveys ( 2016), 178--184. Suleman Khan, Abdullah Gani, Ainuddin Wahid Abdul Wahab, Mustapha Aminu Bagiwa, Muhammad Shiraz, Samee U. Khan, Rajkumar Buyya, and Albert Y. Zomaya. 2016. Cloud log forensics: Foundations, state of the art, and future directions. ACM Comput. Surveys (2016), 178--184.
  19. Jean-Claude Laprie . 1995 . Dependable computing: Concepts, limits, challenges . In Proceedings of the 25th International Symposium on Fault-tolerant Computing (FTCS’95) . 42--54. Jean-Claude Laprie. 1995. Dependable computing: Concepts, limits, challenges. In Proceedings of the 25th International Symposium on Fault-tolerant Computing (FTCS’95). 42--54.
  20. Heng Li , Tse-Hsun (Peter) Chen , Weiyi Shang , and Ahmed E . Hassan . 2018 . Studying software logging using topic models. Empir. Softw. Eng . (2018), 2655--2694. Heng Li, Tse-Hsun (Peter) Chen, Weiyi Shang, and Ahmed E. Hassan. 2018. Studying software logging using topic models. Empir. Softw. Eng. (2018), 2655--2694.
  21. Heng Li , Weiyi Shang , and Ahmed E . Hassan . 2017 . Which log level should developers choose for a new logging statement?Empir. Softw. Eng . (2017), 1684--1716. Heng Li, Weiyi Shang, and Ahmed E. Hassan. 2017. Which log level should developers choose for a new logging statement?Empir. Softw. Eng. (2017), 1684--1716.
  22. Heng Li , Weiyi Shang , Ying Zou , and Ahmed E . Hassan . 2017 . Towards just-in-time suggestions for log changes. Empir. Softw. Eng . (2017), 1831--1865. Heng Li, Weiyi Shang, Ying Zou, and Ahmed E. Hassan. 2017. Towards just-in-time suggestions for log changes. Empir. Softw. Eng. (2017), 1831--1865.
  23. Chinghway Lim , Navjot Singh , and Shalini Yajnik . 2008 . A log mining approach to failure analysis of enterprise telephony systems . In Proceedings of the 38th IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’08) . Chinghway Lim, Navjot Singh, and Shalini Yajnik. 2008. A log mining approach to failure analysis of enterprise telephony systems. In Proceedings of the 38th IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’08).
  24. Jinyang Liu , Jieming Zhu , Shilin He , Pinjia He , Zibin Zheng , and Michael R. Lyu . 2019. Logzip: Extracting hidden structures via iterative clustering for log compression . In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE’19) . 863--873. Jinyang Liu, Jieming Zhu, Shilin He, Pinjia He, Zibin Zheng, and Michael R. Lyu. 2019. Logzip: Extracting hidden structures via iterative clustering for log compression. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE’19). 863--873.
  25. Zhongxin Liu Xin Xia David Lo Zhenchang Xing Ahmed E. Hassan and Shanping Li. 2019. Which variables should I log?IEEE Trans. Softw. Eng. (2019). Zhongxin Liu Xin Xia David Lo Zhenchang Xing Ahmed E. Hassan and Shanping Li. 2019. Which variables should I log?IEEE Trans. Softw. Eng. (2019).
  26. Joshua Lockerman , Jose M. Faleiro , Juno Kim , Soham Sankaran , Daniel J. Abadi , James Aspnes , Siddhartha Sen , and Mahesh Balakrishnan . 2018 . The fuzzylog: A partially ordered shared log . In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18) . 357--372. Joshua Lockerman, Jose M. Faleiro, Juno Kim, Soham Sankaran, Daniel J. Abadi, James Aspnes, Siddhartha Sen, and Mahesh Balakrishnan. 2018. The fuzzylog: A partially ordered shared log. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI’18). 357--372.
  27. Jian-Guang Lou , Qiang Fu , Shengqi Yang , Ye Xu , and Jiang Li . 2010 . Mining invariants from console logs for system problem detection . In Proceedings of the USENIX Annual Technical Conference (ATC’10) . 1--14. Jian-Guang Lou, Qiang Fu, Shengqi Yang, Ye Xu, and Jiang Li. 2010. Mining invariants from console logs for system problem detection. In Proceedings of the USENIX Annual Technical Conference (ATC’10). 1--14.
  28. Liang Luo , Suman Nath , Lenin Ravindranath Sivalingam , Madan Musuvathi , and Luis Ceze . 2018 . Troubleshooting transiently recurring errors in production systems with blame-proportional logging . In Proceedings of the USENIX Annual Technical Conference (ATC’18) . 321--334. Liang Luo, Suman Nath, Lenin Ravindranath Sivalingam, Madan Musuvathi, and Luis Ceze. 2018. Troubleshooting transiently recurring errors in production systems with blame-proportional logging. In Proceedings of the USENIX Annual Technical Conference (ATC’18). 321--334.
  29. Michael R. Lyu (ed.). 1996. Handbook of Software Reliability Engineering. IEEE Computer Society Press. Michael R. Lyu (ed.). 1996. Handbook of Software Reliability Engineering. IEEE Computer Society Press.
  30. Adetokunbo Makanju , A. Nur Zincir-Heywood , and Evangelos E . Milios . 2012 . A lightweight algorithm for message type extraction in system application logs. IEEE Trans. Knowl. Data Eng . (2012). Adetokunbo Makanju, A. Nur Zincir-Heywood, and Evangelos E. Milios. 2012. A lightweight algorithm for message type extraction in system application logs. IEEE Trans. Knowl. Data Eng. (2012).
  31. Adetokunbo A. O. Makanju , A. Nur Zincir-Heywood , and Evangelos E. Milios . 2009. Clustering event logs using iterative partitioning . In Proceedings of International Conference on Knowledge Discovery and Data Mining (KDD’09) . 1255--1264. Adetokunbo A. O. Makanju, A. Nur Zincir-Heywood, and Evangelos E. Milios. 2009. Clustering event logs using iterative partitioning. In Proceedings of International Conference on Knowledge Discovery and Data Mining (KDD’09). 1255--1264.
  32. Meiyappan Nagappan and Mladen A. Vouk . 2010. Abstracting log lines to log event types for mining software system logs . In Proceedings of the 7th IEEE Working Conference on Mining Software Repositories (MSR’10) . 114--117. Meiyappan Nagappan and Mladen A. Vouk. 2010. Abstracting log lines to log event types for mining software system logs. In Proceedings of the 7th IEEE Working Conference on Mining Software Repositories (MSR’10). 114--117.
  33. Karthik Nagaraj , Charles Killian , and Jennifer Neville . 2012 . Structured comparative analysis of systems logs to diagnose performance problems . In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI’12) . 353--366. Karthik Nagaraj, Charles Killian, and Jennifer Neville. 2012. Structured comparative analysis of systems logs to diagnose performance problems. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI’12). 353--366.
  34. Adam J. Oliner , Archana Ganapathi , and Wei Xu. 2012. Advances and challenges in log analysis. ACM Communication ( 2012 ), 55--61. Adam J. Oliner, Archana Ganapathi, and Wei Xu. 2012. Advances and challenges in log analysis. ACM Communication (2012), 55--61.
  35. James E. Prewett . 2003 . Analyzing cluster log files using logsurfer . In Proceedings of the 4th Annual Conference on Linux Clusters. Citeseer. James E. Prewett. 2003. Analyzing cluster log files using logsurfer. In Proceedings of the 4th Annual Conference on Linux Clusters. Citeseer.
  36. John P. Rouillard . 2004 . Real-time log file analysis using the simple event correlator (sec) . In Proceedings of the 18th USENIX Large Installation System Administration Conference (LISA’04) , Vol. 4 . 133--150. John P. Rouillard. 2004. Real-time log file analysis using the simple event correlator (sec). In Proceedings of the 18th USENIX Large Installation System Administration Conference (LISA’04), Vol. 4. 133--150.
  37. Bianca Schroeder and Garth A. Gibson . 2006. A large-scale study of failures in high-performance computing systems . In Proceedings of the International Conference on Dependable Systems and Networks (DSN’06) . 249--258. Bianca Schroeder and Garth A. Gibson. 2006. A large-scale study of failures in high-performance computing systems. In Proceedings of the International Conference on Dependable Systems and Networks (DSN’06). 249--258.
  38. Weiyi Shang , Meiyappan Nagappan , and Ahmed E . Hassan . 2015 . Studying the relationship between logging characteristics and the code quality of platform software. Empir. Softw. Eng . (2015), 1--27. Weiyi Shang, Meiyappan Nagappan, and Ahmed E. Hassan. 2015. Studying the relationship between logging characteristics and the code quality of platform software. Empir. Softw. Eng. (2015), 1--27.
  39. Keiichi Shima. 2016. Length matters: Clustering system log messages using length of words. Retrieved from https://arXiv:1611.03213. Keiichi Shima. 2016. Length matters: Clustering system log messages using length of words. Retrieved from https://arXiv:1611.03213.
  40. Przemysław Skibiński and Jakub Swacha . 2007 . Fast and efficient log file compression . In Proceedings of 11th East-European Conference on Advances in Databases and Information Systems (ADBIS’07) . 56--69. Przemysław Skibiński and Jakub Swacha. 2007. Fast and efficient log file compression. In Proceedings of 11th East-European Conference on Advances in Databases and Information Systems (ADBIS’07). 56--69.
  41. Wil Van Der Aalst . 2012. Process mining. Commun. ACM ( 2012 ), 76--83. Wil Van Der Aalst. 2012. Process mining. Commun. ACM (2012), 76--83.
  42. Stephen Yang , Seo Jin Park , and John K. Ousterhout . 2018. NanoLog: A nanosecond scale logging system . In Proceedings of the USENIX Annual Technical Conference (ATC). 335--350 . Stephen Yang, Seo Jin Park, and John K. Ousterhout. 2018. NanoLog: A nanosecond scale logging system. In Proceedings of the USENIX Annual Technical Conference (ATC). 335--350.
  43. Kundi Yao , Guilherme B. de Pádua , Weiyi Shang , Catalin Sporea , Andrei Toma , and Sarah Sajedi . 2020. Log4Perf: Suggesting and updating logging locations for web-based systems ’ performance monitoring. Empir. Softw. Eng . ( 2020 ), 488--531. Kundi Yao, Guilherme B. de Pádua, Weiyi Shang, Catalin Sporea, Andrei Toma, and Sarah Sajedi. 2020. Log4Perf: Suggesting and updating logging locations for web-based systems’ performance monitoring. Empir. Softw. Eng. (2020), 488--531.
  44. Kundi Yao , Heng Li , Weiyi Shang , and Ahmed E . Hassan . 2019 . A study of the performance of general compressors on log files. Empir. Softw. Eng . (2019), 3043--3085. Kundi Yao, Heng Li, Weiyi Shang, and Ahmed E. Hassan. 2019. A study of the performance of general compressors on log files. Empir. Softw. Eng. (2019), 3043--3085.
  45. 10.5555/2337223.2337236 · doi:10.5555/2337223.2337236
  46. Xu Zhao , Kirk Rodrigues , Yu Luo , Ding Yuan , and Michael Stumm . 2016 . Non-intrusive performance profiling for entire software stacks based on the flow reconstruction principle . In Proceedings of the 12th Symposium on Operating Systems Design and Implementation (OSDI’16) . 603--618. Xu Zhao, Kirk Rodrigues, Yu Luo, Ding Yuan, and Michael Stumm. 2016. Non-intrusive performance profiling for entire software stacks based on the flow reconstruction principle. In Proceedings of the 12th Symposium on Operating Systems Design and Implementation (OSDI’16). 603--618.
  47. Jieming Zhu , Shilin He , Jinyang Liu , Pinjia He , Qi Xie , Zibin Zheng , and Michael R. Lyu . 2019. Tools and benchmarks for automated log parsing . In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP’19) . 121--130. Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R. Lyu. 2019. Tools and benchmarks for automated log parsing. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP’19). 121--130.

Cited by

See all 532

Metadata

Authors
He, S., He, P., Chen, Z., Yang, T., Su, Y., & Lyu, M. R.
Affiliations
  • Microsoft Research Asia (China) (CN)
  • ETH Zurich (CH)
  • Chinese University of Hong Kong (HK)
Venue
ACM Computing Surveys
Published
2021-07-13
Volume
54
Issue
6
Pages
1-37
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W3183619936
Cited by
334
References
127
Citations by year
2026: 84 · 2025: 84 · 2024: 68 · 2023: 55 · 2022: 29 · 2021: 13
FWCI
18.47
Type
journal article
Language
en
Open access
Locations
  • ACM Computing Surveys (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14210717)
  • Key-Area Research and Development Program of Guangdong Province (2020B010165002)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
Keywords
automated log analysis, software logs, log compression, log parsing, event templates, anomaly detection, failure prediction, fault diagnosis, reliability engineering, open-source software toolkit

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.