The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index; 4 of the 21 sources its claims cite were checked for support.
Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools
Open accessAbstract
Background: We explored the performance of three machine learning tools designed to facilitate title and abstract screening in systematic reviews (SRs) when used to (a) eliminate irrelevant records (automated simulation) and (b) complement the work of a single reviewer (semi-automated simulation). We evaluated user experiences for each tool. Methods: We subjected three SRs to two retrospective screening simulations. In each tool (Abstrackr, DistillerSR, RobotAnalyst), we screened a 200-record training set and downloaded the predicted relevance of the remaining records. We calculated the proportion missed and workload and time savings compared to dual independent screening. To test user experiences, eight research staff tried each tool and completed a survey. Results: Using Abstrackr, DistillerSR, and RobotAnalyst, respectively, the median (range) proportion missed was 5 (0 to 28) percent, 97 (96 to 100) percent, and 70 (23 to 100) percent for the automated simulation and 1 (0 to 2) percent, 2 (0 to 7) percent, and 2 (0 to 4) percent for the semi-automated simulation. The median (range) workload savings was 90 (82 to 93) percent, 99 (98 to 99) percent, and 85 (85 to 88) percent for the automated simulation and 40 (32 to 43) percent, 49 (48 to 49) percent, and 35 (34 to 38) percent for the semi-automated simulation. The median (range) time savings was 154 (91 to 183), 185 (95 to 201), and 157 (86 to 172) hours for the automated simulation and 61 (42 to 82), 92 (46 to 100), and 64 (37 to 71) hours for the semi-automated simulation. Abstrackr identified 33–90% of records missed by a single reviewer. RobotAnalyst performed less well and DistillerSR provided no relative advantage. User experiences depended on user friendliness, qualities of the user interface, features and functions, trustworthiness, ease and speed of obtaining predictions, and practicality of the export file(s). Conclusions: The workload savings afforded in the automated simulation came with increased risk of missing relevant records. Supplementing a single reviewer’s decisions with relevance predictions (semi-automated simulation) sometimes reduced the proportion missed, but performance varied by tool and SR. Designing tools based on reviewers’ self-identified preferences may improve their compatibility with present workflows. Systematic review registration: Not applicable.
Overview
CoChat Summary
TL;DR
In retrospective simulations across three systematic reviews, three machine-learning screening tools differed substantially in performance and usability. Abstrackr was most usable and, when used to complement a single reviewer, sometimes reduced missed records, but automated exclusion carried a high risk of missing relevant studies.
Key findings
- In the automated simulation, median (range) proportion missed was 5% (0–28) for Abstrackr, 97% (96–100) for DistillerSR, and 70% (23–100) for RobotAnalyst; workload savings were 90% (82–93), 99% (98–99), and 85% (84–88), respectively, and estimated time savings were 154 (91–183), 185 (95–201), and 157 (86–172) hours.
- In the semi-automated simulation, median (range) proportion missed was 1% (0–2) for Abstrackr, 2% (0–7) for DistillerSR, and 2% (0–4) for RobotAnalyst; workload savings were 40% (32–43), 49% (48–49), and 35% (34–38), and time savings were 61 (42–82), 92 (46–100), and 64 (37–71) hours. Abstrackr identified 33–90% of records missed by a single reviewer; RobotAnalyst performed less well and DistillerSR provided no relative advantage.
- Among 8 research staff (73% response rate), median (IQR) System Usability Scale scores were 79 (23) for Abstrackr, 64 (31) for DistillerSR, and 31 (8) for RobotAnalyst; 62% chose Abstrackr as first choice, 38% chose DistillerSR as first choice, and 88% ranked RobotAnalyst last.
- In a post hoc 500-record training set tested only in Abstrackr, automated-simulation median proportion missed increased from 28 to 41% for Antipsychotics and 5 to 9% for Bronchiolitis, with no change for Visual Acuity; workload and estimated time savings increased.
Methods
- Retrospective screening simulations using three systematic reviews (Antipsychotics, Bronchiolitis, Visual Acuity; screening workloads 12,156, 5,861, and 11,229 records) and three tools (Abstrackr, DistillerSR/DistillerAI, RobotAnalyst). A 200-record training set was used, except Visual Acuity required 300 records in Abstrackr before predictions were produced.
- Two simulations were tested: automated exclusion of records predicted irrelevant, and semi-automated screening where predictions complemented one human reviewer. Performance metrics were calculated from 2 × 2 cross-tabulations: proportion missed, workload savings, and estimated time savings assuming 0.5 min per record and an 8-h work day.
- User experience testing involved 8 research staff who completed a screening exercise in each tool and a REDCap survey including the System Usability Scale; quantitative data were summarized and qualitative data analyzed thematically.
Limitations
- The paper states that training sets differed across tools and reviews, standard tool settings were used, the training set was 200 records, and the sample was only three systematic reviews; findings should not be generalized to other tools, reviews, or semi-automated screening approaches. It also states that time savings did not account for troubleshooting usability issues, variability in screening time, or differences between obviously excluded and uncertain records, and that the impact of missed studies on review results was not investigated.
- Evident from the text: the systematic reviews and user sample were convenient samples, and RobotAnalyst failed to upload full RIS record lists for two reviews, requiring manual screening assumptions for 170 Bronchiolitis and 183 Visual Acuity records.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
2 Valid · 0 Not valid · 10 Not checked · 53 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Title and abstract screening is a review step that may be particularly amenable to automation or semi-automation [2–4].”
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Background
What the sources say
- Not checked yetcites [2] Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Res Synth Methods. 2011;2:1–14. https://doi.org/10.1002/jrsm.27. source
- Supportedcites [3] Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Syst Rev. 2014;3:74. https://doi.org/10.1186/2046-4053-3-74. source
In the Discussion's task-by-task analysis, 'Task 7: screen abstracts' states that despite its brevity, title-and-abstract screening is error-prone and time-consuming because of the large number of records, and that decision support systems and automatic screening systems 'could thus be used as a tie-breaker to resolve disagreements, and/or replace one or both screeners' — i.e. automation or semi-automation. Its 'Current systems' subsection and the following paragraph describe operating semi-automated tools (Abstrackr halving reviewer workload; appraisal systems cutting manual appraisal by 30-50%), which is more concrete than the hedged 'may be particularly amenable'. By contrast, full-text screening (Task 9) is described as lacking adopted systems for the pertinent elements, supporting the comparative force of the claim. (LLM, conf=0.85)
- Supportedcites [4] Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H, et al. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7:77. https://doi.org/10.1186/s13643-018-0740-7. source
The Background (p. 1) of the ICASR paper states that "many of the tasks in a systematic review are amenable to automation: screening of titles and abstracts, sourcing full texts of included studies, data extraction and even collation of meta-analysis results are all fertile areas for automation", naming title/abstract screening first among them. The Vienna Principles section reinforces this, noting "there are multiple tools for title and abstract screening" and that steps such as title and abstract screening "will potentially use common automation methods". The hedged phrasing of the claim ("may be particularly amenable") is at or below the confidence the source expresses. (LLM, conf=0.90)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Despite their promise, the adoption of ML tools among reviewers has been slow [6–8].”
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Background
What the sources say
- Supportedcites [6] O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5. https://doi.org/10.1186/2046-4053-4-5. source
Our copy consistently treats limited take-up by reviewers as its finding: the Discussion ('Summary of key findings') reports 'few deployed systems, which limits the ability of reviewers to try out these technologies' and a probable need for specialist advice in live reviews; the Results ('Implementation challenges') note that despite the technology being called 'promising', 'it was not always clear how these technologies would be rolled out for use in “live” reviews'; and Background ('The research problem') frames the field as needing accessibility 'for these technologies to achieve broad uptake', with the abstract stating the review aims 'to increase awareness of the potential of these technologies'. The paper addresses text mining/machine learning for study identification specifically and does not quantify a rate of adoption, but the broad claim of slow reviewer adoption is what our copy clearly asserts. (LLM, conf=0.82)
- Supportedcites [7] O'Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Wolfe MS. Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7:3. https://doi.org/10.1186/s13643-017-0667-4. source
In 'Common needs and challenges for systematic reviews across scientific domains' (Background), our copy states that although semi-automated screening is already in practice in ML-based packages (Abstrackr, EPPI-Reviewer), 'the state of the art is not such that producers of systematic reviewers routinely use these systems,' attributing this to a lack of transparency of machine-learning systems and a shortage of studies showing screening-system benefits. Table 2 in the same section lists 'Social acceptance of automation technology' and the 'black box' nature of machine learning among the broader challenges to automation, and the surrounding text frames technology acceptance as a philosophical barrier. Together these treat limited/slow uptake of ML tools by review producers as the source's own point; the 'despite their promise' framing is the citing authors'. (LLM, conf=0.80)
- Not checked yetcites [8] Thomas J. Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evidence-Based Med. 2013;1:12. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DistillerSR did not correctly predict the relevance of any of the records missed by the second reviewer in either review, thus providing no advantage over single-reviewer screening.”
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Results > Semi-automated simulation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Abstrackr identified 33–90% of records missed by a single reviewer.”
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Abstract > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Visual Acuity review was unique in that only one record from the 11,229 screened was included in the final report.”
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Results > Performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
15 more claims
Create a free account to see all 65 claims and to verify the 17 references still open.
- It may state two separate claims as one. 34
- Probably not a substantial claim. 19
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, previous research into the performance of various ML-based tools for abstract screening has been limited in sample size, and the software tools evaluated were not open source, thereby hindering the identification of relevant parameters for these methods (Gates et al., 2019; Robledo et al., 2023).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Gates et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, usability and accessibility of automation tools is a common issue (Gates et al., 2019; Harrison et al., 2020)”
From SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing (2021) · cites this paper as Gates et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Researchers’ current lack of trust in the accuracy and quality of SLR automation tools has been identified as a major barrier to uptake, and transparency from the developers and users of these tools has been suggested as a means to overcome this challenge (Bolaños et al., 2024; de la Torre-López et al., 2023; Gates et al., 2019; O’Connor et al., 2019; van Altena et al., 2019).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Gates et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“More recently, machine learning algorithms have been proposed as a way to increase workload efficiency (Gates et al., 2019).”
From Sysrev: A FAIR Platform for Data Curation and Systematic Evidence Review (2021) · cites this paper as Gates et al., 2019
In the citing paper
“Recent years have seen the development of many ML tools that aim to reduce the immense human resources and time effort required by a multi-disciplinary team to develop such a review [29, 38, 52, 56]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [38]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Abstract > Results
“The median (range) workload savings was 90 (82 to 93) percent, 99 (98 to 99) percent, and 85 (85 to 88) percent for the automated simulation”
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Results > Automated simulation
“The median (range) workload savings was 90 (82, 93)%, 99 (98, 99)%, 85 (84, 88)% for Abstrackr, DistillerSR, and RobotAnalyst, respectively”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Methods > Performance testing
“Although this could affect the predictions, in a previous evaluation, we found little difference in Abstrackr’s predictions over three independent trials [15].”
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > References
“15. The National Centre for Text Mining. RobotAnalyst. 2019.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Results > Performance
“RobotAnalyst predicted that a median (range) 18 (12, 33)%, 0.1 (0, 1)%, and 29 (20, 29)% of the remaining records were relevant, respectively.”
§ Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools > Results > Performance
“| Predicted relevant by RobotAnalyst | 3488 (29) | 1082 (19) | 3221 (29) |”
Citations
Cites
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study
Vaismoradi, M., Turunen, H., & Bondas, T. (2013) · Nursing & Health Sciences · doi:10.1111/nhs.12048 · Cited by 8,327
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., & Pieper, D. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0782-0 · Cited by 626
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472
Research works printed but not linked: 2
- Pillay J, Freeman EE, Hodge W, MacGregor T, Featherstone R, Vandermeer B, et al. Screening for impaired visual acuity and vision-related functional limitations in adults 65 years and older in primary health care: systematic review. Edmonton: Evidence Review Synthesis Centre; 2017. http://canadiantaskforce.ca/ctfphc-guidelines/overview/.
- Chen Y. Developing stopping rules for a ML system in citation screening [dissertation]. Providence: Brown University; 2019.
Cited by
Sharkiya, S. H. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09869-8 · Cited by 416
“Research has shown that combining a machine learning tool and a single reviewer can significantly reduce the risk of missing relevant records [20].”
BackgroundAutomated Paper Screening for Clinical Reviews Using Large Language Models
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00844 · Cited by 230
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
“There is a growing interest in tools to facilitate and speed up the screening process (e.g. [3, 13, 21, 28]).”
BackgroundMethodUse of Real‐World Evidence to Drive Drug Development Strategy and Inform Clinical Trial Design
Dagenais, S., Russo, L., Madsen, A. M., Webster, J., & Becnel, L. B. (2021) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.2480 · Cited by 155
Metadata
- Authors
- Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L.
- Affiliations
- University of Alberta (CA)
- Corresponding author
- Hartling, Lisa
- ORCID
- Venue
- Systematic Reviews
- Published
- 2019-11-15
- Volume
- 8
- Issue
- 1
- Pages
- 278-278
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6857345
- OpenAlex
- W2982683456
- Cited by
- 133
- References
- 28
- Citations by year
- 2026: 24 · 2025: 19 · 2024: 22 · 2023: 23 · 2022: 12 · 2021: 14 · 2020: 18
- FWCI
- 2.23
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Agency for Healthcare Research and Quality (290-2015-00001-I)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- title and abstract screening, machine learning, automated screening, semi-automated screening, reviewer workload, user experience, user interface, user credibility, time savings
- MeSH terms
- Abstracting and Indexing, Machine Learning, Systematic Reviews as Topic, Humans, Software, Time Factors, Reproducibility of Results, Information Storage and Retrieval, Workload
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. https://app.cochat.ai/papers/10.1186/s13643-019-1222-2
- The paper
- Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019). Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. Systematic Reviews. https://doi.org/10.1186/s13643-019-1222-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.