Characterizing Selection Bias Using Experimental Data
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
This paper develops and applies semiparametric econometric methods to estimate the form of selection bias that arises from using nonexperimental comparison groups to evaluate social programs and to test the identifying assumptions that justify three widely-used classes of estimators and our extensions of them: (a) the method of matching; (b) the classical econometric selection model which represents the bias solely as a function of the probability of participation; and (c) the method of difference-in-differences. Using data from an experiment on a prototypical social program combined with unusually rich data from a nonexperimental comparison group, we reject the assumptions justifying matching and our extensions of that method but find evidence in support of the index-sufficient selection bias model and the assumptions that justify application of a conditional semiparametric version of the method of difference-in-difference. Fa comparable people and to appropriately weight participants and nonparticipants a sources of selection bias as conveniently measured. We present a rigorous defin bias and find that in our data it is a small component of conventially meausred it is still substantial when compared with experimentally-estimated program impa matching participants to comparison group members in the same labor market, givi same questionnaire, and making sure they have comparable characteristics substan the performance of any econometric program evaluation estimator. We show how t analysis to estimate the impact of treatment on the treated using ordinary obser
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Model selection is conducted through an iterative process, with the primary goal of achieving covariate balance afterwards (Heckman, Ichimura, Smith, & Todd, 1998; Rosenbaum & Rubin, 1984), as opposed to explicit modeling of the selection process.”
From Differential Effects of Adult Court Transfer on Juvenile Offender Recidivism (2010) · cites this paper as Heckman, Ichimura, Smith, & Todd, 1998
In the citing paper
“Inference can be done using the delta-method for parametric models, and there are also several methods available for semi-/non-parametric models (under some additional regularity conditions), including the bootstrap, as described in Heckman et al. (1998).”
From What's trending in difference-in-differences? A synthesis of the recent econometrics literature (2023) · cites this paper as Heckman et al. (1998)
In the citing paper
“The identification strategy for the effect of the ETS on innovation has been matching-adjusted DID that proceeded in two steps44,45, used for estimation of both the main policy effects and spillovers.”
From Low-carbon innovation induced by emissions trading in China (2019) · cites this paper as [44]
In the citing paper
“The Achilles heel of the survey methodology is its inability to escape from the glaring problems of selection that plague research in this field (see Winship and Morgan 1999; Rubin 1990; Heckman et al. 1998).”
From The Mark of a Criminal Record (2003) · cites this paper as Heckman et al. 1998
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 201–300 of 1,003
Workfare as an Effective Way to Fight Poverty: The Case of India’s NREGS
Ravi, S., & Engler, M. (2015) · World Development · doi:10.1016/j.worlddev.2014.09.029 · Cited by 181
Schools, Teachers, and Education Outcomes in Developing Countries
Glewwe, P. W., & Kremer, M. R. (2005) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 558
Do Subsidies to Commercial R&D Reduce Market Failures? Microeconomic Evaluation Studies
Klette, T. J., Møen, J., & Griliches, Z. (1999) · National Bureau of Economic Research · doi:10.3386/w6947 · Cited by 163
The effect of court-ordered hiring quotas on the composition and quality of police
McCrary, J. (2007) · American Economic Review · doi:10.1257/aer.97.1.318 · Cited by 171
Awarding Price, Contract Performance, and Bids Screening: Evidence from Procurement Auctions
Decarolis, F. (2014) · American Economic Journal: Applied Economics · doi:10.1257/app.6.1.108 · Cited by 174
The Value of Brownfield Remediation
Haninger, K., Ma, L., & Timmins, C. (2017) · Journal of the Association of Environmental and Resource Economists · doi:10.1086/689743 · Cited by 170
Do banks price environmental transition risks? Evidence from a quasi-natural experiment in China
Huang, B., Punzi, M. T., & Wu, Y. (2021) · Journal of Corporate Finance · doi:10.1016/j.jcorpfin.2021.101983 · Cited by 190
Impact evaluation of India's ‘Yeshasvini’ community‐based health insurance programme
Aggarwal, A. (2010) · Health Economics · doi:10.1002/hec.1605 · Cited by 168
The Informative Power of Treaty Commitment: Using the Spatial Model to Address Selection Effects
Lupu, Y. (2013) · American Journal of Political Science · doi:10.1111/ajps.12033 · Cited by 172
Poverty Reduction Effects of Agricultural Technology Adoption: A Micro-evidence from Rural Tanzania
Asfaw, S., Kassie, M., Simtowe, F., & Lipper, L. (2012) · Journal of Development Studies · doi:10.1080/00220388.2012.671475 · Cited by 160
Differential effects of active labour market programs for the unemployed
Sianesi, B. (2008) · Labour Economics · doi:10.1016/j.labeco.2007.04.004 · Cited by 169
The Principles of Experimental Design and Their Application in Sociology
Jackson, M., & Cox, D. R. (2013) · Annual Review of Sociology · doi:10.1146/annurev-soc-071811-145443 · Cited by 169
The Neyman— Rubin Model of Causal Inference and Estimation Via Matching Methods
Sekhon, J. (2009) · The Oxford Handbook of Political Methodology · doi:10.1093/oxfordhb/9780199286546.003.0011 · Cited by 165
Evaluating Continuous Training Programmes by Using the Generalized Propensity Score
Kluve, J., Schneider, H., Uhlendorff, A., & Zhao, Z. (2012) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2011.01000.x · Cited by 174
Evaluation in the Practice of Development
Ravallion, M. D. (2009) · The World Bank Research Observer · doi:10.1093/wbro/lkp002 · Cited by 204
A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies
Smith, J. A. (2000) · RePEc: Research Papers in Economics · Cited by 166
Self-citationThe Economics of Sustainable Land Management Practices in the Ethiopian Highlands
Kassie, M., Zikhali, P., Pender, J., & Köhlin, G. (2010) · Journal of Agricultural Economics · doi:10.1111/j.1477-9552.2010.00263.x · Cited by 164
Shen, C.-H., & Chang, Y. (2009) · Journal of Business Ethics · doi:10.1007/s10551-008-9826-9 · Cited by 162
Assessing the impact of NERICA on income and poverty in central and western Uganda
Kijima, Y., Otsuka, K., & Sserunkuuma, D. (2008) · Agricultural Economics · doi:10.1111/j.1574-0862.2008.00303.x · Cited by 164
When Does Regulation Distort Costs? Lessons from Fuel Procurement in US Electricity Generation
Cicala, S. (2015) · American Economic Review · doi:10.1257/aer.20131377 · Cited by 168
Tang, C., Xue, Y., Wu, H., Irfan, M., & Hao, Y. (2022) · Technology in Society · doi:10.1016/j.techsoc.2022.101963 · Cited by 163
The Experimental Approach to Development Economics
Banerjee, A., & Duflo, E. (2008) · National Bureau of Economic Research · doi:10.3386/w14467 · Cited by 198
Givord, P., Rathelot, R., & Sillard, P. (2013) · Regional Science and Urban Economics · doi:10.1016/j.regsciurbeco.2012.06.006 · Cited by 163
Measuring the effect of M&A on patenting quantity and quality
Valentini, G. (2012) · Strategic Management Journal · doi:10.1002/smj.946 · Cited by 164
Are Experiments the Only Option? A Look at Dropout Prevention Programs
Agodini, R., & Dynarski, M. (2004) · Review of Economics and Statistics · doi:10.1162/003465304323023741 · Cited by 166
Volpe Martincus, C., & Carballo, J. (2010) · Journal of Development Economics · doi:10.1016/j.jdeveco.2009.02.007 · Cited by 160
An evaluation of the active labour market programmes in SwedenPreprint
Sianesi, B. (2001) · RePEc: Research Papers in Economics · Cited by 161
Legislative Veto Players and the Effects of International Human Rights Agreements
Lupu, Y. (2015) · American Journal of Political Science · doi:10.1111/ajps.12178 · Cited by 161
The Impact of Imprisonment on Marriage and Divorce: A Risk Set Matching Approach
Apel, R. J., BLOKLAND, A. A. J., Nieuwbeerta, P., & van Schellen, M. (2009) · Journal of Quantitative Criminology · doi:10.1007/s10940-009-9087-5 · Cited by 189
Hotz, V. J., Imbens, G. W., & Klerman, J. A. (2006) · Journal of Labor Economics · doi:10.1086/505050 · Cited by 158
The Impact of Cash Transfers on Child Labor and School Attendance in BrazilPreprint
Cardoso, E., & Souza, A. P. (2004) · RePEc: Research Papers in Economics · Cited by 157
Crump, R., Hotz, V. J., Imbens, G., & Mitnik, O. (2006) · National Bureau of Economic Research · doi:10.3386/t0330 · Cited by 159
Do high technology acquirers become more innovative?
Desyllas, P., & Hughes, A. (2010) · Research Policy · doi:10.1016/j.respol.2010.05.005 · Cited by 197
High Noon for Microfinance Impact Evaluations: Re-investigating the Evidence from Bangladesh
Duvendack, M., & Palmer‐Jones, R. (2012) · The Journal of Development Studies · doi:10.1080/00220388.2011.646989 · Cited by 179
Evaluation of credit guarantee policy using propensity score matching
Oh, I., Lee, J.-D., Heshmati, A., & Choi, G.-G. (2009) · Small Business Economics · doi:10.1007/s11187-008-9102-5 · Cited by 158
Battistin, E., & Rettore, E. (2008) · Journal of Econometrics · doi:10.1016/j.jeconom.2007.05.006 · Cited by 159
Flores, C. A., Flores-Lagunes, A., Gonzalez, A., & Neumann, T. C. (2012) · Review of Economics and Statistics · doi:10.1162/rest_a_00177 · Cited by 158
Brown, J. D., Earle, J. S., & Lup, D. (2005) · Economic Development and Cultural Change · doi:10.1086/431264 · Cited by 150
Frölich, M. (2007) · The Econometrics Journal · doi:10.1111/j.1368-423x.2007.00212.x · Cited by 155
Remittances and Labor Force Participation in Mexico: An Analysis Using Propensity Score Matching
Edwards, A. C., & Rodríguez-Oreggia, E. (2009) · World Development · doi:10.1016/j.worlddev.2008.09.010 · Cited by 201
Turning Unemployment into Self‐Employment: Effectiveness of Two Start‐Up Programmes*
Baumgartner, H. J., & Caliendo, M. (2008) · Oxford Bulletin of Economics and Statistics · doi:10.1111/j.1468-0084.2008.00505.x · Cited by 182
Trading and Enforcing Patent Rights
Galasso, A., Schankerman, M. A., & Serrano, C. J. (2011) · The RAND Journal of Economics · doi:10.1111/1756-2171.12020 · Cited by 156
Mutual Fund Transparency and Corporate Myopia
Agarwal, V., Vashishtha, R., & Venkatachalam, M. (2016) · Review of Financial Studies · doi:10.1093/rfs/hhx125 · Cited by 168
Science Parks’ tenants versus out-of-Park firms: who innovates more? A duration model
Squicciarini, M. (2007) · The Journal of Technology Transfer · doi:10.1007/s10961-007-9037-z · Cited by 174
Evaluation of Swedish Youth Labor Market Programs
Larsson, L. (2003) · The Journal of Human Resources · doi:10.3368/jhr.xxxviii.4.891 · Cited by 194
The Life-cycle Benefits of an Influential Early Childhood Program
García, J. L., Heckman, J., Ermini Leaf, D., & Prados, M. J. (2016) · doi:10.3386/w22993 · Cited by 98
Self-citationHartman, E., Grieve, R., Ramsahai, R., & Sekhon, J. S. (2015) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12094 · Cited by 149
The Intervention Selection Bias: An Underrecognized Confound in Intervention Research.
Larzelere, R. E., Kuhn, B. R., & Johnson, B. (2004) · Psychological Bulletin · doi:10.1037/0033-2909.130.2.289 · Cited by 144
Land Values in a Newly Zoned City
McMillen, D. P., & McDonald, J. F. (2002) · The Review of Economics and Statistics · doi:10.1162/003465302317331928 · Cited by 171
Did the EU ETS Make a Difference? An Empirical Assessment Using Lithuanian Firm-Level Data
Jaraitė, J., & Maria, C. D. (2016) · The Energy Journal · doi:10.5547/01956574.37.2.jjar · Cited by 40
The Effectiveness of Public Credit Guarantees in the Japanese Loan Market
Uesugi, I., Sakai, K., & Yamashiro, G. M. (2010) · Journal of the Japanese and International Economies · doi:10.1016/j.jjie.2010.08.001 · Cited by 150
The long-run labour market consequences of teenage motherhood in Britain
Chevalier, A., Viitanen, T. K., & Viitanen, T. K. (2003) · Journal of Population Economics · doi:10.1007/s001480200125 · Cited by 149
Qin, J., & Zhang, B. (2007) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.1467-9868.2007.00579.x · Cited by 162
Estimating Gender Disparities in Federal Criminal Cases
Starr, S. B. (2012) · American Law and Economics Review · doi:10.1093/aler/ahu010 · Cited by 187
Chapter 59 Evaluating Anti-Poverty Programs
Ravallion, M. D. (2007) · Handbook of development economics · doi:10.1016/s1573-4471(07)04059-4 · Cited by 237
Diversity among Norwegian Boards of Directors: Does a Quota for Women Improve Firm Performance?
Dale‐Olsen, H., Schøne, P., & Verner, M. (2013) · Feminist Economics · doi:10.1080/13545701.2013.830188 · Cited by 140
Self-Selection and Post-Entry Effects of Exports: Evidence from Italian Manufacturing Firms
Serti, F., & Tomasi, C. (2008) · Review of World Economics · doi:10.1007/s10290-008-0165-9 · Cited by 175
Löschel, A., Lutz, B. J., & Managi, S. (2019) · Resource and Energy Economics · doi:10.1016/j.reseneeco.2018.03.001 · Cited by 140
Programme Evaluation with Multiple Treatments
Frölich, M. (2004) · Journal of Economic Surveys · doi:10.1111/j.0950-0804.2004.00001.x · Cited by 167
Bank margins and profits in a world of negative rates
Molyneux, P., Reghezza, A., & Xie, R. (2019) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2019.105613 · Cited by 178
Gender gap in peer-to-peer lending: Evidence from China
Chen, X., Huang, B., & Ye, D. (2019) · Journal of Banking & Finance · doi:10.1016/j.jbankfin.2019.105633 · Cited by 155
Arnold, B., Arana, B., Mäusezahl, D., Hubbard, A., & Colford, J. M. (2009) · International Journal of Epidemiology · doi:10.1093/ije/dyp241 · Cited by 136
The employment effects of job-creation schemes in Germany: A microeconometric evaluation
Caliendo, M., Hujer, R., & Thomsen, S. L. (2008) · Advances in econometrics · doi:10.1016/s0731-9053(07)00013-8 · Cited by 159
Applying Propensity Score Methods in Medical Research: Pitfalls and Prospects
Luo, Z., Gardiner, J. C., & Bradley, C. J. (2010) · Medical Care Research and Review · doi:10.1177/1077558710361486 · Cited by 138
Women and Power: Unpopular, Unwilling, or Held Back?
Casas‐Arce, P., & Sáiz, A. (2015) · Journal of Political Economy · doi:10.1086/680686 · Cited by 172
Sensitivity of matching-based program evaluations to the availability of control variables
Lechner, M., & Wunsch, C. (2013) · Labour Economics · doi:10.1016/j.labeco.2013.01.004 · Cited by 184
Evaluating the impact of education on earnings in the UK: Models, methods and results from the NCDS
Sianesi, B., Dearden, L., & Blundell, R. (2003) · Working Paper Series · doi:10.1920/wp.ifs.2003.0320 · Cited by 134
Hujer, R., Caliendo, M., & Thomsen, S. L. (2004) · Research in Economics · doi:10.1016/j.rie.2004.07.001 · Cited by 170
The Role of Local Officials in New Democracies: Evidence from Indonesia
Martínez-Bravo, M. (2014) · American Economic Review · doi:10.1257/aer.104.4.1244 · Cited by 132
International connectivity and the digital divide in Sub-Saharan Africa
Cariolle, J. (2020) · Information Economics and Policy · doi:10.1016/j.infoecopol.2020.100901 · Cited by 163
Brodaty, T., Crépon, B., & Fougère, D. (2001) · ZEW economic studies · doi:10.1007/978-3-642-57615-7_5 · Cited by 135
Evaluating Social Programs with Endogenous Program Placement and Selection of the TreatedPreprint
Todd, P. (2008) · RePEc: Research Papers in Economics · Cited by 134
Self-citationCaliendo, M., Mahlstedt, R., & Mitnik, O. A. (2014) · Labour Economics · doi:10.1016/j.labeco.2017.02.001 · Cited by 170
Ghose, A., & Todri-Adamopoulos, V. (2016) · MIS Quarterly · doi:10.25300/misq/2016/40.4.05 · Cited by 132
COMMUTING TIME AND HOUSEHOLD RESPONSIBILITIES: EVIDENCE USING PROPENSITY SCORE MATCHING
Giménez-Nadal, J. I., & Molina, J. A. (2015) · Journal of Regional Science · doi:10.1111/jors.12243 · Cited by 183
Shareholder activism and voluntary disclosure
Bourveau, T., & Schoenfeld, J. (2017) · Review of Accounting Studies · doi:10.1007/s11142-017-9408-0 · Cited by 155
Chang, P.-L., & Lee, M.-J. (2011) · Journal of International Economics · doi:10.1016/j.jinteco.2011.05.011 · Cited by 128
The Effect of Mergers in Search Markets: Evidence from the Canadian Mortgage Industry
Allen, J., Clark, R., & Houde, J.-F. (2014) · American Economic Review · doi:10.1257/aer.104.10.3365 · Cited by 126
Employed 40 Hours or Not Employed 39: Lessons from the 1982 Mandatory Reduction of the Workweek
Crépon, B., & Kramarz, F. (2002) · Journal of Political Economy · doi:10.1086/342807 · Cited by 127
FITZENBERGER, OSIKOMINU, & VÖLTER (2008) · Annales d Économie et de Statistique · doi:10.2307/27917250 · Cited by 150
Mhbounds - Sensitivity Analysis for Average Treatment Effects
Becker, S. O., & Caliendo, M. (2007) · SSRN Electronic Journal · doi:10.2139/ssrn.958699 · Cited by 123
Fu, R., Noguchi, H., Kawamura, A., Takahashi, H., & Tamiya, N. (2017) · Journal of Health Economics · doi:10.1016/j.jhealeco.2017.09.011 · Cited by 148
Securitization and Bank Performance
Casu, B., Clare, A., Sarkisyan, A., & THOMAS, S. (2013) · Journal of money credit and banking · doi:10.1111/jmcb.12064 · Cited by 154
Chabé‐Ferret, S. (2014) · Journal of Econometrics · doi:10.1016/j.jeconom.2014.09.013 · Cited by 156
Raza, M. H., Abid, M., Faisal, M., Yan, T., Akhtar, S., & Adnan, K. M. M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19084753 · Cited by 145
Jacobson, L. S., Lalonde, R. J., & Sullivan, D. G. (2005) · Industrial and Labor Relations Review · doi:10.1177/001979390505800305 · Cited by 149
The new deal for young people: effect on the labour market status of young men
Dorsett, R. (2005) · Labour Economics · doi:10.1016/j.labeco.2004.10.006 · Cited by 156
Kugler, A. D. (1999) · International Tax and Public Finance · doi:10.1023/a:1008711819429 · Cited by 123
Sensitivity of propensity score methods to the specifications
Zhao, Z. (2008) · Economics Letters · doi:10.1016/j.econlet.2007.05.010 · Cited by 129
Fajnzylber, P. R., Maloney, W. F., & Montes‐Rojas, G. (2009) · The Journal of Development Studies · doi:10.1080/00220380802264911 · Cited by 142
Ibarrarán, P., & Rosas Shady, D. (2009) · Journal of Development Effectiveness · doi:10.1080/19439340902918094 · Cited by 122
“ing procedure proposed by Heckman et al. (1998) was used.”
BackgroundMethodInfluentialDaidone, S., Davis, B., Handa, S., & Winters, P. (2019) · American Journal of Agricultural Economics · doi:10.1093/ajae/aay113 · Cited by 165
Costs and benefits of Danish active labour market programmes
Jespersen, S. T., Munch, J. R., & Skipper, L. (2007) · Labour Economics · doi:10.1016/j.labeco.2007.07.005 · Cited by 171
Li, H., Graham, D. J., & Majumdar, A. (2012) · Accident Analysis & Prevention · doi:10.1016/j.aap.2012.02.013 · Cited by 140
Short- and medium-term effects of informal care provision on female caregivers’ health
Schmitz, H., & Westphal, M. (2015) · Journal of Health Economics · doi:10.1016/j.jhealeco.2015.03.002 · Cited by 158
Who Wears the Pants? Gender Identity Norms and Intra-Household Financial Decision Making
Ke, D. (2017) · The Journal of Finance · doi:10.1111/jofi.13002 · Cited by 142
SHARP BOUNDS ON THE DISTRIBUTION OF TREATMENT EFFECTS AND THEIR STATISTICAL INFERENCE
Fan, Y., & Park, S. S. (2009) · Econometric Theory · doi:10.1017/s0266466609990168 · Cited by 199
Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment
Arceneaux, K., Gerber, A. S., & Green, D. P. (2006) · Political Analysis · doi:10.1093/pan/mpj001 · Cited by 116
Microeconometric Evaluation of the Active Labour Market Policy in Switzerland
Lechner, M., & Gerfin, M. (2000) · SSRN Electronic Journal · doi:10.2139/ssrn.233906 · Cited by 117
Where to attend? Estimating the effects of beginning college at a two-year institution
Reynolds, C. L. (2011) · Economics of Education Review · doi:10.1016/j.econedurev.2011.12.001 · Cited by 137
Metadata
- Authors
- Heckman, J., Ichimura, H., Smith, J., & Todd, P.
- Venue
- Econometrica
- Publisher
- JSTOR
- Published
- 1998-09
- Volume
- 66
- Issue
- 5
- Pages
- 1017
- ISSN
- 0012-9682 (ISSN-L)
- OpenAlex
- W3021176487
- Cited by
- 2,286
- References
- 0
- Citations by year
- 2026: 27 · 2025: 49 · 2024: 51 · 2023: 68 · 2022: 95 · 2021: 70 · 2020: 89 · 2019: 72 · 2018: 84 · 2017: 98 · 2016: 90 · 2015: 120 · 2014: 116 · 2013: 120 · 2012: 133
- FWCI
- 30.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Econometrica (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Education › School Choice and Performance
- Keywords
- selection bias, program evaluation, matching estimators, difference-in-differences, selection model, social programs, treatment effect estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.