CoChat

Technology and Big Data Are Changing Economics: Mining Text to Track Methods

Open access
Currie, Janet, Kleven, Henrik, Zwiers, Esmée · 2020 · AEA Papers and Proceedings

The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index; 2 of the 12 sources its claims cite were checked for support.

Abstract

The last 40 years have seen huge innovations in computing and in the availability of data. Data derived from millions of administrative records or by using (as we do) new methods of data generation such as text mining are now common. New data often requires new methods, which in turn can inspire new data collection. If history is any guide, some methods will stick and others will prove to be a flash in the pan. However, the larger trends toward demanding greater credibility and transparency from researchers in applied economics and a 'collage' approach to assembling evidence will likely continue.

Overview

CoChat Summary

TL;DR

This paper uses text mining on 2,830 top-five journal papers and 10,324 NBER working papers in applied microeconomics to document major shifts in methods and data use over time. It argues that applied economics has become more focused on identification, experiments, administrative data, precision, and combining multiple empirical strategies, consistent with a “credibility revolution” and a “collage” approach.

Key findings

  • References to identification rose roughly linearly in both series, from around 4% to 50% of papers.
  • References to experimental and quasi-experimental methods also increased: over 40% of NBER papers and about 35% of top-five papers now mention methods such as RCTs, lab experiments, difference-in-differences, regression discontinuity, event studies, or bunching.
  • References to administrative data rose from the mid-1990s to about 30% today in the NBER series; the top-five series shows a similar increase with a lag of about three years.
  • The paper reports that concern with mechanisms rose from about 20% to about 60% in NBER papers and to more than 70% today in top-five papers; references to “precisely estimated zero” are still only in a little over 1% of papers.

Methods

  • The study is a text-mining analysis of economics papers, using regex searches on plain-text versions of papers after removing references sections.
  • Sample: all NBER working papers from January 1, 1980 to June 30, 2018 in selected applied micro programs, and all papers in the top-five journals from January 1, 2004 to August 2019; final sample sizes are 10,324 NBER papers and 2,830 top-five papers.
  • Main analysis: compute five-year moving averages of the fraction of papers mentioning predefined methods- and data-related trigger phrases (for example, identification, RCTs, administrative data, big data, clustering, fixed effects, structural models).

Limitations

  • The authors state that their search algorithms were built by trial and error, and that some Type I and Type II errors may remain; they say this is not a major concern because their focus is on trends rather than levels.
  • Their within-paper comparison of NBER papers that were later published in top-five journals uses a matched sample of 610 papers, which they describe as arguably too small to draw firm conclusions.
  • Evident from the text: the analysis relies on keyword mentions, so it captures discussion of methods and data rather than direct measures of whether those methods were actually used.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1257/pandp.20201058

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure I: Fraction of Applied Microeconomics Articles in Top-5 Journals
Figure I: Fraction of Applied Microeconomics Articles in Top-5 JournalsFrom the PDF
Figure II: The Credibility Revolution
Figure II: The Credibility RevolutionFrom the PDF
A: RCTs
A: RCTsFrom the PDF
B: Regression Discontinuity
B: Regression DiscontinuityFrom the PDF
Image 5
From the PDF
Image 6
From the PDF
Figure VI: What's New?
Figure VI: What's New?From the PDF
B: Selection
B: SelectionFrom the PDF
B: Proprietary Data
B: Proprietary DataFrom the PDF
Figure A.III: External Validity
Figure A.III: External ValidityFrom the PDF
Image 11
From the PDF
Image 12
From the PDF
Figure A.VI: Confidence Interval
Figure A.VI: Confidence IntervalFrom the PDF
Image 14
From the PDF
A: All Papers
A: All PapersFrom the PDF
B: NBER Program: Labor Studies
B: NBER Program: Labor StudiesFrom the PDF
D: NBER Program: Industrial Organization
D: NBER Program: Industrial OrganizationFrom the PDF
B: NBER Program: Labor Studies
B: NBER Program: Labor StudiesFrom the PDF
B: NBER Program: Labor Studies
B: NBER Program: Labor StudiesFrom the PDF
Figure B.I: Number of papers in sample over time
Figure B.I: Number of papers in sample over timeFrom the PDF

Verification

Claims the paper makes

1 Valid · 3 Not valid · 9 Not checked · 75 Own results and interpretations

41 more claims

Create a free account to see all 88 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 22
  • Probably not a substantial claim. 22
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Citations

Cites

Show all 15 references

Cited by

Metadata

Authors
Currie, J., Kleven, H., & Zwiers, E.
Affiliations
Princeton University (US)
Venue
AEA Papers and Proceedings
Published
2020-05-01
Volume
110
Pages
42-48
ISSN
  • 2574-0768 (ISSN-L)
  • 2574-0776
OpenAlex
W3003454468
Cited by
86
References
5
Citations by year
2026: 7 · 2025: 13 · 2024: 10 · 2023: 12 · 2022: 20 · 2021: 16 · 2020: 6
FWCI
38.44
Type
journal article
Language
en
Open access
Locations
  • AEA Papers and Proceedings (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords
text mining, administrative data, big data, data collection methods, research credibility, evidence aggregation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.