CoChat
Sign Up
Cited by 881Computational and Text Analysis MethodsWeb Data Mining and AnalysisSentiment Analysis and Opinion Mining

The Pushshift Reddit Dataset

Open access
Baumgartner, Jason, Zannettou, Savvas, Keegan, Brian, Squire, Megan, Blackburn, Jeremy·2020·Proceedings of the International AAAI Conference on Web and Social Media
DOI 10.1609/icwsm.v14i1.7347
Open this paper in CoChat — free accountOpen PDF

Abstract

Social media data has become crucial to the advancement of scientific understanding. However, even though it has become ubiquitous, just collecting large-scale social media data involves a high degree of engineering skill set and computational resources. In fact, research is often times gated by data engineering problems that must be overcome before analysis can proceed. This has resulted recognition of datasets as meaningful research contributions in and of themselves.Reddit, the so called “front page of the Internet,” in particular has been the subject of numerous scientific studies. Although Reddit is relatively open to data acquisition compared to social media platforms like Facebook and Twitter, the technical barriers to acquisition still remain. Thus, Reddit's millions of subreddits, hundreds of millions of users, and billions of comments are at the same time relatively accessible, but time consuming to collect and analyze systematically.In this paper, we present the Pushshift Reddit dataset. Pushshift is a social media data collection, analysis, and archiving platform that since 2015 has collected Reddit data and made it available to researchers. Pushshift's Reddit dataset is updated in real-time, and includes historical data back to Reddit's inception. In addition to monthly dumps, Pushshift provides computational tools to aid in searching, aggregating, and performing exploratory analysis on the entirety of the dataset. The Pushshift Reddit dataset makes it possible for social media researchers to reduce time spent in the data collection, cleaning, and storage phases of their projects.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/icwsm.v14i1.7347

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Submissions and comments are obtained from the publicly distributed per-subreddit Pushshift dumps, yielding 107,948 submissions and 2,373,115 comments.

    “Submissions and comments are obtained from the publicly distributed per-subreddit Pushshift dumps (Baumgartner et al., 2020)-a widely mirrored archive of the top ∼ 40K subreddits distributed via community-maintained tooling 3 -yielding 107,948 submissions and 2,373,115 comments.”

    From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Baumgartner et al., 2020

    Section 3 describes publicly available Pushshift submissions and comments, and the Discussion & Conclusion says the data are offered as monthly dumps. The source does not describe per-subreddit dumps, a widely mirrored archive of the top ∼40K subreddits, or the citing study’s totals of 107,948 submissions and 2,373,115 comments. No page numbers are available in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • E-GEO queries are derived from Reddit's r/BuyItForLife subreddit via Pushshift dumps.

    “Queries are derived from Reddit's r/BuyItForLife subreddit via the publicly distributed per-subreddit Pushshift dumps (Baumgartner et al., 2020)”

    From E-GEO: A Testbed for Generative Engine Optimization in E-Commerce (2025) · cites this paper as Baumgartner et al., 2020

    The Abstract and Section 3 describe publicly available Pushshift dumps of Reddit submissions and comments, supporting the citation as an identification of the data source. Section 3 says the dump files are organized by month, not by subreddit. The paper does not establish that E-GEO queries were derived from r/BuyItForLife. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. Ahmadvand, A.; Sahijwani, H.; Choi, J. I.; and Agichtein, E. 2019. Concet: Entity-aware topic classification for open-domain conver- sational agents. In CIKM.
  2. Ammari, T.; Schoenebeck, S.; and Romero, D. 2019. Self- declared throwaway accounts on reddit: How platform affordances and shared norms enable parenting disclosure and support. CSCW.
  3. Borgman, C. L. 2012. The conundrum of sharing research data. Journal of the American Society for Information Science and Tech- nology 63(6):1059-1078.
  4. Bowen, D. A.; O'Donnell, J.; and Sumner, S. A. 2019. Increases in online posts about synthetic opioids preceding increases in syn- thetic opioid death rates: a retrospective observational study. Jour- nal of general internal medicine 34(12):2702-2704.
  5. boyd, d., and Crawford, K. 2012. Critical Questions for Big Data. Information, Communication & Society 15(5):662-679.
  6. boyd, d. 2016. Untangling research and practice: What Facebook's "emotional contagion" study teaches us. Research Ethics 12(1):4- 13.
  7. Boyle, J. 2017. The second enclosure movement and the construc- tion of the public domain. In Copyright Law. Routledge. 63-104.
  8. Bruns, A. 2019. After the 'APIcalypse': Social media platforms and their fight against critical scholarly research. Information, Communication & Society 22(11):1544-1566.
  9. Burris, V.; Smith, E.; and Strahm, A. 2000. White supremacist networks on the internet. Sociological Focus 33(2):215-235.
  10. Chandrasekharan, E.; Pavalanathan, U.; Srinivasan, A.; Glynn, A.; Eisenstein, J.; and Gilbert, E. 2017a. You can't stay here: The efficacy of reddit's 2015 ban examined through hate speech. Proc. ACM Hum.-Comput. Interact.
  11. Chandrasekharan, E.; Pavalanathan, U.; Srinivasan, A.; Glynn, A.; Eisenstein, J.; and Gilbert, E. 2017b. You can't stay here: The ef- ficacy of reddit's 2015 ban examined through hate speech. CSCW.
  12. Chandrasekharan, E.; Samory, M.; Srinivasan, A.; and Gilbert, E. 2017c. The bag of communities: Identifying abusive behavior on- line with preexisting internet data. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, 3175-3187. ACM.
  13. Chandrasekharan, E.; Samory, M.; Jhaver, S.; Charvat, H.; Bruck- man, A.; Lampe, C.; Eisenstein, J.; and Gilbert, E. 2018. The in- ternet's hidden rules: An empirical study of reddit norm violations at micro, meso, and macro scales. CHI.
  14. Datta, S.; Phelan, C.; and Adar, E. 2017. Identifying misaligned inter-group links and communities. CSCW.
  15. Derczynski, L.; Nichols, E.; van Erp, M.; and Limsopatham, N. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, 140-147.
  16. Ekbia, H.; Mattioli, M.; Kouper, I.; Arave, G.; Ghazinejad, A.; Bowman, T.; Suri, V. R.; Tsou, A.; Weingart, S.; and Sugimoto, C. R. 2015. Big data, bigger dilemmas: A critical review. Jour- nal of the Association for Information Science and Technology 66(8):1523-1545.
  17. Gaffney, D., and Matias, J. N. 2018. Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. PLOS ONE 13(7):e0200162.
  18. Gibney, E. 2019. Privacy hurdles thwart Facebook democracy research. Nature 574(7777):158-159.
  19. Golder, S. A., and Macy, M. W. 2014. Digital Footprints: Opportu- nities and Challenges for Online Social Research. Annual Review of Sociology 40(1):129-152.
  20. Golovanov, S.; Tselousov, A.; Kurbanov, R.; and Nikolenko, S. I. 2020. Lost in conversation: A conversational agent based on the transformer and transfer learning. In The NeurIPS'18 Competition. Springer. 295-315.
  21. Gousios, G. 2013. The ghtorrent dataset and tool suite. In Proceed- ings of the 10th Working Conference on Mining Software Reposi- tories, MSR '13. IEEE Press.
  22. Grant, R. N.; Kucher, D.; León, A. M.; Gemmell, J. F.; Raicu, D. S.; and Fodeh, S. J. 2018. Automatic extraction of informal topics from online suicidal ideation. BMC bioinformatics 19(8):211.
  23. Halavais, A. 2019. Overcoming terms of service: A proposal for ethical distributed research. Information, Communication & Soci- ety 22(11):1567-1581.
  24. Halder, K.; Kan, M.-Y.; and Sugiyama, K. 2019. Predicting helpful posts in open-ended discussion forums: A neural architecture. In NAACL.
  25. Hampton, K. N. 2017. Studying the Digital: Directions and Challenges for Digital Methods. Annual Review of Sociology 43(1):167-188.
  26. Hidey, C., and McKeown, K. 2019. Fixed that for you: Generating contrastive claims with semantic edits. In NAACL.
  27. Howison, J.; Wiggins, A.; and Crowston, K. 2011. Validity Is- sues in the Use of Social Network Analysis with Digital Trace Data. Journal of the Association for Information Systems; Atlanta 12(12):767-797.
  28. Hunter, D. 2003. Cyberspace as place and the tragedy of the digital anticommons. California Law Review 91:439.
  29. Ingram, M. 2019. Silicon Valley's Stonewalling. Columbia Jour- nalism Review.
  30. Japec, L.; Kreuter, F.; Berg, M.; Biemer, P.; Decker, P.; Lampe, C.; Lane, J.; O'Neil, C.; and Usher, A. 2015. Big Data in Survey Research: AAPOR Task Force Report. Public Opinion Quarterly 79(4):839-880.
  31. Jhaver, S.; Appling, D. S.; Gilbert, E.; and Bruckman, A. 2019. "did you suspect the post would be removed?": Understanding user reactions to content removals on reddit. Proc. ACM Hum.-Comput. Interact. 3. Jhaver, S.; Bruckman, A.; and Gilbert, E. 2019. Does transparency in moderation really matter? user behavior after content removal explanations on reddit. CHI.
  32. Jiang, J.-Y.; Chen, F.; Chen, Y.-Y.; and Wang, W. 2018. Learning to disentangle interleaved conversational threads with a siamese hi- erarchical network and similarity ranking. In NAACL, 1812-1822.
  33. Jonell, P.; Fallgren, P.; Dogan, F. I.; Lopes, J.; Wennberg, U.; and Skantze, G. 2019. Crowdsourcing a self-evolving dialog graph. In Proceedings of the 1st International Conference on Conversational User Interfaces, 14. ACM.
  34. Kasper, P.; Koncar, P.; Walk, S.; Santos, T.; Wölbitsch, M.; Strohmaier, M.; and Helic, D. 2017. Modeling user dynamics in collaboration websites. In Dynamics on and of Complex Networks, 113-133. Springer.
  35. Kunft, A.; Katsifodimos, A.; Schelter, S.; Rabl, T.; and Markl, V. 2017. Blockjoin: efficient matrix partitioning through joins. VLDB Endowment.
  36. Kunft, A.; Katsifodimos, A.; Schelter, S.; Breß, S.; Rabl, T.; and Markl, V. 2019. An intermediate representation for optimizing machine learning pipelines. VLDB Endowment.
  37. Lama, Y.; Hu, D.; Jamison, A.; Quinn, S. C.; and Broniatowski, D. A. 2019. Characterizing trends in human papillomavirus vaccine discourse on reddit (2007-2015): An observational study. JMIR public health and surveillance 5(1):e12480.
  38. Lazer, D., and Radford, J. 2017. Data ex Machina: Introduction to Big Data. Annual Review of Sociology 43(1):19-39.
  39. Lazer, D.; Pentland, A.; Adamic, L.; Aral, S.; Barabási, A.-L.; Brewer, D.; Christakis, N.; Contractor, N.; Fowler, J.; Gutmann, M.; Jebara, T.; King, G.; Macy, M.; Roy, D.; and Alstyne, M. V. 2009. Computational Social Science. Science 323(5915):721-723.
  40. Lu, J.; Sridhar, S.; Pandey, R.; Hasan, M. A.; and Mohler, G. 2019. Investigate transitions into drug addiction through text mining of reddit data. In KDD.
  41. Matias, J. N. 2016. Going dark: Social factors in collective action against platform operators in the reddit blackout. In CHI.
  42. Medvedev, A. N.; Delvenne, J.-C.; and Lambiotte, R. 2018. Mod- elling structure and predicting dynamics of discussion threads in online boards. Journal of Complex Networks 7(1):67-82.
  43. Mervis, J. 2019. Privacy concerns could derail Facebook data- sharing plan. Science 365(6460):1360-1361.
  44. Ozcan, F. 2017. Bayesian Nonparametric Models on Big Data. Ph.D. Dissertation, UC Irvine.
  45. Patel, K. S. 2018. Testing the Limits of the First Amendment: How Online Civil Rights Testing is Protected Speech Activity. Columbia Law Review 118(5):1473-1516.
  46. Pirina, I., and C ¸öltekin, C ¸. 2018. Identifying depression on reddit: The effect of training data. In EMNLP Workshop SMM4H.
  47. Puschmann, C. 2019. An end to the wild west of social media research: A response to Axel Bruns. Information, Communication & Society 22(11):1582-1589.
  48. Seering, J.; Wang, T.; Yoon, J.; and Kaufman, G. 2019. Moderator engagement and community development in the age of algorithms. New Media & Society 21(7):1417-1443.
  49. Shen, Q., and Rose, C. 2019. The discourse of online content moderation: Investigating polarized user responses to changes in reddit's quarantine policy. In Proceedings of the Third Workshop on Abusive Language Online, 58-69.
  50. Srinivasan, K. B.; Danescu-Niculescu-Mizil, C.; Lee, L.; and Tan, C. 2019. Content removal as a moderation strategy: Compliance and other outcomes in the changemyview community. CSCW.
  51. Tan, C. 2018. Tracing community genealogy: how new communi- ties emerge from the old. In ICWSM. Terrorism, T. A. 2019. Insights from the centre for analysis of the radical right's inaugural conference in london. https://bit.ly/ 39gKFCQ.
  52. Völske, M.; Potthast, M.; Syed, S.; and Stein, B. 2017. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, 59-63.
  53. Walker, S.; Mercea, D.; and Bastos, M. 2019. The disinforma- tion landscape and the lockdown of social platforms. Information, Communication & Society 22(11):1531-1543.
  54. Wang, F.-Y.; Carley, K. M.; Zeng, D.; and Mao, W. 2007. Social Computing: From Social Informatics to Social Intelligence. IEEE Intelligent Systems 22(2):79-83.
  55. Wang, A.; Hula, J.; Xia, P.; Pappagari, R.; McCoy, R. T.; Patel, R.; Kim, N.; Tenney, I.; Huang, Y.; Yu, K.; et al. 2019. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In ACL.
  56. Zhan, Y.; Zhang, Z.; Okamoto, J. M.; Zeng, D. D.; and Leischow, S. J. 2019. Underage juul use patterns: Content analysis of reddit messages. Journal of medical Internet research 21(9):e13038.
  57. Zheng, W., and Zhou, K. 2019. Enhancing conversational dialogue models with grounded knowledge. In CIKM.
  58. Zhuang, Y.; Xie, J.; Zheng, Y.; and Zhu, X. 2018. Quantifying context overlap for training word embeddings. In EMNLP.

Cited by

Sorted by citations

Authors
Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., & Blackburn, J.
Affiliations
  • Max Planck Institute for Informatics (DE)
  • University of Colorado Boulder (US)
  • Elon University (US)
  • Binghamton University (US)
ORCID
Venue
Proceedings of the International AAAI Conference on Web and Social Media
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2020-05-26
Volume
14
Pages
830-839
ISSN
  • 2162-3449 (ISSN-L)
  • 2334-0770
DOI
10.1609/icwsm.v14i1.7347
OpenAlex
W3002330681
Cited by
881
References
10
Citations by year
2026: 113 · 2025: 137 · 2024: 131 · 2023: 186 · 2022: 160 · 2021: 126 · 2020: 27 · 2019: 1
FWCI
177.09
Type
conference paper
Language
en
Open access
Licence
public-domain
Locations
  • Proceedings of the International AAAI Conference on Web and Social Media (publishedVersion, public-domain)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • MPG.PuRe (Max Planck Society) (submittedVersion)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
Reddit, social media data collection, data archiving, subreddits, online comments, real-time data acquisition, data cleaning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review