Studying Reddit: A Systematic Overview of Disciplines, Approaches, Methods, and Ethics

“Studies such as these suggest that researchers may need to consider the generalizability and representativeness of models built using Reddit data, particularly in the context of ongoing conversations regarding language modeling (see Bender et al., 2021).”

“The Pushshift dataset contains submissions and comments posted on Reddit since June 2005, and has been popular for researchers due to its ease of use and larger querying limits (Baumgartner et al., 2020).”

“Tufekci (2014) once called Twitter the “model organism” for academic study because tweets are considered to be “public,” because Twitter has open APIs which fosters easy data collection, and because Twitter users often respond to world events as they unfold, making it a useful location to gather…”

“For example, personally identifiable information from more than 87 million Facebook users was collected in an academic study, but then the data were used by Cambridge Analytica to micro-target political advertisements (Isaak & Hanna, 2018).”

“Moreover, the smaller the community being studied in any context, the more difficult it is to maintain the anonymity of research subjects (Saunders et al., 2015), even without the additional complication of direct quotes being discoverable by search engines (Bruckman, 2002; Markham, 2012).”

“Fiesler and Proferes (2018) conducted a survey of Twitter users to assess how they felt about their data being used.”

“One of the major concerns of research using public data is privacy, including what constitutes “public,” whether measures should be taken to prevent the discoverability of data sources, and to what extent research subjects should be disguised (Bruckman, 2002; Markham, 2012; Zimmer, 2018).”

“Reddit has also been home to a number of prominent and controversial events, such as attempts to identify the Boston-city bombing terrorists (Starbird et al., 2014); a massive leak of hacked celebrity photos (Marwick, 2017); the coordinated attempt to take on short-sellers of the GameStop stock (Roose, 2021); as well…”

“One of the major concerns of research using public data is privacy, including what constitutes “public,” whether measures should be taken to prevent the discoverability of data sources, and to what extent research subjects should be disguised (Bruckman, 2002; Markham, 2012; Zimmer, 2018).”

“…communities and among Institutional Review Boards (IRBs), there is disagreement about the ethical practices that should follow from the use of public data for research purposes and if, or when, using social media constitutes human subject research (Metcalf & Crawford, 2016; Vitak et al., 2016).”

“Researchers have used social media data for a wide range of purposes—from predicting postpartum depression from Facebook posts (De Choudhury et al., 2014) to trying to predict movements in the stock market based on the sentiment of Tweets (Mittal & Goel, 2010).”

“Researchers have used social media data for a wide range of purposes—from predicting postpartum depression from Facebook posts (De Choudhury et al., 2014) to trying to predict movements in the stock market based on the sentiment of Tweets (Mittal & Goel, 2010).”

“Some have suggested that dataset sharing upon request is a good compromise (Fiesler, 2019), and Twitter has dealt with this issue by changing its terms of service so that full JSON data is not allowed to be redistributed, instead, only Tweet IDs can be shared which must then be “rehydrated” (see Summers, 2016/2021).”

“For these reasons, Reddit has an ecosystem of bots created by its user base to help in several ways, such as content moderation (Jhaver et al., 2019), adding functionality through summarizing information and linking to other websites, or providing humor through parody bot accounts (Long et al.,…”

“While there are a few overarching Reddit rules about content, subreddits vary considerably regarding what they allow, and in their specific cultures and norms (Chandrasekharan et al., 2018; Fiesler et al., 2018).”

“Hudson and Bruckman (2004) found in their study of online chatrooms that the smaller the group being observed, the less comfortable they were with researcher presence.”

“, 2020), sexist (Farrell et al., 2019), and vitriolic political discourse (Mills, 2018).”

“…communities and among Institutional Review Boards (IRBs), there is disagreement about the ethical practices that should follow from the use of public data for research purposes and if, or when, using social media constitutes human subject research (Metcalf & Crawford, 2016; Vitak et al., 2016).”

“Open science is a laudable goal, particularly in the wake of concerns over a reproducibility crisis in social science research (Baker, 2016), though concerns about ethics are one potential barrier to sharing datasets and other research artifacts (Wacharamanotham et al.”

“While there are a few overarching Reddit rules about content, subreddits vary considerably regarding what they allow, and in their specific cultures and norms (Chandrasekharan et al., 2018; Fiesler et al., 2018).”

““Throwaway” accounts, or single-purpose accounts created for limited time use, are commonly used when users do not want a post or comment associated with their main or primary account, such as sharing sensitive or personal information (Ammari et al., 2019; Leavitt, 2015).”

“While this is a fairly common practice in research papers (Ayers et al., 2018), where subreddit content is potentially sensitive (such as when the quote involves mental health, drug use, sexual activity, and is potentially from a minor), there may be outsized safety or privacy risks to those data subjects if their con…”

““Throwaway” accounts, or single-purpose accounts created for limited time use, are commonly used when users do not want a post or comment associated with their main or primary account, such as sharing sensitive or personal information (Ammari et al., 2019; Leavitt, 2015).”

“And a group of Danish researchers were criticized after they publicly released a data set of nearly 70,000 users of the online dating site OkCupid, which included “usernames, age, gender, location, what kind of relationship (or sex) they are interested in, personality traits, and answers to thousands of personal profi…”

“Open science is a laudable goal, particularly in the wake of concerns over a reproducibility crisis in social science research (Baker, 2016), though concerns about ethics are one potential barrier to sharing datasets and other research artifacts (Wacharamanotham et al., 2020).”

“Many (though not all) ethics review bodies would likely view most Reddit data as “public” and therefore not require researchers to seek consent (see Vitak et al., 2017).”

“However, PushShift is not an exact mirror of data from Reddit (see: Gaffney & Matias, 2018 and the rejoinder from Baumgartner, 2018 for more).”

“As Fiesler et al. (2020) note in their analysis of data scraping provisions in a large number of social media TOS, these policies largely lack the context that would be relevant to an ethical decision (e.g., what kind of data or what it is being used for).”

“In recent years, major social media platforms like Facebook (Freelon, 2018), Instagram, and Twitter (Bruns, 2019) have begun restricting API access (Tromble, 2021).”

“…et al., 2014); a massive leak of hacked celebrity photos (Marwick, 2017); the coordinated attempt to take on short-sellers of the GameStop stock (Roose, 2021); as well as sometimes racist (Mittos et al., 2020), sexist (Farrell et al., 2019), and vitriolic political discourse (Mills, 2018).”

“…subreddit content is potentially sensitive (such as when the quote involves mental health, drug use, sexual activity, and is potentially from a minor), there may be outsized safety or privacy risks to those data subjects if their content is shared beyond its intended context (Dym & Fiesler, 2020).”

“…these reasons, Reddit has an ecosystem of bots created by its user base to help in several ways, such as content moderation (Jhaver et al., 2019), adding functionality through summarizing information and linking to other websites, or providing humor through parody bot accounts (Long et al., 2017).”

“…events, such as attempts to identify the Boston-city bombing terrorists (Starbird et al., 2014); a massive leak of hacked celebrity photos (Marwick, 2017); the coordinated attempt to take on short-sellers of the GameStop stock (Roose, 2021); as well as sometimes racist (Mittos et al.,…”

“Some have suggested that dataset sharing upon request is a good compromise (Fiesler, 2019), and Twitter has dealt with this issue by changing its terms of service so that full JSON data is not allowed to be redistributed, instead, only Tweet IDs can be shared which must then be “rehydrated” (see Summers, 2016/2021).”

“Their findings showed that users were largely unaware that their data were used for research purposes and that perceptions of data use varied by contextual factors, such as who the researchers are and the topic of study, a finding echoing that of Beninger (2017).”

“Although there have been calls for more open discussion of ethical issues within these communities to help establish these kinds of norms (Bruckman et al., 2017), explicit discussion of ethical considerations in this dataset of papers is uncommon.”

“, 2019), and vitriolic political discourse (Mills, 2018).”

“There is also sometimes gaming of Reddit’s sorting algorithm which can also drive conversational patterns (Shepherd, 2020).”

“data such as those in online forums may also be an alternative to asking survivors to recount their stories, although researchers should thoroughly weigh the ethical and privacy implications of using such data [144] to avoid potential violations of users’ expectations [54].”

“A comprehensive review of Reddit research between 2010 and 2020 revealed a steady increase in the number of papers published year over year [91].”

“This relatively rare occurrence of explicit research ethics discussions is in line with the findings from a recent systematic literature review of Reddit-based research studies [86].”

“The ethical issues of research on Reddit are rarely engaged with in academic publications (Proferes et al., 2021).”

“Reddit features user-created forums called “subreddits” or “communities” that organize posts by subjects [124, 149].”

“Proferes et al., (2021) identified 727 such studies published between 2010 and 2020-May.”

“Proferes et al. (2021) further pointed out that in the case of Reddit, two prominent contextual factors—community culture and demographics—may influence model generalizability.”

“Reddit is an increasingly utilized data source in HCI and adjacent disciplines [59]; we selected it due to its broad coverage of gig work types and popularity among workers.”

“As discussed by Proferes et al [12], there remains a lack of uniformity in the current ethical practices among researchers using public data from Reddit.”

“Reviews of research conducted using reddit data have found that the generalizability of results from reddit to other contexts are promising, with certain caveats, and this is an important area for additional research (Proferes et al. 2021).”

“Notably, Proferes et al. [21] provide procedure descriptions for gathering their data, the metadata of the final dataset, and even the list of subreddits selected as the sources.”

“However, we align with the growing emphasis in CSCW and HCI research on the ethical handling of mental health-related content in online communities [2, 12, 23, 71] highlighted the importance of protecting the confidentiality of sensitive health information shared in online communities like Reddit.”

“Reddit is a distinctive platform compared to Facebook and Twitter (Table 1), as users can maintain some degree of anonymity because the platform does not require identity verification, which may increase users’ willingness to engage in (uncivil) political discussions (Proferes et al., 2021).”

“Previous research has highlighted ethical challenges in using data from social media platforms such as Reddit [128,164], including concerns about privacy, user anonymity, and data discoverability.”

“[75] suggest that researchers should consider the risks to subjects when including usernames and direct quotations in published work, especially if the data could be considered sensitive.”

“A number of ideas have been proposed to manage this conflict, although there is still no clearly defined solution [85-87].”

“Strategies for disguising the source of content are increasingly common in research fields investigating social media data [26, 78], as well as recommended by digital-safety researchers [9], particularly when obtaining informed consent is not feasible.”

“Reddit is home to over 52 million monthly active users and is both a prominent and influential data source (37).”

“Throwaway accounts tend to be commonly used by users who wish to avoid associating their main account with the purpose that made them create the throwaway account (Ammari et al., 2018; Proferes et al., 2021).”

“…public archive for Reddit and complies with Reddit’s terms of service in data collection and maintenance [12], [59]. pushshift.io ingests data through Reddit’s offi-cial application programming interface (API) and handles removal requests, although some removal requests may not be timely [11], [58].”

“Among these social media platforms, Reddit ranks the 5 th most visited website in the United States, having 52 million users per day and more than 430 million active users per month [15, 16].”

“First, it relies on data from a single social media platform, Reddit, where the users’ demographic skews towards being male, young, white, and highly educated (63% of Reddit users have a Bachelor’s degree or higher) [12, 57].”

“Probing Questions In recent years, data from online public forums, have been used for research to surface a multitude of harms and concerns facing various communities, drawing attention across multiple disciplines [87].”

“The anonymous nature of Reddit may allow for the collection of disinhibited youth mental health narratives revealing information that may otherwise not be available through traditional forms of data collection (e.g., interviews, surveys) (Pew Research Center, 2019; Proferes et al. 2021).”

“The IRB deemed that we did not need to apply for ethics approval, according to the interpretation shared by other IRBs [88] that the data used in our research are all publicly available discussion data, they are pseudoanonymous data, demographic information cannot be easily obtained, and they did…”

“Reddit has been a frequent topic of study for empirical investigations into the different types and values of online communities [70].”

“!) social media platforms, which, making things worse, mostly attract male users from the USA (Ruiz Soler, 2017; Proferes et al., 2021).”

“However, Reddit is often viewed as a social media site in CSCW research [105, 108] rather than a peer-production site that creates knowledge.”

“One such platform is Reddit, a social sharing and news aggregation site, where individual communities, known as subreddits, are free to enact their own rules and policies [17, 19, 59, 66].”

“Text data from online forums, such as Reddit, has been used to study issues affecting diverse communities [37].”

“There are ethical considerations with researchers using public data on social media platforms while at the same time respecting confidentiality and consent, such as whether this use aligns with the users’ expectations when they registered (Proferes et al., 2021).”

“…for asexual and aromantic indivi-duals—Reddit was selected as another prominent site of anonymous online interaction where individuals can gather in subreddits to have conversations about topics they are passionate about or struggle with, such as aceness or aroness (Proferes et al., 2021).”

“…involving personal expression, intimate sharing, and subjective experiences – on sensitive, controversial, or stigmatized topics, such as mental health, involuntary celibacy, and youth withdrawal (Maxwell et al., 2020; Proferes et al., 2021; Singer et al., 2014, p. 517; Sparrow et al., 2020).”

“As an additional precaution, all examples presented in the manuscript are paraphrased from direct quotes [41].”

“Per the recommendations of Proferes et al. (2021), we refrain from directly quoting the Reddit questions and instead show rephrased and summarized examples in Table 1.”

“Research conducted on Reddit data is biased to a WEIRD (Western, Educated, Industrialized, Rich, Democratic) demographic, and results may not generalize to a broader set of users (Proferes et al., 2021).”

“Drawing from this, Shilton et al. (2021) recommend that those collecting social media data consider issues of power and awareness to help inform ethical decision-making in this space.”

“First, in comparison to Twitter, Reddit shares many of the same research friendly features (e.g., responsiveness to current events, public posts, and available APIs) (Proferes et al., 2021), but is organized into what are called subreddits.”

“Alongside the analysis of computer vision datasets are larger conversations about the ethics of using public data for research purposes, particularly as the use of data scraped from the web becomes more ubiquitous [83].”

“While we felt that it was important to provide direct quotes to ground and contextualise the results, we omitted the real usernames of the posters (in line with 90% of the studies reported in [67]).”

“To our knowledge, there are no prior examinations of how publications in criminology report their ethical approval and ethics processes; however, such analyses have been undertaken for gray literature publications (Anabo et al., 2019) and research on internet message board Reddit (Proferes et al., 2021); these studies…”

“Scholars have noted an uptake in research using computational-driven analyses built on Reddit data (Proferes et al., 2021) and directives for scholar-ship as a whole addressing health communication (Rains, 2020).”

“Leveraging Twitter data to analyze public sentiment in the modern research literature is common [7]; however, it is partial to another popular platform, Reddit, which has shown to be influential in its rights [8].”

“This sparked various studies [23], including large-scale temporal analysis on users.”

“In line with ethical guidelines (Proferes et al. 2021), we include our full codebooks here: https://osf.io/2k7tj/.”

“For users who constructed this distinction, it proved foundational to how they engaged in subsequent attribution work.”

“A recent analysis of 727 publications since 2010 [29] shows that Reddit has become a substantially more popular medium in various disciplines, including computer science and software development processes.”

“Further, while the participants on Reddit accepted the platform ToS that the data is publicly available did not explicitly consent to their data being used for research purposes [50].”

“For example, Leavitt and Robinson reported that Reddit users used the functions on the platform such as voting to appropriate the visibility of information during crises, where in-time and accurate information is crucial [32].”

“Reddit ’ s anonymity and facilitated in-depth discussions within specific communities (subreddits) create a unique environment for comprehensive mental health research (Proferes et al., 2021).”

“Structured into thousands of “ subreddits, ” each dedicated to a speci fi c topic, Reddit offers a tailored experience to its users (Proferes et al. 2021), enabling them to navigate through a wide array of content areas, from news to technology and beyond.”

“In stark contrast to the vibrant body of work on Black Twitter [13, 25, 49, 52, 65, 79], and despite increasing research attention on Reddit in general [109], there is a serious gap in the literature specifically on the experiences of Black Redditors.”

“In this paper, we only present results on an aggregated level, and any quotes presented are carefully rephrased to maintain the privacy of the corresponding users [41].”

“…with recruitment from Reddit, noting that Reddit ’ s broad demographics and highly speci fi c subreddits offer the ability to communicate with members of distinct communities or af fi nities quickly, easily, and in suf fi cient numbers (Amaya et al., 2021; Proferes et al., 2021; Shatz, 2017).”

“We focused on Reddit data given its heterogeneity and scale [71], which has led to its use in numerous HCI studies (e.g., [19, 46, 55, 75]).”

“The IRB deemed that our research was exempt from human subjects review, according to the interpretation shared by other IRBs [66], the data used are all publicly available and expected to be viewed by the public, and they did not involve sensitive information such as physiological signals.”

“Due to the sensitive nature of the topics discussed, two steps were taken to further ensure anonymity of the Redditors quoted (Proferes et al., 2021): 1) no real usernames are provided in the text or in any shared data (Ayers et al., 2018), and 2) the ease of reverse identification was checked by…”

“Reddit has become one of the most popular social networking spaces and can generate insights into online communities (Proferes et al., 2021).”

“Datasets arising in applications as diverse as crime prediction [3], social media analytics [4], and epidemiology [5] consist of temporal interaction events between distinct categories of items—for example, users and comment threads in social media data or infected persons and locations in…”

“Through graph unlearning, researchers and organizations can strike a delicate equilibrium between leveraging the power of machine learning in understanding social dynamics and respecting and preserving users’ privacy rights in the ever-changing social network landscape [68], [69].”

“Utilizing internet forums as a method of analysis is becoming increasingly popular ( Proferes et al., 2021 ; van Hout & Hearne, 2017 ) with sites such as Reddit ( Pestana et al., 2020 ), Twitter ( Tassone et al., 2020 ) and YouTube ( Andersson & Kjellgren, 2019 ; Prevete et al., 2021 ) all being…”

“All of the submission quotes presented in this paper have been rephrased for privacy consideration and demonstration purpose (Proferes et al. 2021).”

“Hence, using Reddit data for research purposes typically exempts from institutional ethical review [10], [46].”

“The platform architecture, cultural practices, and terms of service establish that r/AskReddit posts are intended for broad public consumption (Gliniecka, 2023; Proferes et al., 2021).”

“…and should be considered a valuable piece of data-triangulation when considered in the context of the larger body of work that draws upon techniques like field work, interviews, etc. Additionally, the demographics of Reddit community members are also known to skew towards young male Americans [64].”

“2) Reddit is one of a few online platforms that have been actively studied [69] by many researchers due to the richness of UGC [64, 81].”

“Unlike other social media platforms such as Facebook or X/Twitter, Reddit allows users to interact using pseudonyms without revealing their true identities (Haythornthwaite et al., 2018; Proferes et al., 2021).”

“Reddit has been used in research across a range of disciplines (Proferes et al. 2021), with accessing, searching and collecting of data for research purposes not excluded from its user agreement.”

“…through topic-specific subreddits is widely used for mental health support-seeking, enabling targeted analysis of depression-related discussions (Proferes et al., 2021), ii) the platform’s anonymity facilitates more open self-disclosure about mental health experiences, and iii) its threaded…”

“58% of Reddit users are between the ages of 19 and 34, and 57% are men, mostly based in urban geographical locations in wealthier nations [52].”