Garbage in, Garbage Out: Data Collection, Quality Assessment and Reporting Standards for Social Media Data Use in Health Research, Infodemiology and Digital Disease Detection
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Social media have transformed the communications landscape. People increasingly obtain news and health information online and via social media. Social media platforms also serve as novel sources of rich observational data for health research (including infodemiology, infoveillance, and digital disease detection detection). While the number of studies using social data is growing rapidly, very few of these studies transparently outline their methods for collecting, filtering, and reporting those data. Keywords and search filters applied to social data form the lens through which researchers may observe what and how people communicate about a given topic. Without a properly focused lens, research conclusions may be biased or misleading. Standards of reporting data sources and quality are needed so that data scientists and consumers of social media research can evaluate and compare methods and findings across studies. OBJECTIVE: We aimed to develop and apply a framework of social media data collection and quality assessment and to propose a reporting standard, which researchers and reviewers may use to evaluate and compare the quality of social data across studies. METHODS: We propose a conceptual framework consisting of three major steps in collecting social media data: develop, apply, and validate search filters. This framework is based on two criteria: retrieval precision (how much of retrieved data is relevant) and retrieval recall (how much of the relevant data is retrieved). We then discuss two conditions that estimation of retrieval precision and recall rely on--accurate human coding and full data collection--and how to calculate these statistics in cases that deviate from the two ideal conditions. We then apply the framework on a real-world example using approximately 4 million tobacco-related tweets collected from the Twitter firehose. RESULTS: We developed and applied a search filter to retrieve e-cigarette-related tweets from the archive based on three keyword categories: devices, brands, and behavior. The search filter retrieved 82,205 e-cigarette-related tweets from the archive and was validated. Retrieval precision was calculated above 95% in all cases. Retrieval recall was 86% assuming ideal conditions (no human coding errors and full data collection), 75% when unretrieved messages could not be archived, 86% assuming no false negative errors by coders, and 93% allowing both false negative and false positive errors by human coders. CONCLUSIONS: This paper sets forth a conceptual framework for the filtering and quality evaluation of social data that addresses several common challenges and moves toward establishing a standard of reporting social data. Researchers should clearly delineate data sources, how data were accessed and collected, and the search filter building process and how retrieval precision and recall were calculated. The proposed framework can be adapted to other public social media platforms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Agresti, A. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/0471249688 · Cited by 6,746
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523
Eysenbach, G. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1157 · Cited by 1,351 · cited 3×
“…as novel sources of observational data for infodemiology, public health surveillance (infoveillance, digital disease detection) [6-11], tracking health attitudes and behavioral intention [6, 7, 9,12-16], and measuring community-level psychological characteristics related to health outcomes [17,18].”
BackgroundTobacco Use Among Middle and High School Students — United States, 2011–2015
Singh, T., Arrazola, R. A., Corey, C. G., Husten, C. G., Neff, L. J., Homa, D. M., & King, B. A. (2016) · MMWR. Morbidity and Mortality Weekly Report · doi:10.15585/mmwr.mm6514a1 · Cited by 1,240
The use of social networking sites for public health practice and research: A systematic review
Capurro, D., Cole, K., Echavarría, M. I., Joe, J., Neogi, T., & Turner, A. M. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.2679 · Cited by 237 · cited 1×
“The number of studies that rely on social media data is increasing [43].”
Background
Cited by
Chae, S. H., Kim, Y., Lee, K.-S., & Park, H.-S. (2020) · JMIR mHealth and uHealth · doi:10.2196/17216 · Cited by 235
“We think that the different exercise motions of other patients contaminated the data consistency and had a bad influence on the ML model [38].”
BackgroundSelf-citationRadwan, E., Radwan, A., & Radwan, W. (2020) · Heliyon · doi:10.1016/j.heliyon.2020.e05807 · Cited by 111
Arillotta, D., Floresta, G., Guirguis, A., Corkery, J. M., Catalani, V., Martinotti, G., Sensi, S. L., & Schifano, F. (2023) · Brain Sciences · doi:10.3390/brainsci13111503 · Cited by 104
“Consistent with the above, and in line with previous social media-based work [36–41], we aimed to assess the possible impact of GLP-1 agonists on mental health in general as being perceived and discussed in popular open web platforms [42].”
ResultThe use of social media for health research purposes: scoping review
Bour, C., Ahne, A., Schmitz, S., Perchoux, C., Dessenne, C., & Fagherazzi, G. (2021) · Journal of Medical Internet Research · doi:10.2196/25736 · Cited by 56
Methods for Coding Tobacco-Related Twitter Data: A Systematic Review
Lienemann, B. A., Unger, J. B., Cruz, T. B., & Chu, K.-H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7022 · Cited by 48
“Search Keyword Selection Kim and colleagues [11] proposed a framework of three steps to develop and validate search filters.”
BackgroundInfluential
Metadata
- Authors
- Kim, Y., Huang, J., & Emery, S.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2016-02-26
- Volume
- 18
- Issue
- 2
- Pages
- e41
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W2283694230
- Cited by
- 176
- References
- 5
- FWCI
- 13.36
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.