Mining big data to extract patterns and predict real-life outcomes.
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
This article aims to introduce the reader to essential tools that can be used to obtain insights and build predictive models using large data sets. Recent user proliferation in the digital environment has led to the emergence of large samples containing a wealth of traces of human behaviors, communication, and social interactions. Such samples offer the opportunity to greatly improve our understanding of individuals, groups, and societies, but their analysis presents unique methodological challenges. In this tutorial, we discuss potential sources of such data and explain how to efficiently store them. Then, we introduce two methods that are often employed to extract patterns and reduce the dimensionality of large data sets: singular value decomposition and latent Dirichlet allocation. Finally, we demonstrate how to use dimensions or clusters extracted from data to build predictive models in a cross-validated way. The text is accompanied by examples of R code and a sample data set, allowing the reader to practice the methods discussed here. A companion website (http://dataminingtutorial.com) provides additional learning resources. (PsycINFO Database Record
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Griffiths, T. L., & Steyvers, M. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0307752101 · Cited by 6,049 · cited 1×
“The common approach is to set 50/k and 0.1, or 200/n (where n is the number of variables in M, and k the number of clusters to be extracted; Griffiths & Steyvers, 2004).”
BackgroundInfluentialThe development of markers for the Big-Five factor structure
Goldberg, L. R. (1992) · Psychological Assessment · doi:10.1037/1040-3590.4.1.26 · Cited by 5,530
Experimental evidence of massive-scale emotional contagion through social networks
Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1320040111 · Cited by 3,687 · cited 13×
“I hate it when i’m taking a drink and all the ice attacks my face Telling inanimate objects to STAY when they look like they’re going to fall I feel stupid when I say what? a thousand times because I can’t hear i finally stop laughing . . . look back over at you and start all over again Ok, If we get caught here’s the…”
BackgroundInfluentialLazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al. (2009) · Science · doi:10.1126/science.1167742 · Cited by 3,365
The international personality item pool and the future of public-domain personality measures
Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006) · Journal of Research in Personality · doi:10.1016/j.jrp.2005.08.007 · Cited by 3,185
Singular value decomposition and least squares solutions
Golub, G. H., & Reinsch, C. (1970) · Numerische Mathematik · doi:10.1007/bf02163027 · Cited by 2,929 · cited 1×
“…on extracting patterns from a user–footprint matrix via two methods representative of two broad families: (a) singular value decomposition (SVD; Golub & Reinsch, 1970), representing eigendecomposition-based methods, projecting a set of data points into a set of dimensions; and (b) latent…”
MethodPrivate traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829
Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E. P., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073791 · Cited by 1,700
A Review of Facebook Research in the Social Sciences
Wilson, R. E., Gosling, S. D., & Graham, L. T. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612442904 · Cited by 1,226
topicmodels : An R Package for Fitting Topic Models
Grün, B., & Hornik, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v040.i13 · Cited by 1,072 · cited 2×
“Most programming languages provide offthe-shelf implementations of LDA: topicmodels package (Grün & Hornik, 2011) for R, scikit-learn package (Pedregosa et al., 2011) for Python, GibbsLDA (Phan & Nguyen, 2007) for C , and topic modeling toolbox (Steyvers, 2011) for Matlab.”
MethodThe Anatomy of the Facebook Social Graph
Ugander, J., Karrer, B., Backstrom, L., & Marlow, C. (2011) · arXiv · doi:10.48550/arxiv.1111.4503 · Cited by 819
Doing it all Bass-Ackwards: The development of hierarchical factor structures from the top down
Goldberg, L. R. (2006) · Journal of Research in Personality · doi:10.1016/j.jrp.2006.01.001 · Cited by 409 · cited 1×
“An interesting approach suitable for SVD has been discussed by Goldberg (2006).”
Method
Cited by
Artificial intelligence and big data in entrepreneurship: a new era has begun
Obschonka, M., & Audretsch, D. B. (2020) · Small Business Economics · doi:10.1007/s11187-019-00202-4 · Cited by 527
Predicting the Big 5 personality traits from digital footprints on social media: A meta-analysis
Azucar, D., Marengo, D., & Settanni, M. (2018) · Personality and Individual Differences · doi:10.1016/j.paid.2017.12.018 · Cited by 526
Using Machine Learning to Advance Personality Assessment and Theory
Bleidorn, W., & Hopwood, C. J. (2019) · Personality and Social Psychology Review · doi:10.1177/1088868318772990 · Cited by 255
Metadata
- Authors
- Kosinski, M., Wang, Y., Lakkaraju, H., & Leskovec, J.
- Venue
- Psychological Methods
- Published
- 2016-12
- Volume
- 21
- Issue
- 4
- Pages
- 493-506
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2560662722
- Cited by
- 200
- References
- 15
- FWCI
- 94.24
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.