The DOI matches its Crossref record; 48 of the 50 references it cites were found in a registry or an index.
Moral Foundations Twitter Corpus: A Collection of 35k Tweets Annotated for Moral Sentiment
Open accessAbstract
Research has shown that accounting for moral sentiment in natural language can yield insight into a variety of on- and off-line phenomena such as message diffusion, protest dynamics, and social distancing. However, measuring moral sentiment in natural language is challenging, and the difficulty of this task is exacerbated by the limited availability of annotated data. To address this issue, we introduce the Moral Foundations Twitter Corpus, a collection of 35,108 tweets that have been curated from seven distinct domains of discourse and hand annotated by at least three trained annotators for 10 categories of moral sentiment. To facilitate investigations of annotator response dynamics, we also provide psychological and demographic metadata for each annotator. Finally, we report moral sentiment classification baselines for this corpus using a range of popular methodologies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further comparisons (Tables S7 and S8 in Rathje et al. (75)) on another moral foundations dataset, on which the BERT model was not fine-tuned (MFTC; (81)), showed that BERT, despite being applied cross-context, outperformed GPT-4 on the Authority and Purity foundations, and GPT-4-Turbo also on the Loyalty foundation.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [81]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Advances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,093
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Support Vector Regression Machines
Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A. J., & Vapnik, V. N. (1996) · Cited by 4,666
Liberals and conservatives rely on different sets of moral foundations.
Graham, J., Haidt, J., & Nosek, B. A. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015141 · Cited by 4,560
The righteous mind: why good people are divided by politics and religion
(2012) · Choice Reviews Online · doi:10.5860/choice.49-7185 · Cited by 4,347
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
An Overview of Multi-Task Learning in Deep Neural NetworksPreprint
Ruder, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05098 · Cited by 3,827
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Graham, J., Haidt, J., Koleva, S. P., Motyl, M. S., Iyer, R., Wojcik, S. P., & Ditto, P. H. (2013) · Advances in experimental social psychology · doi:10.1016/b978-0-12-407236-7.00002-4 · Cited by 1,966
Culture as Consensus: A Theory of Culture and Informant Accuracy
Romney, A. K., Weller, S. C., & Batchelder, W. H. (1986) · American Anthropologist · doi:10.1525/aa.1986.88.2.02a00020 · Cited by 1,667
Emotion shapes the diffusion of moralized content in social networks
Brady, W. J., Wills, J. A., Jost, J. T., Tucker, J. A., & Van Bavel, J. J. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618923114 · Cited by 1,580 · cited 1×
“…Ji, & Dehghani, 2018), charitable donation (Hoover, Johnson, Boghrati, Graham, & Dehghani, 2018), social avoidance (Dehghani et al., 2016), diffusion (Brady, Wills, Jost, Tucker, & Van Bavel, 2017), and political discourse (Dehghani, Sagae, Sachdeva, & Gratch, 2014; Johnson & Goldwasser, 2018).”
Moral tribes: emotion, reason, and the gap between us and them
(2014) · Choice Reviews Online · doi:10.5860/choice.51-4949 · Cited by 978
Assessing miserly information processing: An expansion of the Cognitive Reflection Test
Toplak, M. E., West, R. F., & Stanovich, K. E. (2014) · Thinking & Reasoning · doi:10.1080/13546783.2013.844729 · Cited by 864
Automatic personality assessment through social media language.
Park, G., Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Kosinski, M., Stillwell, D. J., Ungar, L. H., & Seligman, M. E. P. (2015) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000020 · Cited by 802
Facebook language predicts depression in medical records
Eichstaedt, J. C., Smith, R. J., Merchant, R. M., Ungar, L. H., Crutchley, P., Preoţiuc-Pietro, D., Asch, D. A., & Schwartz, H. A. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1802331115 · Cited by 643
Multi-task Sequence to Sequence LearningPreprint
Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, Ł. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06114 · Cited by 606
Predicting the Big 5 personality traits from digital footprints on social media: A meta-analysis
Azucar, D., Marengo, D., & Settanni, M. (2017) · Personality and Individual Differences · doi:10.1016/j.paid.2017.12.018 · Cited by 572
Ellemers, N., van der Toorn, J., Paunov, Y., & van Leeuwen, T. N. (2019) · Personality and Social Psychology Review · doi:10.1177/1088868318811759 · Cited by 469
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K., & Bethard, S. J. (2013) · Empirical Methods in Natural Language Processing · Cited by 422
Gray, K. J., & Graham, J. (2018) · Cited by 391
Kinship, Cooperation, and the Evolution of Moral Systems*
Enke, B. (2019) · The Quarterly Journal of Economics · doi:10.1093/qje/qjz001 · Cited by 365
A solution to the mysteries of morality.
DeScioli, P., & Kurzban, R. (2012) · Psychological Bulletin · doi:10.1037/a0029065 · Cited by 301
Curry, O. S., Jones Chesters, M., & Van Lissa, C. J. (2019) · Journal of Research in Personality · doi:10.1016/j.jrp.2018.10.008 · Cited by 235
Moralization in social networks and the emergence of violence during protests
Mooijman, M., Hoover, J., Lin, Y., Ji, H., & Dehghani, M. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0353-0 · Cited by 229
Self-citationGaining insights from social media language: Methodologies and challenges.
Kern, M. L., Park, G., Eichstaedt, J. C., Schwartz, H. A., Sap, M., Smith, L. K., & Ungar, L. H. (2016) · Psychological Methods · doi:10.1037/met0000091 · Cited by 221
Garten, J., Hoover, J., JOHNSON, K., Boghrati, R., Iskiwitch, C., & Dehghani, M. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0875-9 · Cited by 190
Self-citationPurity homophily in social networks.
Dehghani, M., JOHNSON, K., Hoover, J., Sagi, E., Garten, J., Parmar, N., Vaisey, S., Iliev, R. I., & Graham, J. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000139 · Cited by 169
Self-citationGarcia, D., & Sikström, S. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2013.10.001 · Cited by 161
The Benefits of a Model of Annotation
Passonneau, R. J., & Carpenter, B. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00185 · Cited by 151 · cited 1×
“…approach because it is a well-known and straightforward method for aggregating human annotations; however, we also believe that applying more sensitive annotation aggregation methods (e.g., see Passonneau & Carpenter, 2014; Paun et al., 2018) to the MFTC will be a fruitful area for future research.”
MethodMeasuring Moral Rhetoric in Text
Sagi, E., & Dehghani, M. (2013) · Social Science Computer Review · doi:10.1177/0894439313506837 · Cited by 128
Self-citationAutomated text analysis in psychology: methods, applications, and future developments
Iliev, R., Dehghani, M., & Sagi, E. (2014) · Language and Cognition · doi:10.1017/langcog.2014.30 · Cited by 124
Self-citationAssessing Rater Performance without a "Gold Standard" Using Consensus Theory
Weller, S. C., & Mann, N. C. (1997) · Medical Decision Making · doi:10.1177/0272989x9701700108 · Cited by 110
Comparing Bayesian Models of Annotation
Paun, S., Carpenter, B., Chamberlain, J., Hovy, D., Kruschwitz, U., & Poesio, M. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00040 · Cited by 94
Using Topic Modeling to Improve Prediction of Neuroticism and Depression in College Students
Resnik, P., Garron, A., & Resnik, R. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1133 · Cited by 93
Classification of Moral Foundations in Microblog Political Discourse
Johnson, K., & Goldwasser, D. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1067 · Cited by 79
Hoover, J., JOHNSON, K., Boghrati, R., Graham, J., & Dehghani, M. (2018) · Collabra Psychology · doi:10.1525/collabra.129 · Cited by 73
Self-citationIdentifying Patients with Depression Using Free-text Clinical Documents
Zhou, L., Baughman, A. W., Lei, V. J., Lai, K. H., Navathe, A. S., CHANG, F. Y., Margarita, S., Topaz, M., Zhong, F., Murrali, M., et al. (2015) · Studies in health technology and informatics · doi:10.3233/978-1-61499-564-7-629 · Cited by 55
Acquiring Background Knowledge to Improve Moral Value Prediction
Lin, Y., Hoover, J., Portillo-Wightman, G., Park, C., Dehghani, M., & Ji, H. (2018) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2018.8508244 · Cited by 53
Self-citationDehghani, M., Sagae, K., Sachdeva, S. S., & Gratch, J. S. (2013) · Journal of Information Technology & Politics · doi:10.1080/19331681.2013.826613 · Cited by 51
Self-citationIndividual Differences in the Moralization of Everyday Life
Lovett, B. J., Jordan, A., & Wiltermuth, S. S. (2012) · Ethics & Behavior · doi:10.1080/10508422.2012.659132 · Cited by 43
Measuring the importance of context when modeling language comprehension
Garten, J., Kennedy, B., Sagae, K., & Dehghani, M. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01200-w · Cited by 23
Self-citationMoral Foundations Dictionary 2.0Preprint
darloap, C., & Frimer, J. (2019) · OSF Preprints (OSF Preprints) · doi:10.17605/osf.io/ezn37 · Cited by 20
Incorporating Demographic Embeddings Into Language Understanding
Garten, J., Kennedy, B., Hoover, J., Sagae, K., & Dehghani, M. (2019) · Cognitive Science · doi:10.1111/cogs.12701 · Cited by 15
Self-citationIndirect reciprocity and the evolution of “moral signals”
Smead, R. (2009) · Biology & Philosophy · doi:10.1007/s10539-009-9175-9 · Cited by 13
Keen, I. (2015) · The Australian Journal of Anthropology · doi:10.1111/taja.12154 · Cited by 3
Research works printed but not linked: 4
- Boyd R. L., 2015, Ninth International AAAI
- Garten J., 2016, Proceedings of IJCAI 2016 Workshop on Computational Modeling of Attitudes
- Poulshock J. W. (2006). Language and morality: Evolution, altruism, and linguistic moral mechanisms (Unpublished doctoral dissertation). University of Edinburgh, United Kingdom.
- Zhou L., 2015, Studies in Health Technology and Informatics, 216, 629
Cited by
Sorted by citations
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111
“Further comparisons (Tables S7 and S8in Rathje et al. (75)) on another moral foundations dataset, on which the BERT model was not fine-tuned (MFTC; (81)), showed that BERT, despite being applied cross-context, outperformed GPT-4 on the Authority and Purity foundations, and GPT-4-Turbo also on the…”
Self-citationMacanovic, A. (2022) · Social Science Research · doi:10.1016/j.ssresearch.2022.102784 · Cited by 120
Language-based game theory in the age of artificial intelligence
Capraro, V., Paolo, R. D., Perc, M., & Pizziol, V. (2024) · Journal of The Royal Society Interface · doi:10.1098/rsif.2023.0720 · Cited by 97
Kennedy, B., Atari, M., Davani, A. M., Yeh, L., Omrani, A., Kim, Y., Coombs, K., Havaldar, S., Portillo-Wightman, G., Gonzalez, E., et al. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-021-09569-x · Cited by 98
Self-citationAgainst Empathy Bias: The Moral Value of Equitable Empathy
Fowler, Z., Law, K. F., & Gaesser, B. (2021) · Psychological Science · doi:10.1177/0956797620979965 · Cited by 95
Metadata
- Authors
- Hoover, J., Portillo-Wightman, G., Yeh, L., Havaldar, S., Davani, A. M., Lin, Y., Kennedy, B., Atari, M., Kamel, Z., Mendlen, M., Moreno, G., Park, C., Chang, T. E., Chin, J., Leong, C., Leung, J. Y., Mirinjian, A., & Dehghani, M.
- Affiliations
- University of Southern California (US)
- Rensselaer Polytechnic Institute (US)
- Corresponding author
- Dehghani, Morteza
- ORCID
- Venue
- Social Psychological and Personality Science
- Publisher
- SAGE Publications
- Published
- 2020-02-19
- Volume
- 11
- Issue
- 8
- Pages
- 1057-1071
- ISSN
- 1948-5506 (ISSN-L)
- 1948-5514
- OpenAlex
- W3008655042
- Semantic Scholar
- a95ee39bda53cf7cbf1677d1bbac9d2be34fcc5a
- Cited by
- 177
- References
- 50
- Citations by year
- 2026: 10 · 2025: 19 · 2024: 36 · 2023: 41 · 2022: 39 · 2021: 24 · 2020: 7
- FWCI
- 10.91
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Locations
- Social Psychological and Personality Science (publishedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- Army Research Laboratory
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
- Keywords
- moral sentiments, moral foundations, Twitter, annotated corpus, social media discourse, message dissemination, protest dynamics, social distancing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
