Sliding into My DMs: Detecting Uncomfortable or Unsafe Sexual Risk Experiences within Instagram Direct Messages Grounded in the Perspective of Youth
Open accessAbstract
We collected Instagram data from 150 adolescents (ages 13-21) that included 15,547 private message conversations of which 326 conversations were flagged as sexually risky by participants. Based on this data, we leveraged a human-centered machine learning approach to create sexual risk detection classifiers for youth social media conversations. Our Convolutional Neural Network (CNN) and Random Forest models outperformed in identifying sexual risks at the conversation-level (AUC=0.88), and CNN outperformed at the message-level (AUC=0.85). We also trained classifiers to detect the severity risk level (i.e., safe, low, medium-high) of a given message with CNN outperforming other models (AUC=0.88). A feature analysis yielded deeper insights into patterns found within sexually safe versus unsafe conversations. We found that contextual features (e.g., age, gender, and relationship type) and Linguistic Inquiry and Word Count (LIWC) contributed the most for accurately detecting sexual conversations that made youth feel uncomfortable or unsafe. Our analysis provides insights into the important factors and contextual features that enhance automated detection of sexual risks within youths' private conversations. As such, we make valuable contributions to the computational risk detection and adolescent online safety literature through our human-centered approach of collecting and ground truth coding private social media conversations of youth for the purpose of risk classification.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288 · see citing works
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · see citing works
Explainable AI: From black box to glass box
Rai, A. (2020) · Journal of the Academy of Marketing Science · doi:10.1007/s11747-019-00710-5 · Cited by 1,317 · see citing works
Mental Health Discourse on reddit: Self-Disclosure, Social Support, and Anonymity
De Choudhury, M., & De, S. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14526 · Cited by 836 · see citing works
Self-citationBruckman, A. (2002) · Ethics and Information Technology · doi:10.1023/a:1021316409277 · Cited by 402 · see citing works
Viewpoint: Human-in-the-loop Artificial Intelligence
Zanzotto, F. M. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11345 · Cited by 380 · see citing works
Andalibi, N., Haimson, O. L., De Choudhury, M., & Forte, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858096 · Cited by 378 · see citing works
A review of online grooming: Characteristics and concerns
Whittle, H., Hamilton-Giachritsis, C., Beech, A., & Collings, G. (2013) · Aggression and Violent Behavior · doi:10.1016/j.avb.2012.09.003 · Cited by 242 · see citing works
Dir, A. L., Coskunpinar, A., Steiner, J., & Cyders, M. (2013) · Cyberpsychology, Behavior, and Social Networking · doi:10.1089/cyber.2012.0545 · Cited by 211 · see citing works
Unwanted but consensual sexting among young adults: Relations with attachment and sexual motivations
Drouin, M. A., & Tobin, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.11.001 · Cited by 183 · see citing works
Unwanted online sexual solicitation and risky sexual online behavior across the lifespan
Baumgartner, S. E., Valkenburg, P. M., & Peter, J. (2010) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2010.07.005 · Cited by 166 · see citing works
Dear Diary: Teens Reflect on Their Weekly Online Risk Experiences
Wisniewski, P., Xu, H., Rosson, M., Perkins, D., & Carroll, J. M. (2016) · International Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858317 · Cited by 141 · see citing works
Self-citationGaur, M., Faldu, K., & Sheth, A. (2021) · IEEE Internet Computing · doi:10.1109/mic.2020.3031769 · Cited by 140 · see citing works
Giménez Gualdo, A. M., Hunter, S. C., Durkin, K., Arnaiz, P., & Maquilón, J. J. (2015) · Computers & Education · doi:10.1016/j.compedu.2014.11.013 · Cited by 131 · see citing works
Toward human-centered algorithm design
Baumer, E. P. (2017) · Big Data & Society · doi:10.1177/2053951717718854 · Cited by 125 · see citing works
The Association Between Adolescent Sexting, Psychosocial Difficulties, and Risk Behavior
Ouytsel, J. V., Walrave, M., Ponnet, K. S., & Heirman, W. (2014) · The Journal of School Nursing · doi:10.1177/1059840514541964 · Cited by 114 · see citing works
de Santisteban, P., & Gámez‐Guadix, M. (2017) · The Journal of Sex Research · doi:10.1080/00224499.2017.1386763 · Cited by 107 · see citing works
Razi, A., Kim, S., Alsoubai, A., Stringhini, G., Solorio, T., De Choudhury, M., & Wisniewski, P. J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479609 · Cited by 95 · see citing works
Self-citationLet's Talk about Sext: How Adolescents Seek Support and Advice about Their Online Sexual Experiences
Razi, A., Badillo-Urquiola, K., & Wisniewski, P. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376400 · Cited by 89 · see citing works
Self-citationA Human-Centered Systematic Literature Review of Cyberbullying Detection Algorithms
Kim, S.-H., Razi, A., Stringhini, G., Wiśniewski, P., & Choudhury, M. D. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476066 · Cited by 86 · see citing works
Self-citationDon't work. Can't work? Why it's time to rethink security warnings
Krol, K., Moroz, M., & Sasse, M. A. (2012) · International Conference on Risks and Security of Internet and Systems (CRiSIS) · doi:10.1109/crisis.2012.6378951 · Cited by 81 · see citing works
A multi-platform dataset for detecting cyberbullying in social media
Van Bruwaene, D., Huang, Q., & Inkpen, D. (2020) · Language Resources and Evaluation · doi:10.1007/s10579-020-09488-3 · Cited by 74 · see citing works
Risk-taking as a Learning Process for Shaping Teen's Online Information Privacy Behaviors
Jia, H., Wisniewski, P. J., Xu, H., Rosson, M. B., & Carroll, J. M. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675287 · Cited by 70 · see citing works
Self-citationDisguising Reddit sources and the efficacy of ethical research
Reagle, J. M. (2022) · Ethics and Information Technology · doi:10.1007/s10676-022-09663-w · Cited by 70 · see citing works
Lorenzo-Dus, N., Kinzel, A., & Cristofaro, M. D. (2019) · Journal of Pragmatics · doi:10.1016/j.pragma.2019.09.010 · Cited by 61 · see citing works
Hartikainen, H., Razi, A., & Wiśniewski, P. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449116 · Cited by 60 · see citing works
Self-citationStigma in help-seeking: the case of adolescents
Shechtman, Z., Vogel, D. L., Strass, H. A., & Heath, P. J. (2016) · British Journal of Guidance and Counselling · doi:10.1080/03069885.2016.1255717 · Cited by 53 · see citing works
Razi, A., Alsoubai, A., Kim, S., Naher, N., Ali, S., Stringhini, G., Choudhury, M. D., & Wiśniewski, P. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3503569 · Cited by 53 · see citing works
Self-citationContextual Augmentation: Data Augmentation by Words with Paradigmatic RelationsPreprint
Kobayashi, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.06201 · Cited by 50 · see citing works
Ali, S., Razi, A., Kim, S., Alsoubai, A., Gracie, J., Choudhury, M. D., Wiśniewski, P., & Stringhini, G. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501969 · Cited by 43 · see citing works
Self-citationSafeCity: Understanding Diverse Forms of Sexual Harassment Personal Stories
Karlekar, S., & Bansal, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1303 · Cited by 43 · see citing works
3LC: Lightweight and Effective Traffic Compression for Distributed Machine LearningPreprint
Lim, H., Andersen, D. G., & Kaminsky, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07389 · Cited by 42 · see citing works
Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment
Chowdhury, A. G., Sawhney, R., Mathur, P., Mahata, D., & Shah, R. R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/n19-3018 · Cited by 37 · see citing works
Alsoubai, A., Song, J., Razi, A., Naher, N., De Choudhury, M., & Wisniewski, P. J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555136 · Cited by 36 · see citing works
Self-citationEnsemble Method for Sexual Predators Identification in Online Chats
Fauzi, M. A., & Bours, P. (2020) · 2020 8th International Workshop on Biometrics and Forensics (IWBF) · doi:10.1109/iwbf49977.2020.9107945 · Cited by 34 · see citing works
Hillstrom, L. C. (2018) · doi:10.5040/9798400605062 · Cited by 32 · see citing works
Broome, L. J., Izura, C., & Davies, J. (2020) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2020.104647 · Cited by 29 · see citing works
Detection of Cyber Grooming in Online Conversation
Bours, P., & Kulsrud, H. (2019) · 2019 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs47025.2019.9035090 · Cited by 27 · see citing works
Defining risk: Normative considerations
Perhac, R. M. (1996) · Human and Ecological Risk Assessment An International Journal · doi:10.1080/10807039609383615 · Cited by 24 · see citing works
You Don't Know How I Feel: Insider-Outsider Perspective Gaps in Cyberbullying Risk Detection
Kim, S.-H., Razi, A., Stringhini, G., Wiśniewski, P., & Choudhury, M. D. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18061 · Cited by 24 · see citing works
Self-citationTowards Automated Sexual Violence Report Tracking
Hassan, N., Poudel, A., Hale, J. G., Hubacek, C., Huq, K. T., Santu, S. K. K., & Ahmed, S. I. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7296 · Cited by 23 · see citing works
Hartikainen, H., Razi, A., & Wisniewski, P. (2021) · Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3462204.3481739 · Cited by 23 · see citing works
Self-citationChat Analysis Triage Tool: Differentiating contact-driven vs. fantasy-driven child sex offenders
Seigfried‐Spellar, K. C., Rogers, M., Rayz, J. T., Yang, S.-F., Misra, K., & Ringenberg, T. R. (2019) · Forensic Science International · doi:10.1016/j.forsciint.2019.02.028 · Cited by 20 · see citing works
Towards Conducting Responsible Research with Teens and Parents regarding Online Risks
Agha, Z., Chatlani, N., Razi, A., & Wiśniewski, P. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3383073 · Cited by 20 · see citing works
Self-citationMOSafely, Is that Sus? A Youth-Centric Online Risk Assessment Dashboard
Alsoubai, A., Caddle, X., Doherty, R., Koehler, A. T., Sanchez, E., Choudhury, M. D., & Wiśniewski, P. (2022) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3500868.3559710 · Cited by 19 · see citing works
Self-citationA Step Towards Detecting Online Grooming -- Identifying Adults Pretending to be Children
Ashcroft, M., Kaati, L., & Meyer, M. (2015) · 2015 European Intelligence and Security Informatics Conference · doi:10.1109/eisic.2015.41 · Cited by 18 · see citing works
Sexual Harassment Story Classification and Key Information Identification
Liu, Y., Li, Q., Liu, X., et al. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3358146 · Cited by 17 · see citing works
Dev, P., Medina, J., Agha, Z., Choudhury, M. D., Razi, A., & Wiśniewski, P. (2022) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3500868.3559469 · Cited by 17 · see citing works
Self-citationTariq, M. U., Razi, A., Badillo-Urquiola, K. A., & Wiśniewski, P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22636-7_6 · Cited by 16 · see citing works
Self-citationPrivacy Challenges for Adolescents as a Vulnerable PopulationPreprint
Razi, A., Agha, Z., Chatlani, N., & Wiśniewski, P. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3587558 · Cited by 14 · see citing works
Self-citationNot So Cute but Fuzzy: Estimating Risk of Sexual Predation in Online Conversations
Ringenberg, T. R., Misra, K., & Rayz, J. T. (2019) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2019.8914528 · Cited by 13 · see citing works
Identifying Victim Blaming Language in Discussions about Sexual Assaults on Twitter
Suvarna, A., Bhalla, G., Kumar, S., & Bhardwaj, A. (2020) · International Conference on Social Media and Society (SMSociety) · doi:10.1145/3400806.3400825 · Cited by 13 · see citing works
MOSafely: Building an Open-Source HCAI Community to Make the Internet a Safer Place for Youth
Caddle, X., Razi, A., Kim, S.-H., Ali, S., Popo, T., Stringhini, G., Choudhury, M. D., & Wiśniewski, P. (2021) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3462204.3481731 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 16
- Markus Bayer, Marc-André Kaufhold, and Christian Reuter. 2021. A survey on data augmentation for text classification. arXiv preprint arXiv:2107.03158 (2021). · arXiv
- Michael Brenner, Jeannie Suk Gersen, Michael Haley, Matthew Lin, Amil Merchant, Richard Jagdishwar Millett, Suproteem K Sarkar, and Drew Wegner. 2020. Constitutional Dimensions of Predictive Algorithms in Criminal Justice. Harv. CR-CLL Rev. 55 (2020), 267.
- Maryam Daniali and Edward Kim. 2022. Perception Over Time: Temporal Dynamics for Robust Image Understanding. arXiv preprint arXiv:2203.06254 (2022). · arXiv
- Allyson L Dir, Ayca Coskunpinar, Jennifer L Steiner, and Melissa A Cyders. 2013. Understanding differences in sexting behaviors across gender, relationship status, and sexual identity, and the role of expectancies in sexting. Cyberpsychology, Behavior, and Social Networking 16, 8 (2013), 568-574.
- Michael Hind, Sameep Mehta, Aleksandra Mojsilovic, Ravi Nair, Karthikeyan Natesan Ramamurthy, Alexandra Olteanu, and Kush R Varshney. 2018. Increasing trust in ai services through supplier's declarations of conformity. arXiv preprint arXiv:1808.07261 18 (2018), 2813-2869. · arXiv
- Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three approaches to qualitative content analysis. Qualitative health research 15, 9 (2005), 1277-1288.
- Giacomo Inches and Fabio Crestani. 2012. Overview of the International Sexual Predator Identification Competition at PAN-2012.. In CLEF (Online working notes/labs/workshop), Vol. 30.
- Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 (2019). · arXiv
- James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: LIWC 2001. Mahway: Lawrence Erlbaum Associates 71, 2001 (2001), 2001.
- Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe Global Vectors for Word Representation. , 1532-1543 pages. http://www.aclweb.org/anthology/D14-1162
- Gonzalo Ramos, Jina Suh, Soroush Ghorashi, Christopher Meek, Richard Banks, Saleema Amershi, Rebecca Fiebrink, Alison Smith-Renner, and Gagan Bansal. 2019. Emerging Perspectives in Human-Centered Machine Learning. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1-8.
- Dominik Sacha, Michael Sedlmair, Leishi Zhang, John Aldo Lee, Daniel Weiskopf, Stephen North, and Daniel Keim. 2016. Human-centered machine learning through interactive visualization. ESANN, Bruges, Belgium, 641-646. https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016-166.pdf
- Donald Sharpe. 2015. Chi-square test is statistically significant: Now what? Practical Assessment, Research, and Evaluation 20, 1 (2015), 8.
- Svetlana Smirnova, Sonia Livingstone, and Mariya Stoilova. 2021. Understanding of user needs and problems: a rapid evidence review of age assurance and parental controls. (2021).
- Jessica Vitak and Jinyoung Kim. 2014. " You can't block people offline" examining how facebook's affordances shape the disclosure process. In In Proc 17th ACM CSCW. 461-474.
- Michele L Ybarra and Kimberly J Mitchell. 2008. How risky are social networking sites? A comparison of places online where youth sexual solicitation and harassment occurs. Pediatrics 121, 2 (2008).
Cited by
Sorted by citations
Understanding Teen Overreliance on AI Companion Chatbots Through Self-Reported Reddit Narratives
Namvarpour, M. (., Brofsky, B., Medina, J. Y., Akter, M., & Razi, A. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790597 · Cited by 24 · see citing works
Self-citationAgha, Z., Park, J., Wan, R., Ali, N. S., Wang, Y., Difranzo, D., Badillo-Urquiola, K., & Wisniewski, P. J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642313 · Cited by 13 · see citing works
Zhang, X., Akter, M., An, H., Liu, M., Zhang, Q., Huang, L., Cho, J.-H., J. Wisniewski, P., & Lee, S. W. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790949
Self-citation
- Authors
- Razi, A., Alsoubai, A., Kim, S., Ali, S., Stringhini, G., De Choudhury, M., & Wisniewski, P. J.
- Affiliations
- Drexel University (US)
- Vanderbilt University (US)
- Georgia Institute of Technology (US)
- Boston University (US)
- Georgia Institute of Technology, Atlanta, GA, USA
- Vanderbilt University, Nashville, TN, USA
- ORCID
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-04-14
- Volume
- 7
- Issue
- CSCW1
- Pages
- 1-29
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4365999004
- Cited by
- 52
- References
- 57
- Citations by year
- 2026: 10 · 2025: 14 · 2024: 23 · 2023: 4 · 2022: 1
- FWCI
- 4.51
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Funders
- William T. Grant Foundation (#187941)
- National Science Foundation (IIP-1827700 and IIS-1844881)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Health Sciences › Health Professions › General Health Professions › Adolescent Sexual and Reproductive Health
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- adolescents, sexual safety, online safety, human-centered machine learning, convolutional neural networks, random forest, LIWC, contextual features
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
