The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
‘Just What do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP
Open accessAbstract
A key part of the NLP ethics movement is responsible use of data, but exactly what that means or how it can be best achieved remain unclear. This position paper discusses the core legal and ethical principles for collection and sharing of textual data, and the tensions between them. We propose a potential checklist for responsible data (re-)use that could both standardise the peer review of conference submissions, as well as enable a more in-depth view of published research across the community. Our proposal aims to contribute to the development of a consistent standard for data (re-)use, embraced across NLP conferences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some checklists ask authors to indicate the sections of their papers in which certain issues are discussed, such as reproducibility efforts, compensation to human participants, and adherence to data licenses.
“In a similar vein, some checklists 6 ask authors to indicate the sections of their papers in which certain issues are discussed, e.g., reproducibility efforts, compensation to human participants, adherence to data licenses (Dodge et al., 2019; Beygelzimer et al., 2021a; Rogers et al., 2021).”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Rogers et al., 2021
GDPR regulates the use of personal data of EU subjects anywhere in the world.
“which regulates the use of personal data of EU subjects anywhere in the world (Rogers et al., 2021)”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Rogers et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 2×
“…of ethics by industry, government, and academia remain highly contextual, and have “substantive divergence in relation to how these principles are interpreted, why they are deemed important, what issue, domain or actors they pertain to, and how they should be implemented” (Jobin et al., 2019).”
BackgroundDuan, Y., Edwards, J. S., & Dwivedi, Y. K. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.01.021 · Cited by 2,628 · cited 1×
“The AI community has already started to work on the latter (Mantelero, 2018; Duan et al., 2019; Sielemann et al., 2020; Asilomar, 2017).”
BackgroundUnique in the Crowd: The privacy bounds of human mobility
de Montjoye, Y.-A., Hidalgo, C. A., Verleysen, M., & Blondel, V. D. (2013) · Scientific Reports · doi:10.1038/srep01376 · Cited by 1,645 · cited 1×
“Anonymisation techniques are an active research area, but at present “even coarse datasets provide little anonymity” (de Montjoye et al., 2013).”
BackgroundLanguage (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050 · cited 2×
“, 2019; Xiang and Raji, 2019) and “bias” (Blodgett et al., 2020).”
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548
On the Feasibility of Internet-Scale Author Identification
Narayanan, A., Paskov, H., Gong, N. Z., Bethencourt, J., Stefanov, E., Shin, E. C. R., & Song, D. (2012) · 2012 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2012.46 · Cited by 261
Digital social norm enforcement: Online firestorms in social media
Rost, K., Stahel, L., & Frey, B. S. (2016) · PLOS ONE · doi:10.1371/journal.pone.0155923 · Cited by 190
Changing the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51
Self-citationBandy, J., & Vincent, N. (2021) · arXiv · doi:10.48550/arxiv.2105.05241 · Cited by 36
Flagged first, then by citations
Research works printed but not linked: 50
- Mark Alfano, Dirk Hovy, Margaret Mitchell, and Michael Strube, editors. 2018. Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing. Association for Computational Linguistics, New Orleans, Louisiana, USA. · DOI
- Derek C. Angus, Brian M. Alexander, Scott Berry, Meredith Buxton, Roger Lewis, Melissa Paoloni, Steven A. R. Webb, Steven Arnold, Anna Barker, Donald A. Berry, Marc J. M. Bonten, Mary Brophy, Christopher Butler, Timothy F. Cloughesy, Lennie P. G. Derde, Laura J. Esserman, Ryan Ferguson, Louis Fiore, Sarah C. Gaffey, J. Michael Gaziano, Kathy Giusti, Herman Goossens, Stephane Heritier, Bradley Hyman, Michael Krams, Kay Larholt, Lisa M. LaVange, Philip Lavori, Andrew W. Lo, Alex John London, Victoria Manax, Colin McArthur, Genevieve O&, apos, Neill, Giovanni Parmigiani, Jane Perlmutter, Elizabeth A. Petzold, Craig Ritchie, Kathryn M. Rowan, Christopher W. Seymour, Nathan I. Shapiro, Diane M. Simeone, Bradley Smith, Bradley Spellberg, Ariel Dora Stern, Lorenzo Trippa, Mark Trusheim, Kert Viele, Patrick Y. Wen, and Janet Woodcock. 2019. Adaptive platform trials: Definition, design, conduct and reporting considerations. Nature Reviews Drug Discovery, 18(10):797-808.
- Carolyn Ashurst, Markus Anderljung, Karina Prunkl, Jan Leike, Yarin Gal, Toby Shelvane, and Allan Dafoe. 2020. A Guide to Writing the NeurIPS Impact Statement. Asilomar. 2017. Asilomar AI Principles.
- Priyam Basu, Tiasa Singha Roy, Rakshit Naidu, Zumrut Muftuoglu, Sahib Singh, and Fatemehsadat Mireshghallah. 2021. Benchmarking Differential Privacy and Federated Learning for BERT Models. arXiv:2106.13973 [cs]. · arXiv
- Justin Brickell and Vitaly Shmatikov. 2008. The cost of privacy: Destruction of data-mining utility in anonymized data publishing. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, pages 70-78, New York, NY, USA. Association for Computing Machinery. · DOI
- Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2020. Extracting Training Data from Large Language Models. arXiv:2012.07805 [cs]. · arXiv
- Fred H. Cate and Beth E. Cate. 2012. The Supreme Court and information privacy. International Data Privacy Law, 2(4):255-267. · DOI
- K. Bretonnel Cohen, Jingbo Xia, Pierre Zweigenbaum, Tiffany Callahan, Orin Hargraves, Foster Goss, Nancy Ide, Aurelie Neveol, Cyril Grouin, and Lawrence Hunter. 2018. Three Dimensions of Reproducibility in Natural Language Processing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
- Matt Crane. 2018. Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results. Transactions of the Association for Computational Linguistics, 6:241- 252. · DOI
- Chris Culnane and Kobi Leins. 2019. Misconceptions in privacy protection and regulation. Law in Context. · DOI
- Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show Your Work: Improved Reporting of Experimental Results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2185-2194, Hong Kong, China. Association for Computational Linguistics. · DOI · Paper page
- Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. 2021. Documenting the English Colossal Clean Crawled Corpus. arXiv:2104.08758 [cs]. · arXiv
- Jesse Dodge and Noah A. Smith. 2020. Guest Post: Reproducibility at EMNLP 2020.
- Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating Noise to Sensitivity in Private Data Analysis. In Theory of Cryptography, Lecture Notes in Computer Science, pages 265-284, Berlin, Heidelberg. Springer. · DOI · Paper page
- Perrin Aikens Davis, et al. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961 [cs]. · arXiv
- Casey Fiesler and Nicholas Proferes. 2018. "Participant" Perceptions of Twitter Research Ethics. Social Media + Society, 4(1):2056305118763366. · DOI
- Alan Fleischman, Carol Levine, Lisa Eckenwiler, Christine Grady, Dale E. Hammerschmidt, and Jeremy Sugarman. 2011. Dealing With the Long- Term Social Implications of Research. The American journal of bioethics : AJOB, 11(5):5-9. · DOI
- Catherine Flick. 2016. Informed consent and the Facebook emotional manipulation study. Research Ethics, 12(1):14-28. · DOI
- Don Gotterbarn, Bo Brinkman, Catherine Flick, Michael S. Kirkpatrick, Keith Miller, Kate Varansky, and Marty J. Wolf. 2018. ACM Code of Ethics and Professional Conduct.
- Ben Green. 2019. "Good" isn't good enough. In AI for Social Good Workshop at NeurIPS 2019.
- Benjamin Greze. 2019. The extra-territorial enforcement of the GDPR: A genuine issue and the quest for alternatives. International Data Privacy Law, 9(2):109-128. · DOI
- Hazar Harmouch and Felix Naumann. 2017. Cardinality estimation: An experimental survey. Proceedings of the VLDB Endowment, 11(4):499- 512. · DOI
- Brent Hecht, Lauren Wilcox, Jeffrey P. Bigham, Johannes Schöning, Ehsan Hoque, Jason Ernst, Yonatan Bisk, Luigi de Russis, Lana Yarosh, Bushra Anjum, Danish Contractor, and Wu Cathy. 2018. It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process. ACM Future of Computing Blog.
- Dirk Hovy, Shannon Spruit, Margaret Mitchell, Emily M. Bender, Michael Strube, and Hanna Wallach, editors. 2017. Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. Association for Computational Linguistics, Valencia, Spain. · DOI
- Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. 2014. Experimental evidence of massive- scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111(24):8788-8790. · DOI · Paper page
- Yarik Kryvoi and Shaun Matos. 2021. Non- Retroactivity as a General Principle of Law. Utrecht Law Review, 17(1):46-58. · DOI
- Alessandro Mantelero. 2018. AI and Big Data: A blueprint for a human rights, social and ethical impact assessment. Computer Law & Security Review, 34(4):754-772. · DOI
- Jacob Metcalf, Emanuel Moss, and danah boyd. 2019. Owning Ethics: Corporate Logics, Silicon Valley, and the Institutionalization of Ethics. Social Research: An International Quarterly, 86(2):449- 476.
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).
- Timo Minssen, Neethu Rajam, and Marcel Bogers. 2020. Clinical trial data transparency and GDPR compliance: Implications for data sharing and open innovation. Science and Public Policy, 47(5):616- 626. · DOI
- Brent Mittelstadt. 2019. The Ethics of Biomedical 'Big Data' Analytics. Philosophy & Technology, 32(1):17-21. · DOI
- Maximilian Mozes and Bennett Kleinberg. 2021. No Intruder, no Validity: Evaluation Criteria for Privacy-Preserving Text Anonymization. arXiv:2103.09263 [cs. · arXiv
- Deirdre K. Mulligan, Joshua A. Kroll, Nitin Kohli, and Richmond Y. Wong. 2019. This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology. Proceedings of the ACM on Human- Computer Interaction, 3(CSCW):119:1-119:36. · DOI · Paper page
- Dean Neu, Greg Saxton, Abu Rahaman, and Jeffery Everett. 2019. Twitter and social accountability: Reactions to the Panama Papers. Critical Perspectives on Accounting, 61:38-53. · DOI
- Jonathan A. Obar and Anne Oeldorf-Hirsch. 2020. The biggest lie on the Internet: Ignoring the privacy policies and terms of service policies of social networking services. Information, Communication & Society, 23(1):128-147. · DOI · Paper page
- Official Journal of the European Communities. 2000. Charter of Fundamental Rights of the European Union. Official Journal of the European Communities.
- Stephanie M. Ortiz. 2020. Trolling as a Collective Form of Harassment: An Inductive Study of How Online Users Understand Trolling. Social Media + Society, 6(2):2056305120928512. · DOI
- Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543. · DOI · Paper page
- Edward Raff. 2019. A Step Toward Quantifying Independently Reproducible Machine Learning Research. In NeurIPS.
- Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2019. Calls to Action on Social Media: Potential for Censorship and Social Impact. In EMNLP-IJCNLP 2019 Second Workshop on Natural Language Processing for Internet Freedom.
- Sebastin Santy, Anku Rani, and Monojit Choudhury. 2021. Use of Formal Ethical Reviews in NLP Literature: Historical Trends and Current Practices. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4704-4710, Online. Association for Computational Linguistics. · DOI
- Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758-3769, Online. Association for Computational Linguistics. · DOI
- Prasha Shrestha, Sebastian Sierra, Fabio González, Manuel Montes, Paolo Rosso, and Thamar Solorio. 2017. Convolutional neural networks for authorship attribution of short texts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 669-674, Valencia, Spain. Association for Computational Linguistics.
- Katharina Sielemann, Alenka Hafner, and Boas Pucker. 2020. The reuse of public datasets in the life sciences: Potential risks and rewards. PeerJ, 8(e9954). · DOI
- Jason Tashea. 2019. France bans publishing of judicial analytics and prompts criminal penalty. ABA Journal.
- Rachael Tatman, Jake VanderPlas, and Sohier Dane. 2018. A Practical Taxonomy of Reproducibility for Machine Learning Research. In Reproducibility in Machine Learning Workshop (Co-Located with ICML).
- Silvio Waisbord. 2020. Mob Censorship: Online Harassment of US Journalists in Times of Digital Hate and Populism. Digital Journalism, 8(8):1030- 1046. · DOI
- Martijn Wieling, Josine Rawee, and Gertjan van Noord. 2018. Reproducibility in Computational Linguistics: Are We Willing to Share? Computational Linguistics, 44(4):641-649. · DOI
- Alice Xiang and Inioluwa Deborah Raji. 2019. On the Legal Compatibility of Fairness Definitions. In Workshop on Human-Centric Machine Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).
- Hui Zang and Jean Bolot. 2011. Anonymization of location data does not work: A large-scale measurement study. In Proceedings of the 17th Annual International Conference on Mobile Computing and Networking, MobiCom '11, pages 145-156, New York, NY, USA. Association for Computing Machinery. · DOI
Cited by
Sorted by citations
Bauer, E., Greisel, M., Kuznetsov, I., Berndt, M., Kollar, I., Dresel, M., Fischer, M. R., & Fischer, F. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13336 · Cited by 130
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Jernite, Y., Nguyen, H., Biderman, S., Rogers, A., Masoud, M., Danchev, V., Tan, S., Luccioni, A. S., Subramani, N., Johnson, I., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534637 · Cited by 75
“PREAMBLE BigScience is an open research collaboration involving over 1000 participants from 60 countries, focusing its collaborative research efforts in the study and development of natural language processing systems (hereinafter NLP).”
BackgroundInfluentialSelf-citationTen simple rules for building and maintaining a responsible data science workflow
Stoudt, S., Jernite, Y., Marshall, B., Marwick, B., Sharan, M., Whitaker, K., & Danchev, V. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1012232 · Cited by 11
What Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6
Self-citation
Metadata
- Authors
- Rogers, A., Baldwin, T., & Leins, K.
- Affiliations
- University of Copenhagen (DK)
- Microsoft (United States) (US)
- King's College London (GB)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2021
- Published
- 2021
- Pages
- 4821-4833
- OpenAlex
- W3201197540
- Cited by
- 53
- References
- 12
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 8 · 2023: 22 · 2022: 13 · 2021: 1
- FWCI
- 4.14
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, public-domain)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- responsible data use, data reuse, data sharing, research reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
