Proceedings of the First ACL Workshop on Ethics in Natural Language Processing
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
We present results on a quantitative analysis of publications in the NLP domain on collecting, publishing and availability of research data.We find that a wide range of publications rely on data crawled from the web, but few give details on how potentially sensitive data was treated.Additionally, we find that while links to repositories of data are given, they often do not work even a short time after publication.We put together several suggestions on how to improve this situation based on publications from the NLP domain, but also other research areas.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dilemmas in a general theory of planning
Rittel, H. W. J., & Webber, M. M. (1973) · Policy Sciences · doi:10.1007/bf01405730 · Cited by 16,338
Crenshaw, K. (2018) · Living with Contradictions · doi:10.4324/9780429499142-5 · Cited by 7,232
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Twitter mood predicts the stock market
Bollen, J., Mao, H., & Zeng, X. (2011) · Journal of Computational Science · doi:10.1016/j.jocs.2010.12.007 · Cited by 5,206
Experimental evidence of massive-scale emotional contagion through social networks
Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1320040111 · Cited by 3,687
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Private traits and attributes are predictable from digital records of human behavior
Kosinski, M., Stillwell, D., & Graepel, T. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1218772110 · Cited by 2,829
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372
SWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
"From Jerusalem to Jericho": A study of situational and dispositional variables in helping behavior.
Darley, J. M., & Batson, C. D. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0034449 · Cited by 1,665
Rethinking validity and reliability in content analysis
Potter, W. J., & Levine‐Donnerstein, D. (1999) · Journal of Applied Communication Research · doi:10.1080/00909889909365539 · Cited by 1,662
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305
Savova, G. K., Kipper-Schuler, K. C., Hurdle, J. F., & Meystre, S. M. (2008) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638592 · Cited by 895
MOOR, J. H. (1985) · Metaphilosophy · doi:10.1111/j.1467-9973.1985.tb00173.x · Cited by 863
Gender differences in language use: An analysis of 14,000 text samples
Newman, M. L., Groom, C. J., Handelman, L. D., & Pennebaker, J. W. (2008) · Discourse Processes · doi:10.1080/01638530802073712 · Cited by 788
The ethics of big data: current and foreseeable issues in biomedical contexts
Mittelstadt, B. D., & Floridi, L. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-015-9652-2 · Cited by 714
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694
Classifying latent user attributes in twitter
Rao, D., Yarowsky, D., Shreevats, A., & Gupta, M. (2010) · Proceedings of the 2nd international workshop on Search and mining user-generated contents · doi:10.1145/1871985.1871993 · Cited by 655
Automatically Categorizing Written Texts by Author Gender
Koppel, M. (2002) · Literary and Linguistic Computing · doi:10.1093/llc/17.4.401 · Cited by 623
Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567
A framework for evaluation and use of automated scoring
Williamson, D. M., Xi, X., & Breyer, F. J. (2012) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.2011.00223.x · Cited by 430
Classifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409
Rickford, J. R., & King, S. (2016) · Language · doi:10.1353/lan.2016.0078 · Cited by 374
Artificial Morality: Top-down, Bottom-up, and Hybrid Approaches
Allen, C., Smit, I., & Wallach, W. (2005) · Ethics and Information Technology · doi:10.1007/s10676-006-0004-4 · Cited by 346
Althoff, T., Clark, K., & Leskovec, J. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00111 · Cited by 322
Decision Theory for Discrimination-Aware Classification
Kamiran, F., Karim, A., & Zhang, X. (2012) · 2012 IEEE 12th International Conference on Data Mining · doi:10.1109/icdm.2012.45 · Cited by 300
Chapman, W. W., Nadkarni, P. M., Hirschman, L., D'Avolio, L. W., Savova, G. K., & Uzuner, O. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000465 · Cited by 296
A multidisciplinary survey on discrimination analysis
Romei, A., & Ruggieri, S. (2014) · The Knowledge Engineering Review · doi:10.1017/s0269888913000039 · Cited by 288
k-NN as an implementation of situation testing for discrimination discovery and prevention
Luong, B. T., Ruggieri, S., & Turini, F. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020488 · Cited by 263
On the (im)possibility of fairness
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · arXiv · doi:10.48550/arxiv.1609.07236 · Cited by 247
Mining the Blogosphere: Age, gender and the varieties of self-expression
Argamon, S., Koppel, M., Pennebaker, J. W., & Schler, J. (2007) · First Monday · doi:10.5210/fm.v12i9.2003 · Cited by 210
Goldwater, S., Jurafsky, D., & Manning, C. D. (2010) · Speech Communication · doi:10.1016/j.specom.2009.10.001 · Cited by 166
Automated essay scoring: Psychometric guidelines and practices
Ramineni, C., & Williamson, D. M. (2013) · Assessing Writing · doi:10.1016/j.asw.2012.10.004 · Cited by 127
Big Data, Big Problems: Emerging Issues in the Ethics of Data Science and Journalism
Fairfield, J., & Shtein, H. (2014) · Journal of Mass Media Ethics · doi:10.1080/08900523.2014.863126 · Cited by 89
Padrez, K. A., Ungar, L., Schwartz, H. A., Smith, R. J., Hill, S., Antanavicius, T., Brown, D. M., Crutchley, P., Asch, D. A., & Merchant, R. M. (2016) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004489 · Cited by 75
Cited by
HONEST: Measuring Hurtful Sentence Completion in Language Models
Nozza, D., Bianchi, F., & Hovy, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.191 · Cited by 95
Uneven success: automatic speech recognition and ethnicity-related dialects
Wassink, A. B., Gansen, C., & Bartholomew, I. (2022) · Speech Communication · doi:10.1016/j.specom.2022.03.009 · Cited by 77
‘Just What do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP
Rogers, A., Baldwin, T., & Leins, K. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.414 · Cited by 53
Onitilo, A. A., Shour, A. R., Puthoff, D. S., Tanimu, Y., Joseph, A., & Sheehan, M. T. (2023) · PLOS ONE · doi:10.1371/journal.pone.0272545 · Cited by 21
Metadata
- Published
- 2017
- OpenAlex
- W2774408488
- Cited by
- 39
- References
- 39
- Type
- paratext
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.