CoChat
Sign Up
Cited by 9scientometrics and bibliometrics researchInterdisciplinary Research and CollaborationPsychology Research and Bibliometrics

We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields

Open access
Wahle, Jan, Ruas, Terry, Abdalla, Mohamed, Gipp, Bela, Mohammad, Saif·2023·Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2023.emnlp-main.797
Open this paper in CoChat — free accountOpen PDF

Abstract

Natural Language Processing (NLP) is poised to substantially influence the world.However, significant progress comes hand-in-hand with substantial risks.Addressing them requires broad engagement with various fields of study.Yet, little empirical work examines the state of such engagement (past or current).In this paper, we quantify the degree of influence between 23 fields of study and NLP (on each other).We analyzed ∼77k NLP papers, ∼3.1m citations from NLP papers to other papers, and ∼1.8m citations from other papers to NLP papers.We show that, unlike most fields, the cross-field engagement of NLP, measured by our proposed Citation Field Diversity Index (CFDI), has declined from 0.58 in 1980 to 0.31 in 2022 (an all-time low).In addition, we find that NLP has grown more insular-citing increasingly more NLP papers and having fewer papers that act as bridges between fields.NLP citations are dominated by computer science; Less than 8% of NLP citations are to linguistics, and less than 3% are to math and psychology.These findings underscore NLP's urgent need to reflect on its engagement with various fields.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.emnlp-main.797

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. Mohamed Abdalla, Jan Philip Wahle, Terry Lima Ruas, Aurélie Névéol, Fanny Ducel, Saif Mohammad, and Karen Fort. 2023. The elephant in the room: Ana- lyzing the presence of big tech in natural language processing research. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13141- 13160, Toronto, Canada. Association for Computa- tional Linguistics. · DOI
  2. Ashton Anderson, Dan Jurafsky, and Daniel A. Mc- Farland. 2012. Towards a computational history of the ACL: 1980-2008. In Proceedings of the ACL- 2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 13-21, Jeju Island, Korea. Asso- ciation for Computational Linguistics.
  3. Mark Bear, Barry Connors, and Michael A Paradiso. 2020. Neuroscience: exploring the brain, enhanced edition: exploring the brain. Jones & Bartlett Learn- ing.
  4. Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2023. On the dangers of stochastic parrots: Can language models be too big? volume abs/2305.18554. · arXiv
  5. Niels Bohr. 1913. I. on the constitution of atoms and molecules. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 26(151):1-25.
  6. Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language mod- els in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Nat- ural Language Processing and Computational Nat- ural Language Learning (EMNLP-CoNLL), pages 858-867, Prague, Czech Republic. Association for Computational Linguistics.
  7. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in com- mercial gender classification. In Conference on fair- ness, accountability and transparency, pages 77-91.
  8. PMLR.
  9. Miriam Louise Carnot, Jorge Bernardino, Nuno Laran- jeiro, and Hugo Gonçalo Oliveira. 2020. Applying text analytics for studying research trends in depend- ability. Entropy, 22(11):1303.
  10. Mengjie Cheng, Daniel Scott Smith, Xiang Ren, Hancheng Cao, Sanne Smith, and Daniel A McFar- land. 2023. How new ideas diffuse in science. Amer- ican Sociological Review, 88(3):522-561.
  11. Jin M Choi. 1988. Citation analysis of intra-and inter- disciplinary communication patterns of anthropology in the usa. Behavioral & Social Sciences Librarian, 6(3-4):65-84.
  12. James A Evans. 2008. Electronic publication and the narrowing of science and scholarship. science, 321(5887):395-399.
  13. Sireesh Gururaja, Amanda Bertsch, Clara Na, David Gray Widder, and Emma Strubell. 2023. To build our future, we must know our past: Contextual- izing paradigm shifts in natural language processing. ArXiv preprint, abs/2310.07715. · arXiv
  14. Bengt Hansson. 1999. Interdisciplinarity: For what purpose? Policy Sciences, 32(4):339-343.
  15. Dan Jurafsky. 2016. Ketchup, interdisciplinarity, and the spread of innovation in speech and language pro- cessing. In Interspeech 2016, 17th Annual Confer- ence of the International Speech Communication As- sociation, San Francisco, CA, USA, September 8-12, 2016, page 3111. ISCA.
  16. Loet Leydesdorff and Ismael Rafols. 2011. Indicators of the interdisciplinarity of journals: Diversity, central- ity, and citations. Journal of Informetrics, 5(1):87- 100.
  17. Loet Leydesdorff, Caroline S Wagner, and Lutz Born- mann. 2019. Interdisciplinarity as diversity in cita- tion patterns among journals: Rao-Stirling diversity, relative variety, and the gini coefficient. Journal of Informetrics, 13(1):255-269.
  18. Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Ja- son Phang, et al. 2022. What do nlp researchers believe? results of the nlp community metasurvey. ArXiv preprint, abs/2208.12852. · arXiv
  19. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, account- ability, and transparency, pages 220-229.
  20. Saif Mohammad. 2022. Ethics sheets for AI tasks. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 8368-8379, Dublin, Ireland. As- sociation for Computational Linguistics. · DOI
  21. Saif M. Mohammad. 2020a. Examining citations of nat- ural language processing literature. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5199-5209, On- line. Association for Computational Linguistics. · DOI
  22. Saif M. Mohammad. 2020b. Gender gap in natural lan- guage processing research: Disparities in authorship and citations. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7860-7870, Online. Association for Computational Linguistics. · DOI
  23. Mathias Wullum Nielsen and Jens Peter Andersen. 2021. Global citation inequality is on the rise. Proceedings of the National Academy of Sciences, 118(7):e2012208118.
  24. Engineering Committee on Science, Public Policy, Institute of Medicine (US), National Academies (US). Committee on Facilitating Interdisciplinary Re- search, National Academy of Engineering (US), and National Academy of Sciences (US). 2004. Facilitat- ing interdisciplinary research. National Academies Press.
  25. Raj Kumar Pan, Sitabhra Sinha, Kimmo Kaski, and Jari Saramäki. 2012. The evolution of interdisciplinarity in physics research. Scientific reports, 2(1):551.
  26. David W Pearce and R Kerry Turner. 1989. Economics of natural resources and the environment. Johns Hopkins University Press.
  27. Max Planck. 1901. On the law of distribution of en- ergy in the normal spectrum. Annalen der physik, 4(553):1.
  28. Alan Porter and Ismael Rafols. 2009. Is science be- coming more interdisciplinary? measuring and map- ping six research fields over time. Scientometrics, 81(3):719-745.
  29. Alan L Porter, David J Roessner, and Anne E Heberger. 2008. How interdisciplinary is a given body of re- search? Research evaluation, 17(4):273-282.
  30. Sandra Postel, Kamaljit Bawa, Les Kaufman, Charles H Peterson, Stephen Carpenter, David Tillman, Paul Dayton, Susan Alexander, Kalen Lagerquist, Larry Goulder, et al. 2012. Nature's services: Societal dependence on natural ecosystems. Island Press.
  31. Susie Xi Rao, Peter H Egger, and Ce Zhang. 2023. Hierarchical classification of research fields in the" web of science" using deep learning. ArXiv preprint, abs/2302.00390. · arXiv
  32. Terry Ruas, Jan Philip Wahle, Lennart Küll, Saif M Mohammad, and Bela Gipp. 2022. CS-Insights: A System for Analyzing Computer Science Research. ArXiv preprint, abs/2210.06878. · arXiv
  33. Mukund Rungta, Janvijay Singh, Saif M. Mohammad, and Diyi Yang. 2022. Geographic citation gaps in NLP research. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1371-1383, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
  34. Eemeli Saari. 2019. Trend analysis in ai research over time using nlp techniques. B.S. thesis.
  35. Angelo A Salatino, Francesco Osborne, Thiviyan Thana- palasingam, and Enrico Motta. 2019. The CSO clas- sifier: Ontology-driven detection of research topics in scholarly articles. In Digital Libraries for Open Knowledge: 23rd International Conference on The- ory and Practice of Digital Libraries, TPDL 2019, Oslo, Norway, September 9-12, 2019, Proceedings 23, pages 296-311. Springer.
  36. Angelo A. Salatino, Thiviyan Thanapalasingam, Andrea Mannocci, Aliaksandr Birukou, Francesco Osborne, and Enrico Motta. 2020. The Computer Science On- tology: A Comprehensive Automatically-Generated Taxonomy of Research Areas. Data Intelligence, 2(3):379-416. · DOI
  37. W Mark Saltzman. 2009. Biomedical engineering: bridging medicine and technology. Cambridge Uni- versity Press.
  38. Natalie Schluter. 2018. The glass ceiling in NLP. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2793-2798, Brussels, Belgium. Association for Com- putational Linguistics. · DOI
  39. Janvijay Singh, Mukund Rungta, Diyi Yang, and Saif Mohammad. 2023. Forgotten knowledge: Examin- ing the citational amnesia in NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6192-6208, Toronto, Canada. Association for Computational Linguistics. · DOI
  40. Alexandre Truc, Olivier Santerre, Yves Gingras, and François Claveau. 2020. The interdisciplinarity of economics. Available at SSRN 3669335.
  41. Marco Valenzuela, Vu Ha, and Oren Etzioni. 2015. Iden- tifying meaningful citations. In Proc. of AAAI.
  42. Adam Vogel and Dan Jurafsky. 2012. He said, she said: Gender in the ACL Anthology. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 33-41, Jeju Island, Korea. Association for Computational Linguistics.
  43. Jan Philip Wahle, Terry Ruas, Saif Mohammad, and Bela Gipp. 2022. D3: A massive dataset of schol- arly metadata for analyzing the state of computer sci- ence research. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 2642-2651, Marseille, France. European Language Resources Association.
  44. Jan Philip Wahle, Terry Ruas, Saif M Mohammad, Nor- man Meuschke, and Bela Gipp. 2023. AI Usage Cards: Responsibly Reporting AI-generated Content. ArXiv preprint, abs/2303.03886. · arXiv
  45. Jian Wang, Bart Thijs, and Wolfgang Glänzel. 2015. Interdisciplinarity and impact: Distinct effects of variety, balance, and disparity. PloS one, 10(5):e0127298.
  46. Xiaodan Zhu, Peter Turney, Daniel Lemire, and André Vellino. 2015. Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology, 66(2):408-427.

Cited by

Sorted by citations

Authors
Wahle, J., Ruas, T., Abdalla, M., Gipp, B., & Mohammad, S.
Affiliations
National Research Council Canada (CA)
ORCID
Venue
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2023
Pages
12896-12913
DOI
10.18653/v1/2023.emnlp-main.797
OpenAlex
W4389523824
Cited by
9
References
4
Citations by year
2026: 1 · 2025: 6 · 2024: 1 · 2023: 1
FWCI
9.29
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • NPARC (submittedVersion, cc-by)
  • GRO (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Niedersächsische Ministerium für Wissenschaft und Kultur
  • Deutscher Akademischer Austauschdienst
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Decision Sciences › Information Systems and Management › Interdisciplinary Research and Collaboration
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
natural language processing, citation analysis, interdisciplinarity, citation patterns, disciplinary engagement, interdisciplinary research, bibliometric analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review