Word association norms, mutual information, and lexicography
Open accessAbstract
The term word assaciation is used in a very particular sense in the psycholinguistic literature.(Generally speaking, subjects respond quicker than normal to the word "nurse" if it follows a highly associated word such as "doctor.")We wilt extend the term to provide the basis for a statistical description of a variety of interesting linguistic phenomena, ranging from semantic relations of the doctor/nurse type (content word/content word) to lexico-syntactic co-occurrence constraints between verbs and prepositions (content word/function word).This paper will propose a new objective measure based on the information theoretic notion of mutual information, for estimating word association norms from computer readable corpora.(The standard method of obtaining word association norms, testing a few thousand subjects on a few hundred words, is both costly and unreliable.)The , proposed measure, the association ratio, estimates word association norms directly from computer readable corpora, waki,~g it possible to estimate norms for tens of thousands of words. I. Meaning and AssociationIt is common practice in linguistics to classify words not only on the basis of their meanings but also on the basis of their co-occurrence with other words.Running through the whole Firthian tradition, for example, is the theme that "You shall know a word by the company it keeps" (Firth, 1957)."On the one hand, bank ¢o.occors with words and expression such u money, nmu.loan, account, ~m.c~z~c.o~.ctal, manager, robbery, vaults, wortln# in a, lu action, Fb~Nadonal. of F.ngland, and so forth.On the other hand, we find bank m-occorring with r~r.~bn, boa:.am (end of course West and Sou~, which have tcqu/red special meanings of their own), on top of the, and of the Rhine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The form of contrastive perplexity is equivalent to conditional pointwise mutual information.
“Finally, we observe that the form of contrastive perplexity is equivalent to conditional pointwise mutual information (Church and Hanks, 1989).”
From LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2024) · cites this paper as Church and Hanks, 1989
Section 4, “An Information Theoretic Measure,” introduces mutual information for pairs of words, and the following section describes its comparison of joint and independent probabilities. Our copy does not discuss contrastive perplexity or conditional pointwise mutual information, so it cannot verify the claimed equivalence. (LLM, conf=0.73)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 10
- Church, K., (1988), "A Stochastic Pans Program and Noun Phrase Parser for Unrestricted Text," Second Conference on AppU~ Natural Language Processing, Austin, Texas.
- Fano, R., (1961), Tranamlx~n of Information, MIT Press, Cambridge, Massechusens.
- Firth, J., (1957), "A Synopsis of Linguistic Theory 1930-1955" in Smdiea in l.AnguLvd¢ Analysis, Philological Society, Oxford; reprinted in Palmer, F., (ed. 1968), Selected Papers Of J.R. Firth, Longman, Httlow.
- Good, I. J., (1953), The Population Frequemctea of Species and the F..tttnmrlan of Population Parametera, Biomelxika, Vol. 40, pp, 237-264.
- Hindle, D., (1983a), "Deterministic Parsing of Syntactic Non- fluancks," ACL Proceedings.
- Hornby, A., (1948), The Advanced Learner's D/cn'onary, Oxford Univenity Press.
- Kahaa, $., Pavlidis, T., and Baird, H., (1987) "On the Recognition of Printed Characters of any Font or She," IEEE Transections PAMI, pp. 274-287.
- Meyer, D., Schvaneveldt, R.. and Ruddy, M., (1975), "Loci of Contextual Effects on Visual Word-Reoognition," in Rabbin, P., and Domic, S., (ads.), Attention and Performance V, Academic Press, London, New York, San PrantAwo.
- Pakn-mo, D,, and Jenkins, J., (1964) "Word Asr,~:iation Norms," University of Minnesota Press, Minn~po~.
- Sinclair, J., (lgSTo), "The Nature of the Evidence," in Sinclair, J. (ed.), Looking Up: an account of the COBUILD Project in lexical co.orang, Collins, London and Glasgow.
Cited by
Sorted by citations
Neural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821 · see citing works
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630 · see citing works
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · see citing works
“Given a question q and an answer option a i , it uses PMI or point-wise mutual information (Church and Hanks, 1989) to measure the strength of the associations between parts of q and parts of a i .”
MethodPhishStorm: Detecting Phishing With Streaming Analytics
Marchal, S., Francois, J., State, R., & Engel, T. (2014) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2014.2377295 · Cited by 271 · see citing works
Deniz, F., Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019) · The Journal of Neuroscience · doi:10.1523/jneurosci.0675-19.2019 · Cited by 214 · see citing works
“The semantic content of the stories was estimated continuously by projecting the narrative into a word embedding space based on word co-occurrence statistics (Church and Hanks, 1990; Lund and Burgess, 1996; Mitchell et al., 2008; Turney and Pantel, 2010; Wehbe et al., 2014).”
MethodLongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.91 · Cited by 96 · see citing works
Privacy lost: Appropriating surveillance technology in China’s fight against COVID-19
Liu, J., & Zhao, H. (2021) · Business Horizons · doi:10.1016/j.bushor.2021.07.004 · Cited by 38 · see citing works
Zhou, J., Chen, Z., Wang, B., & Huang, M. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.96 · Cited by 25 · see citing works
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12 · see citing works
“…based importance metric s i for each token x i in { x doc k } K ′ k =1 can be formulated as: (3) Additionally, we provide the derivation of its mathematical significance in the Appendix A, concluding that it is equivalent to conditional pointwise mutual information (Church and Hanks, 1989).”
Background
- Authors
- Church, K. W., & Hanks, P.
- Affiliations
- AT&T (United States) (US)
- Portuguese Air Transportations (Portugal) (PT)
- ORCID
- Church, Kenneth Ward
- Venue
- Proceedings of the 27th annual meeting on Association for Computational Linguistics -
- Published
- 1989
- Pages
- 76-83
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2159637323
- Cited by
- 1,094
- References
- 0
- Citations by year
- 2026: 27 · 2025: 50 · 2024: 48 · 2023: 59 · 2022: 50 · 2021: 74 · 2020: 80 · 2019: 43 · 2018: 27 · 2017: 36 · 2016: 56 · 2015: 31 · 2014: 62 · 2013: 50 · 2012: 39
- FWCI
- 5.67
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion)
- OLAC - Open Language Archives Community (submittedVersion)
- Americanae (AECID Library) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Lexicography and Language Studies
- Social Sciences › Arts and Humanities › Language and Linguistics › linguistics and terminology studies
- Keywords
- word association norms, mutual information, lexicography, word co-occurrence, semantic relations, content words, function words
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
