CoChat
Sign Up
Cited by 966scientometrics and bibliometrics researchWeb visibility and informetricsPsychology Research and Bibliometrics

How to normalize cooccurrence data? An analysis of some well‐known similarity measures

Open access
Eck, Nees Jan van, Waltman, Ludo·2009·Journal of the American Society for Information Science and Technology
DOI 10.1002/asi.21075
Open this paper in CoChat — free accountOpen PDF

Abstract

In scientometric research, the use of cooccurrence data is very common. In many cases, a similarity measure is employed to normalize the data. However, there is no consensus among researchers on which similarity measure is most appropriate for normalization purposes. In this article, we theoretically analyze the properties of similarity measures for cooccurrence data, focusing in particular on four well‐known measures: the association strength, the cosine, the inclusion index, and the Jaccard index. We also study the behavior of these measures empirically. Our analysis reveals that there exist two fundamentally different types of similarity measures, namely, set‐theoretic measures and probabilistic measures. The association strength is a probabilistic measure, while the cosine, the inclusion index, and the Jaccard index are set‐theoretic measures. Both our theoretical and our empirical results indicate that cooccurrence data can best be normalized using a probabilistic measure. This provides strong support for the use of the association strength in scientometric research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/asi.21075

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Reviewing co-occurrences of data is a way to identify the impact of variables in a research topic on one another and to reveal their potential correlations.

    “Reviewing the co-occurrences of data is a way to identify the impact of a variety of variables in a research topic on one another and to reveal their potential correlations.”

    From Components reuse in the building sector – A systematic review (2020) · cites this paper as van Eck and Waltman, 2009

    The Introduction (page unavailable) says co-occurrence data can be used to study relations among authors or journals and to derive similarities; the Empirical comparison section (page unavailable) uses word co-occurrences to assess whether words have a strong relation. These passages support revealing potential associations, but they do not say co-occurrences identify the impact of research variables on one another. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Identifying the correlation between key variables helps in better devising solutions to achieve the objectives of the study.

    “Moreover, identifying the correlation between the key variables helps in better devising solutions to achieve the objectives of the study (Rameezdeen et al., 2016; van Eck and Waltman, 2009).”

    From Components reuse in the building sector – A systematic review (2020) · cites this paper as van Eck and Waltman, 2009

    The Empirical comparison section uses correlations to compare similarity measures, and the Conclusions say that choosing an appropriate measure has practical relevance. The paper does not establish the broader claim that identifying correlations between a study’s key variables helps devise solutions to achieve its objectives. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 88

  1. P. Ahlgren, B. Jarneving, and R. Rousseau. Requirements for a cocitation similarity measure, with special reference to Pearson's correlation coefficient. Journal of the American Society for Information Science and Technology, 54(6):550-560, 2003.
  2. M.R. Anderberg. Cluster analysis for applications. Academic Press, 1973.
  3. F.B. Baulieu. A classification of presence/absence based dissimilarity coefficients. Journal of Classification, 6(1):233-246, 1989.
  4. F.B. Baulieu. Two variant axiom systems for presence/absence based dissimilarity coefficients. Journal of Classification, 14(1):159-170, 1997.
  5. I. Borg and P.J.F. Groenen. Modern multidimensional scaling. Springer, 2nd edition, 2005.
  6. K.W. Boyack, R. Klavans, and K. Börner. Mapping the backbone of science. Scientometrics, 64(3):351-374, 2005.
  7. R.R. Braam, H.F. Moed, and A.F.J. van Raan. Mapping of science by combined co-citation and word analysis. I. Structural aspects. Journal of the American Society for Information Science, 42(4):233-251, 1991a.
  8. R.R. Braam, H.F. Moed, and A.F.J. van Raan. Mapping of science by combined co-citation and word analysis. II. Dynamical aspects. Journal of the American Society for Information Science, 42(4):252-266, 1991b.
  9. M. Callon, J.P. Courtial, and F. Laville. Co-word analysis as a tool for describing the network of interactions between basic and technological research: The case of polymer chemistry. Scientometrics, 22(1):155-205, 1991.
  10. Y.M. Chung and J.Y. Lee. A corpus-based approach to comparative evaluation of statistical term association measures. Journal of the American Society for Information Science and Technology, 52(4):283-296, 2001.
  11. K.W. Church and P. Hanks. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29, 1990.
  12. M.A.A. Cox and T.F. Cox. Multidimensional scaling. In C. Chen, W. Härdle, and A. Unwin, editors, Handbook of data visualization, pages 315-347. Springer, 2008.
  13. T.F. Cox and M.A.A. Cox. Multidimensional scaling. Chapman & Hall/CRC, 2nd edition, 2001.
  14. D. de Solla Price. The analysis of scientometric matrices for policy implications. Scientomet- rics, 3(1):47-53, 1981.
  15. F. Drasgow and L.E. Jones. Multidimensional scaling of derived dissimilarities. Multivariate Behavioral Research, 14(2):227-244, 1979.
  16. L. Egghe. New relations between similarity measures for vectors based on vector norms. Jour- nal of the American Society for Information Science and Technology, 2008. In press.
  17. L. Egghe and C. Michel. Strong similarity measures for ordered sets of documents in informa- tion retrieval. Information Processing and Management, 38(6):823-848, 2002.
  18. L. Egghe and C. Michel. Construction of weak and strong similarity measures for ordered sets of documents using fuzzy set techniques. Information Processing and Management, 39(5): 771-807, 2003.
  19. L. Egghe and R. Rousseau. Classical retrieval and overlap measures satisfy the requirements for rankings based on a Lorenz curve. Information Processing and Management, 42(1):106-120, 2006.
  20. W. Glänzel. National characteristics in international scientific co-authorship relations. Sciento- metrics, 51(1):69-115, 2001.
  21. W. Glänzel, A. Schubert, and H.-J. Czerwon. A bibliometric analysis of international scientific cooperation of the European Union (1985-1995). Scientometrics, 45(2):185-202, 1999.
  22. M. Gmür. Co-citation analysis and the search for invisible colleges: A methodological evalua- tion. Scientometrics, 57(1):27-57, 2003.
  23. J.C. Gower. Measures of similarity, dissimilarity, and distance. In S. Kotz and N.L. Johnson, editors, Encyclopedia of statistical sciences, volume 5, pages 397-405. Wiley, 1985.
  24. J.C. Gower and P. Legendre. Metric and Euclidean properties of dissimilarity coefficients. Journal of Classification, 3(1):5-48, 1986.
  25. J.P. Guilford. Fundamental statistics in psychology and education. McGraw-Hill, 5th edition, 1973.
  26. L. Hamers, Y. Hemeryck, G. Herweyers, M. Janssen, H. Keters, R. Rousseau, and A. Van- houtte. Similarity measures in scientometric research: The Jaccard index versus Salton's cosine formula. Information Processing and Management, 25(3):315-318, 1989.
  27. G.H. Hardy, J.E. Littlewood, and G. Pólya. Inequalities. Cambridge University Press, 2nd edition, 1952.
  28. G. Heimeriks, M. Hörlesberger, and P. van den Besselaar. Mapping communication and collab- oration in heterogeneous research networks. Scientometrics, 58(2):391-413, 2003.
  29. S. Hinze. Bibliographical cartography of an emerging interdisciplinary discipline: The case of bioelectronics. Scientometrics, 29(3):353-376, 1994.
  30. Z. Hubálek. Coefficients of association and similarity, based on binary (presence-absence) data: An evaluation. Biological Reviews, 57(4):669-689, 1982.
  31. S. Janson and J. Vegelius. Measures of ecological association. Oecologia, 49(3):371-376, 1981.
  32. B. Jarneving. A variation of the calculation of the first author cocitation strength in author cocitation analysis. Scientometrics, 2008. In press.
  33. W.P. Jones and G.W. Furnas. Pictures of relevance: A geometric analysis of similarity measures. Journal of the American Society for Information Science, 38(6):420-442, 1987.
  34. R. Klavans and K.W. Boyack. Identifying a better measure of relatedness for mapping science. Journal of the American Society for Information Science and Technology, 57(2):251-263, 2006a.
  35. R. Klavans and K.W. Boyack. Quantitative evaluation of large maps of science. Scientometrics, 68(3):475-499, 2006b.
  36. A. Kopcsa and E. Schiebel. Science and technology mapping: A new iteration model for rep- resenting multidimensional relationships. Journal of the American Society for Information Science, 49(1):7-17, 1998.
  37. R.N. Kostoff, H.J. Eberhart, and D.R. Toothman. Hypersonic and supersonic flow roadmaps us- ing bibliometrics and database tomography. Journal of the American Society for Information Science, 50(5):427-447, 1999.
  38. R.N. Kostoff, J.A. del Río, J.A. Humenik, E.O. García, and A.M. Ramírez. Citation mining: Integrating text mining and bibliometrics for research user profiling. Journal of the American Society for Information Science and Technology, 52(13):1148-1156, 2001.
  39. J. Law and J. Whittaker. Mapping acidification research: A test of the co-word method. Scien- tometrics, 23(3):417-461, 1992.
  40. M. Leclerc and J. Gagné. International scientific cooperation: The continentalization of science. Scientometrics, 31(3):261-292, 1994.
  41. L. Leydesdorff. Words and co-words as indicators of intellectual organization. Research Policy, 18(4):209-223, 1989.
  42. L. Leydesdorff. On the normalization and visualization of author co-citation data: Salton's cosine versus the Jaccard index. Journal of the American Society for Information Science and Technology, 59(1):77-85, 2008.
  43. L. Leydesdorff and L. Vaughan. Co-occurrence matrices and their applications in information science: Extending ACA to the Web environment. Journal of the American Society for Information Science and Technology, 57(12):1616-1628, 2006.
  44. T. Luukkonen, O. Persson, and G. Sivertsen. Understanding patterns of international scientific collaboration. Science, Technology, and Human Values, 17(1):101-126, 1992.
  45. T. Luukkonen, R.J.W. Tijssen, O. Persson, and G. Sivertsen. The measurement of international scientific collaboration. Scientometrics, 28(1):15-36, 1993.
  46. C.D. Manning and H. Schütze. Foundations of statistical natural language processing. MIT Press, 1999.
  47. K.W. McCain. Mapping authors in intellectual space: A technical overview. Journal of the American Society for Information Science, 41(6):433-443, 1990.
  48. K.W. McCain. Mapping economics through the journal literature: An experiment in journal cocitation analysis. Journal of the American Society for Information Science, 42(4):290- 296, 1991.
  49. K.W. McCain. The structure of biotechnology R & D. Scientometrics, 32(2):153-175, 1995.
  50. F. Morillo, M. Bordons, and I. Gómez. Interdisciplinarity in science: A tentative typology of disciplines and research areas. Journal of the American Society for Information Science and Technology, 54(13):1237-1249, 2003.
  51. C.L. Palmer. Structures and strategies of interdisciplinary science. Journal of the American Society for Information Science, 50(3):242-253, 1999.
  52. H.P.F. Peters and A.F.J. van Raan. Co-word-based science maps of chemical engineering. Part I: Representations by direct multidimensional scaling. Research Policy, 22(1):23-45, 1993a.
  53. H.P.F. Peters and A.F.J. van Raan. Co-word-based science maps of chemical engineering. Part II: Representations by combined clustering and multidimensional scaling. Research Policy, 22(1):23-45, 1993b.
  54. H.P.F. Peters, R.R. Braam, and A.F.J. van Raan. Cognitive resemblance and citation relations in chemical engineering publications. Journal of the American Society for Information Science, 46(1):9-21, 1995.
  55. J. Qin. Semantic similarities between a keyword database and a controlled vocabulary database: An investigation in the antibiotic resistance literature. Journal of the American Society for Information Science, 51(3):166-180, 2000.
  56. A. Rip and J.-P. Courtial. Co-word maps of biotechnology: An example of cognitive sciento- metrics. Scientometrics, 6(6):381-400, 1984.
  57. M. Rorvig. Images of similarity: A visual exploration of optimal similarity metrics and scaling properties of TREC topic-document sets. Journal of the American Society for Information Science, 50(8):639-651, 1999.
  58. S. Rosenberg and R. Jones. A method for investigating and representing a person's implicit theory of personality: Theodore Dreiser's view of people. Journal of Personality and Social Psychology, 22(3):372-386, 1972.
  59. S. Rosenberg, C. Nelson, and P.S. Vivekananthan. A multidimensional approach to the structure of personality impressions. Journal of Personality and Social Psychology, 9(4):283-294, 1968.
  60. G. Salton. Associative document retrieval techniques using bibliographic information. Journal of the ACM, 10(4):440-457, 1963.
  61. G. Salton and M.J. McGill. Introduction to modern information retrieval. McGraw-Hill, 1983.
  62. J.W. Schneider and P. Borlund. Matrix comparison, part 1: Motivation and important issues for measuring the resemblance between proximity measures or ordination results. Journal of the American Society for Information Science and Technology, 58(11):1586-1595, 2007a.
  63. J.W. Schneider and P. Borlund. Matrix comparison, part 2: Measuring the resemblance between proximity measures or ordination results by use of the Mantel and Procrustes statistics. Jour- nal of the American Society for Information Science and Technology, 58(11):1596-1609, 2007b.
  64. J.W. Schneider, B. Larsen, and P. Ingwersen. A comparative study of first and all-author co-citation counting, and two different matrix generation approaches applied for author co- citation analyses. Scientometrics, 2008. In press.
  65. A. Schubert and T. Braun. International collaboration in the sciences, 1981-1985. Scientomet- rics, 19(1-2):3-10, 1990.
  66. M.W. Simmen. Multidimensional scaling of binary dissimilarities: Direct and derived ap- proaches. Multivariate Behavioral Research, 31(1):47-67, 1996.
  67. H. Small. Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for Information Science, 24(4):265-269, 1973.
  68. H. Small. The relationship of information science to the social sciences: A co-citation analysis. Information Processing and Management, 17(1):39-50, 1981.
  69. H. Small. A SCI-Map case study: Building a map of AIDS research. Scientometrics, 30(1): 229-241, 1994.
  70. H. Small and E. Greenlee. Citation context analysis of a co-citation cluster: Recombinant-DNA. Scientometrics, 2(4):277-301, 1980.
  71. H. Small and E. Sweeney. Clustering the science citation index using co-citations. I. A compar- ison of methods. Scientometrics, 7(3-6):391-409, 1985.
  72. H. Small, E. Sweeney, and E. Greenlee. Clustering the science citation index using co-citations. II. Mapping science. Scientometrics, 8(5-6):321-340, 1985.
  73. R.J.W. Tijssen. A quantitative assessment of interdisciplinary structures in science and technol- ogy: Co-classification analysis of energy research. Research Policy, 21(1):27-44, 1992.
  74. R.J.W. Tijssen. A scientometric cognitive study of neural network research: Expert mental maps versus bibliometric maps. Scientometrics, 28(1):111-136, 1993.
  75. R.J.W. Tijssen and A.F.J. van Raan. Mapping co-word structures: A comparison of multidi- mensional scaling and LEXIMAPPE. Scientometrics, 15(3-4):283-295, 1989.
  76. W.A. van der Kloot and H. van Herk. Multidimensional scaling of sorting data: A comparison of three procedures. Multivariate Behavioral Research, 26(4):563-581, 1991.
  77. N.J. van Eck and L. Waltman. Bibliometric mapping of the computational intelligence field. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 15(5):625- 645, 2007.
  78. N.J. van Eck and L. Waltman. Appropriate similarity measures for author co-citation analysis. Journal of the American Society for Information Science and Technology, 59(10):1653-1661, 2008.
  79. N.J. van Eck, L. Waltman, J. van den Berg, and U. Kaymak. Visualizing the computational intelligence field. IEEE Computational Intelligence Magazine, 1(4):6-10, 2006.
  80. A.F.J. van Raan and R.J.W. Tijssen. The neural net of neural network research: An exercise in bibliometric mapping. Scientometrics, 26(1):169-192, 1993.
  81. L. Vaughan. Visualizing linguistic and cultural differences using Web co-link data. Journal of the American Society for Information Science and Technology, 57(9):1178-1193, 2006.
  82. L. Vaughan and J. You. Comparing business competition positions based on Web co-link data: The global market vs. the Chinese market. Scientometrics, 68(3):611-628, 2006.
  83. L. Waltman and N.J. van Eck. Some comments on the question whether co-occurrence data should be normalized. Journal of the American Society for Information Science and Technol- ogy, 58(11):1701-1703, 2007.
  84. M.J. Warrens. Similarity coefficients for binary data. PhD thesis, Leiden University, 2008.
  85. H.D. White and B.C. Griffith. Author cocitation: A literature measure of intellectual structure. Journal of the American Society for Information Science, 32(3):163-171, 1981.
  86. H.D. White and K.W. McCain. Visualizing a discipline: An author co-citation analysis of information science, 1972-1995. Journal of the American Society for Information Science, 49(4):327-355, 1998.
  87. F.E. Zegers and J.M.F. ten Berge. A family of association coefficients for metric scales. Psy- chometrika, 50(1):17-24, 1985.
  88. M. Zitt, E. Bassecoulard, and Y. Okubo. Shadows of the past in international cooperation: Collaboration profiles of the top five producers of science. Scientometrics, 47(3):627-657, 2000.

Cited by

Sorted by citations

Authors
Eck, N. J. V., & Waltman, L.
Affiliations
  • Erasmus University Rotterdam (NL)
  • Leiden University (NL)
Corresponding author
  • Eck, Nees Jan van
  • Waltman, Ludo
ORCID
Venue
Journal of the American Society for Information Science and Technology
Publisher
Wiley
Published
2009-04-13
Volume
60
Issue
8
Pages
1635-1651
ISSN
  • 1532-2882 (ISSN-L)
  • 1532-2890
DOI
10.1002/asi.21075
OpenAlex
W1503479763
Cited by
966
References
0
Citations by year
2026: 105 · 2025: 110 · 2024: 110 · 2023: 103 · 2022: 140 · 2021: 89 · 2020: 87 · 2019: 55 · 2018: 42 · 2017: 27 · 2016: 15 · 2015: 21 · 2014: 25 · 2013: 8 · 2012: 6
FWCI
11.60
Type
journal article
Language
en
Open access
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations
  • Journal of the American Society for Information Science and Technology (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • ERIM Report Series Research in Management (submittedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
similarity measures, normalization, associative strength, cosine similarity, inclusion index, Jaccard similarity, probabilistic measures, co-word analysis, scientometrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review