Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 216

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Measuring academic influence: Not all citations are equal

Open access
Zhu, Xiaodan, Turney, Peter, Lemire, Daniel, Vellino, André · 2015 · Journal of the Association for Information Science and Technology

Abstract

The importance of a research article is routinely measured by counting how many times it has been cited. However, treating all citations with equal weight ignores the wide variety of functions that citations perform. We want to automatically identify the subset of references in a bibliography that have a central academic influence on the citing paper. For this purpose, we examine the effectiveness of a variety of features for determining the academic influence of a citation. By asking authors to identify the key references in their own work, we created a data set in which citations were labeled according to their academic influence. Using automatic feature selection with supervised machine learning, we found a model for predicting academic influence that achieves good performance on this data set using only four features. The best features, among those we evaluated, were those based on the number of times a reference is mentioned in the body of a citing paper. The performance of these features inspired us to design an influence‐primed h‐index (the hip‐index). Unlike the conventional h‐index, it weights citations by how many times a reference is mentioned. According to our experiments, the hip‐index is a better indicator of researcher performance than the conventional h‐index.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/asi.23179

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 69

  1. Abu‐Jbara A. &Radev D.(2011).Coherent citation‐based summarization of scientific papers. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies—Volume 1 HLT '11 (pp.500–509). Stroudsburg PA: Association for Computational Linguistics.
  2. Adler R., 2009, Citation statistics, Statistical Science, 24, 1
  3. Agarwal S. Choubey L. &Yu H.(2010).Automatically classifying the role of citations in biomedical articles. In AMIA Annual Symposium Proceedings Vol. 2010 (p.11). American Medical Informatics Association.
  4. 10.1002/asi.21383 · DOI
  5. Athar A. &Teufel S.(2012).Detection of implicit citations for sentiment detection. In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse ACL '12 (pp.18–261). Stroudsburg PA: Association for Computational Linguistics.
  6. 10.1007/s11192-012-0637-5 · DOI
  7. 10.5860/crln.68.5.7804 · DOI · Paper page
  8. 10.1371/journal.pone.0006022 · DOI · Paper page
  9. Bornmann L., 2008, Functional use of frequently and infrequently cited articles in citing publications, European Science Editing, 34, 35
  10. 10.1002/asi.20806 · DOI · Paper page
  11. Buckley C. &Voorhees E.M.(2000).Evaluating evaluation measure stability. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval SIGIR '00 (pp.33–40). New York NY: ACM. · DOI · Paper page
  12. Councill I.G. Giles C.L. &Kan M.‐Y.(2008).Parscit: An open‐source crf reference string parsing package. In Proceedings of the Language Resources and Evaluation Conference. European Language Resources Association.
  13. Darwin C., 1859, On the origin of species by means of natural selection
  14. Dietz L. Bickel S. &Scheffer T.(2007).Unsupervised prediction of citation influences. In Proceedings of the 24th International Conference on Machine Learning (pp.233–240). ACM. · DOI · Paper page
  15. 10.1002/asi.21452 · DOI
  16. 10.1016/j.ipm.2010.01.002 · DOI · Paper page
  17. Dubin D., 2004, The most influential paper Gerard Salton never wrote, Library Trends, 52, 748
  18. 10.1007/s11192-006-0144-7 · DOI · Paper page
  19. Garfield E.(1965).Can citation indexing be automated?In Statistical association methods for mechanized documentation Symposium proceedings ed. M.E. Stevens et al. (Washington: National Bureau of Standards Miscellaneous Publication 269 1965) pp. 188–192.
  20. Garzone M., 2000, Advances in artificial intelligence, vol. 1822 of lecture notes in computer science, 337
  21. Giles C.L. Bollacker K.D. &Lawrence S.(1998).CiteSeer: An automatic citation indexing system. InProceedings of the third ACM conference on Digital libraries DL '98(pp.89–98). New York NY: ACM. · DOI · Paper page
  22. 10.1007/s11192-009-0035-9 · DOI
  23. 10.1016/j.joi.2010.03.002 · DOI · Paper page
  24. 10.1007/s11192-005-0279-y · DOI
  25. Haque A. &Ginsparg P.(2011).Phrases as subtopical concepts in scholarly text. In Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries JCDL '11 (pp.25–28). New York NY: ACM. · doi:10.1145/1998076.1998081
  26. 10.1073/pnas.0507655102 · DOI · Paper page
  27. 10.1073/pnas.0707962104 · DOI · Paper page
  28. 10.1002/bies.201100067 · DOI
  29. Kaplan D. Iida R. &Tokunaga T.(2009).Automatic extraction of citation contexts for research paper summarization: A coreference‐chain based approach. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries NLPIR4DL '09 (pp.88–95). Stroudsburg PA: Association for Computational Linguistics. · DOI
  30. Le M.‐H., 2006, Advances in knowledge discovery and data mining, vol. 3918 of lecture notes in computer science, 265
  31. 10.1038/4441003a · DOI · Paper page
  32. Leskovec J. Backstrom L. &Kleinberg J.(2009).Meme‐tracking and the dynamics of the news cycle. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD '09 (pp.497–506). New York NY: ACM. · DOI · Paper page
  33. Lewis D.(1991).Evaluating text categorization. In Proceedings of the Speech and Natural Language Workshop pp. 312–318. · DOI · Paper page
  34. Liang Y., 2011, Web‐age information management, vol. 6897 of lecture notes in computer science, 403
  35. Long S.J., 1997, Regression models for categorical and limited dependent variables
  36. 10.1002/asi.21059 · DOI
  37. 10.1007/1-4020-4102-0_19 · DOI
  38. 10.1371/journal.pone.0049246 · DOI
  39. Mercer R.E., 2004, Advances in artificial intelligence, vol. 3060 of lecture notes in computer science, 75
  40. Mohammad S. &Turney P.(2010).Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL‐HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text.
  41. 10.1177/030631277500500106 · DOI · Paper page
  42. Nanba H. &Okumura M.(1999).Towards multi‐paper summarization using reference information. In International Joint Conference on Artificial Intelligence Vol. 16 pp. 926–931.
  43. Osgood C.E., 1957, The measurement of meaning
  44. Pham S., 2003, AI 2003: Advances in artificial intelligence, vol. 2903 of lecture notes in computer science, 759
  45. 10.1016/0306-4573(76)90048-0 · DOI · Paper page
  46. 10.1108/eb046814 · DOI · Paper page
  47. Qazvinian V. &Radev D.R.(2008).Scientific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics—Volume 1 COLING '08 (pp.689–696). Stroudsburg PA: Association for Computational Linguistics. · DOI · Paper page
  48. Qazvinian V. &Radev D.R.(2010).Identifying non‐explicit citing sentences for citation‐based summarization. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics ACL '10 (pp.555–564). Stroudsburg PA: Association for Computational Linguistics.
  49. Qi X. Nie L. &Davison B.D.(2007).Measuring similarity to detect qualified links. In Proceedings of the 3rd International Workshop on Adversarial Information Retrieval on the Web AIRWeb '07 (pp.49–56). New York NY: ACM. · doi:10.1145/1244408.1244418
  50. Radev D.R. Muthukrishnan P. &Qazvinian V.(2009).The ACL anthology network corpus. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries NLPIR4DL '09 (pp.54–61). Stroudsburg PA: Association for Computational Linguistics. · DOI
  51. Ritchie A. Robertson S. &Teufel S.(2008).Comparing citation contexts for information retrieval. In Proceedings of the 17th ACM Conference on Information and Knowledge Management CIKM '08 (pp.213–222). New York NY: ACM. · DOI · Paper page
  52. 10.1002/asi.21638 · DOI
  53. 10.1088/1367-2630/10/4/040201 · DOI
  54. 10.1186/2041-1480-1-S1-S6 · DOI
  55. Small H., 1982, Citation context analysis, Progress in Communication Sciences, 3, 287
  56. 10.1162/089120101753342653 · DOI · Paper page
  57. Stone P.J., 1966, The general inquirer: A computer approach to content analysis
  58. Taheriyan M.(2011).Subject classification of research papers based on interrelationships analysis. In Proceedings of the 2011 Workshop on Knowledge Discovery Modeling and Simulation KDMS '11 (pp.39–54). New York NY: ACM. · doi:10.1145/2023568.2023579
  59. Teufel S. Siddharthan A. &Tidhar D.(2006).Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing EMNLP '06 (pp.103–110). Stroudsburg PA: Association for Computational Linguistics. · DOI · Paper page
  60. Turney P.D.(1993).Robust classification with context‐sensitive features. In Proceedings of the Sixth International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems pp. 268–276.
  61. Turney P.D.(1996).The management of context‐sensitive features: A review of strategies. In Proceedings of Workshop on Learning in Context‐Sensitive Domains at the 13th International Conference on Machine Learning (ICML96) pp. 60–66.
  62. 10.1145/944012.944013 · DOI · Paper page
  63. 10.1613/jair.2934 · DOI · Paper page
  64. 10.1016/j.joi.2012.11.009 · DOI · Paper page
  65. 10.1002/meet.14504701330 · doi:10.1002/meet.14504701330
  66. 10.1002/asi.21678 · DOI · Paper page
  67. Witten I.H., 2011, Data mining: Practical machine learning tools and techniques
  68. 10.1002/asi.21349 · DOI
  69. 10.1002/asi.20880 · DOI · Paper page

Cited by

See all 216

Metadata

Authors
Zhu, X., Turney, P., Lemire, D., & Vellino, A.
Venue
Journal of the Association for Information Science and Technology
Publisher
Wiley
Published
2014-05-21
Volume
66
Issue
2
Pages
408-427
ISSN
  • 2330-1635 (ISSN-L)
  • 2330-1643
OpenAlex
W1799284535
Cited by
216
References
11
FWCI
10.25
Type
journal article
Language
en
Open access
green
Funders
Natural Sciences and Engineering Research Council of Canada (26143)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.