The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Measuring academic influence: Not all citations are equal
Open accessAbstract
The importance of a research article is routinely measured by counting how many times it has been cited. However, treating all citations with equal weight ignores the wide variety of functions that citations perform. We want to automatically identify the subset of references in a bibliography that have a central academic influence on the citing paper. For this purpose, we examine the effectiveness of a variety of features for determining the academic influence of a citation. By asking authors to identify the key references in their own work, we created a data set in which citations were labeled according to their academic influence. Using automatic feature selection with supervised machine learning, we found a model for predicting academic influence that achieves good performance on this data set using only four features. The best features, among those we evaluated, were those based on the number of times a reference is mentioned in the body of a citing paper. The performance of these features inspired us to design an influence‐primed h‐index (the hip‐index). Unlike the conventional h‐index, it weights citations by how many times a reference is mentioned. According to our experiments, the hip‐index is a better indicator of researcher performance than the conventional h‐index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
The History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770 · cited 3×
“Nevertheless, a weighted citation measure based on the number of occurrences of a citation could significantly alter other evaluation metrics that depend on simple citation counts, such as the Impact Factor (Garfield, 2006), Eigenfactor, and Article Influence (Bergstrom, 2007).”
BackgroundWhat do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607 · cited 1×
“Citations are a noisy way to track how ideas spread, because a reference may be cited for many reasons other than being the source of an influential idea (Bornmann & Daniel, 2008b).”
BackgroundNigel Gilbert, G. (1977) · Social Studies of Science · doi:10.1177/030631277700700112 · Cited by 691
Coercive Citation in Academic Publishing
Wilhite, A. W., & Fong, E. A. (2012) · Science · doi:10.1126/science.1212540 · Cited by 433
Content Analysis of References: Adjunct or Alternative to Citation Counting?
Chubin, D. E., & Moitra, S. D. (1975) · Social Studies of Science · doi:10.1177/030631277500500403 · Cited by 357
FaBiO and CiTO: Ontologies for describing bibliographic resources and citations
Peroni, S., & Shotton, D. (2012) · Journal of Web Semantics · doi:10.1016/j.websem.2012.08.001 · Cited by 212 · cited 1×
“Thus CiTO, the Citation Typing Ontology (Peroni & Shotton, 2012; Shotton, 2010), provides a rich machine-readable RDF metadata ontology for the characterization of bibliographic citations.”
BackgroundSimkin, M. V., Roychowdhury, V. P., & Department of Electrical Engineering, U. O. C. (2024) · Complex Systems · doi:10.25088/complexsystems.14.3.269 · Cited by 192
Concentration of the Most-Cited Papers in the Scientific Literature: Analysis of Journal Ecosystems
Ioannidis, J. P. A. (2006) · PLoS ONE · doi:10.1371/journal.pone.0000005 · Cited by 122 · cited 1×
“It is known that survey papers and methodology papers tend to be more highly cited than research contributions in general (Ioannidis, 2006).”
BackgroundTowards an Automated Citation Classifier
Garzone, M., & Mercer, R. E. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45486-1_28 · Cited by 101
A New Approach for Scientific Citation Classification Using Cue Phrases
Pham, S. B., & Hoffmann, A. (2003) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24581-0_65 · Cited by 42
Research works printed but not linked: 69
- Abu‐Jbara A. &Radev D.(2011).Coherent citation‐based summarization of scientific papers. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies—Volume 1 HLT '11 (pp.500–509). Stroudsburg PA: Association for Computational Linguistics.
- Adler R., 2009, Citation statistics, Statistical Science, 24, 1
- Agarwal S. Choubey L. &Yu H.(2010).Automatically classifying the role of citations in biomedical articles. In AMIA Annual Symposium Proceedings Vol. 2010 (p.11). American Medical Informatics Association.
- 10.1002/asi.21383 · DOI
- Athar A. &Teufel S.(2012).Detection of implicit citations for sentiment detection. In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse ACL '12 (pp.18–261). Stroudsburg PA: Association for Computational Linguistics.
- 10.1007/s11192-012-0637-5 · DOI
- 10.5860/crln.68.5.7804 · DOI · Paper page
- 10.1371/journal.pone.0006022 · DOI · Paper page
- Bornmann L., 2008, Functional use of frequently and infrequently cited articles in citing publications, European Science Editing, 34, 35
- 10.1002/asi.20806 · DOI · Paper page
- Buckley C. &Voorhees E.M.(2000).Evaluating evaluation measure stability. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval SIGIR '00 (pp.33–40). New York NY: ACM. · DOI · Paper page
- Councill I.G. Giles C.L. &Kan M.‐Y.(2008).Parscit: An open‐source crf reference string parsing package. In Proceedings of the Language Resources and Evaluation Conference. European Language Resources Association.
- Darwin C., 1859, On the origin of species by means of natural selection
- Dietz L. Bickel S. &Scheffer T.(2007).Unsupervised prediction of citation influences. In Proceedings of the 24th International Conference on Machine Learning (pp.233–240). ACM. · DOI · Paper page
- 10.1002/asi.21452 · DOI
- 10.1016/j.ipm.2010.01.002 · DOI · Paper page
- Dubin D., 2004, The most influential paper Gerard Salton never wrote, Library Trends, 52, 748
- 10.1007/s11192-006-0144-7 · DOI · Paper page
- Garfield E.(1965).Can citation indexing be automated?In Statistical association methods for mechanized documentation Symposium proceedings ed. M.E. Stevens et al. (Washington: National Bureau of Standards Miscellaneous Publication 269 1965) pp. 188–192.
- Garzone M., 2000, Advances in artificial intelligence, vol. 1822 of lecture notes in computer science, 337
- Giles C.L. Bollacker K.D. &Lawrence S.(1998).CiteSeer: An automatic citation indexing system. InProceedings of the third ACM conference on Digital libraries DL '98(pp.89–98). New York NY: ACM. · DOI · Paper page
- 10.1007/s11192-009-0035-9 · DOI
- 10.1016/j.joi.2010.03.002 · DOI · Paper page
- 10.1007/s11192-005-0279-y · DOI
- Haque A. &Ginsparg P.(2011).Phrases as subtopical concepts in scholarly text. In Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries JCDL '11 (pp.25–28). New York NY: ACM. · doi:10.1145/1998076.1998081
- 10.1073/pnas.0507655102 · DOI · Paper page
- 10.1073/pnas.0707962104 · DOI · Paper page
- 10.1002/bies.201100067 · DOI
- Kaplan D. Iida R. &Tokunaga T.(2009).Automatic extraction of citation contexts for research paper summarization: A coreference‐chain based approach. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries NLPIR4DL '09 (pp.88–95). Stroudsburg PA: Association for Computational Linguistics. · DOI
- Le M.‐H., 2006, Advances in knowledge discovery and data mining, vol. 3918 of lecture notes in computer science, 265
- 10.1038/4441003a · DOI · Paper page
- Leskovec J. Backstrom L. &Kleinberg J.(2009).Meme‐tracking and the dynamics of the news cycle. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD '09 (pp.497–506). New York NY: ACM. · DOI · Paper page
- Lewis D.(1991).Evaluating text categorization. In Proceedings of the Speech and Natural Language Workshop pp. 312–318. · DOI · Paper page
- Liang Y., 2011, Web‐age information management, vol. 6897 of lecture notes in computer science, 403
- Long S.J., 1997, Regression models for categorical and limited dependent variables
- 10.1002/asi.21059 · DOI
- 10.1007/1-4020-4102-0_19 · DOI
- 10.1371/journal.pone.0049246 · DOI
- Mercer R.E., 2004, Advances in artificial intelligence, vol. 3060 of lecture notes in computer science, 75
- Mohammad S. &Turney P.(2010).Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL‐HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text.
- 10.1177/030631277500500106 · DOI · Paper page
- Nanba H. &Okumura M.(1999).Towards multi‐paper summarization using reference information. In International Joint Conference on Artificial Intelligence Vol. 16 pp. 926–931.
- Osgood C.E., 1957, The measurement of meaning
- Pham S., 2003, AI 2003: Advances in artificial intelligence, vol. 2903 of lecture notes in computer science, 759
- 10.1016/0306-4573(76)90048-0 · DOI · Paper page
- 10.1108/eb046814 · DOI · Paper page
- Qazvinian V. &Radev D.R.(2008).Scientific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics—Volume 1 COLING '08 (pp.689–696). Stroudsburg PA: Association for Computational Linguistics. · DOI · Paper page
- Qazvinian V. &Radev D.R.(2010).Identifying non‐explicit citing sentences for citation‐based summarization. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics ACL '10 (pp.555–564). Stroudsburg PA: Association for Computational Linguistics.
- Qi X. Nie L. &Davison B.D.(2007).Measuring similarity to detect qualified links. In Proceedings of the 3rd International Workshop on Adversarial Information Retrieval on the Web AIRWeb '07 (pp.49–56). New York NY: ACM. · doi:10.1145/1244408.1244418
- Radev D.R. Muthukrishnan P. &Qazvinian V.(2009).The ACL anthology network corpus. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries NLPIR4DL '09 (pp.54–61). Stroudsburg PA: Association for Computational Linguistics. · DOI
- Ritchie A. Robertson S. &Teufel S.(2008).Comparing citation contexts for information retrieval. In Proceedings of the 17th ACM Conference on Information and Knowledge Management CIKM '08 (pp.213–222). New York NY: ACM. · DOI · Paper page
- 10.1002/asi.21638 · DOI
- 10.1088/1367-2630/10/4/040201 · DOI
- 10.1186/2041-1480-1-S1-S6 · DOI
- Small H., 1982, Citation context analysis, Progress in Communication Sciences, 3, 287
- 10.1162/089120101753342653 · DOI · Paper page
- Stone P.J., 1966, The general inquirer: A computer approach to content analysis
- Taheriyan M.(2011).Subject classification of research papers based on interrelationships analysis. In Proceedings of the 2011 Workshop on Knowledge Discovery Modeling and Simulation KDMS '11 (pp.39–54). New York NY: ACM. · doi:10.1145/2023568.2023579
- Teufel S. Siddharthan A. &Tidhar D.(2006).Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing EMNLP '06 (pp.103–110). Stroudsburg PA: Association for Computational Linguistics. · DOI · Paper page
- Turney P.D.(1993).Robust classification with context‐sensitive features. In Proceedings of the Sixth International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems pp. 268–276.
- Turney P.D.(1996).The management of context‐sensitive features: A review of strategies. In Proceedings of Workshop on Learning in Context‐Sensitive Domains at the 13th International Conference on Machine Learning (ICML96) pp. 60–66.
- 10.1145/944012.944013 · DOI · Paper page
- 10.1613/jair.2934 · DOI · Paper page
- 10.1016/j.joi.2012.11.009 · DOI · Paper page
- 10.1002/meet.14504701330 · doi:10.1002/meet.14504701330
- 10.1002/asi.21678 · DOI · Paper page
- Witten I.H., 2011, Data mining: Practical machine learning tools and techniques
- 10.1002/asi.21349 · DOI
- 10.1002/asi.20880 · DOI · Paper page
Cited by
A review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356
The science of science: From the perspective of complex systems
Zeng, A., Shen, Z., Zhou, J., Wu, J., Fan, Y., Wang, Y., & Stanley, H. E. (2017) · Physics Reports · doi:10.1016/j.physrep.2017.10.001 · Cited by 410
Tahamtan, I., & Bornmann, L. (2019) · Scientometrics · doi:10.1007/s11192-019-03243-4 · Cited by 236
Measuring the Evolution of a Scientific Field through Citation Frames
Jurgens, D., Kumar, S., Hoover, R., McFarland, D., & Jurafsky, D. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00028 · Cited by 212
Characterizing in-text citations in scientific articles: A large-scale analysis
Boyack, K. W., van Eck, N. J., Colavizza, G., & Waltman, L. (2018) · Journal of Informetrics · doi:10.1016/j.joi.2017.11.005 · Cited by 125
Metadata
- Authors
- Zhu, X., Turney, P., Lemire, D., & Vellino, A.
- Venue
- Journal of the Association for Information Science and Technology
- Publisher
- Wiley
- Published
- 2014-05-21
- Volume
- 66
- Issue
- 2
- Pages
- 408-427
- ISSN
- 2330-1635 (ISSN-L)
- 2330-1643
- OpenAlex
- W1799284535
- Cited by
- 216
- References
- 11
- FWCI
- 10.25
- Type
- journal article
- Language
- en
- Open access
- green
- Funders
- Natural Sciences and Engineering Research Council of Canada (26143)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.