Cited by 210Advanced Graph Neural NetworksGraph Theory and AlgorithmsTopic Modeling

Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs

Open access
Chen, Zhikai, Mao, Haitao, Li, Hang, Jin, Wei, Wen, Hongzhi, Wei, Xiaochi, Wang, Shuaiqiang, Yin, Dawei +3 more·2024·ACM SIGKDD Explorations Newsletter

Abstract

Learning on Graphs has attracted immense attention due to its wide real-world applications. The most popular pipeline for learning on graphs with textual node attributes primarily relies on Graph Neural Networks (GNNs), and utilizes shallow text embedding as initial node representations, which has limitations in general knowledge and profound semantic understanding. In recent years, Large Language Models (LLMs) have been proven to possess extensive common knowledge and powerful semantic comprehension abilities that have revolutionized existing workflows to handle text data. In this paper, we aim to explore the potential of LLMs in graph machine learning, especially the node classification task, and investigate two possible pipelines: LLMs-as-Enhancers and LLMs-as-Predictors. The former leverages LLMs to enhance nodes' text attributes with their massive knowledge and then generate predictions through GNNs. The latter attempts to directly employ LLMs as standalone predictors. We conduct comprehensive and systematical studies on these two pipelines under various settings. From comprehensive empirical results, we make original observations and find new insights that open new possibilities and suggest promising directions to leverage LLMs for learning on graphs. Our codes and datasets are available at: https://github.com/CurryTang/Graph-LLM .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3655103.3655110

Cites

91 of 91 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Only 6 of the 23 surveyed benchmark studies were peer-reviewed at the time of this article.

    “Our review also found that only 6 of the 23 surveyed benchmark studies ( [7], [10], [31], [39], [44], [52]) were peerreviewed at the time of this article, reflecting the early-stage nature of research in this domain.”
  • The HELM benchmark acknowledged the challenge of standardizing language model evaluation across diverse scenarios, indicating an understanding of response variability but not fully addressing it through its methodology.

    “HELM [31]: The study acknowledged the challenge of standardizing language model evaluation across diverse scenarios, indicating an understanding of response variability but not fully addressing it through its methodology.”
  • The HELM benchmark included targeted evaluations and multi-metric measurement across various scenarios, but did not specifically address the tension between helpfulness and harmlessness in its criteria or methodologies, indicating an unresolved challenge in balancing these aspects.

    “HELM [31]: The HELM evaluation framework included targeted evaluations and multi-metric measurement across various scenarios, but did not specifically address the tension between helpfulness and harmlessness in its criteria or methodologies, indicating an unresolved challenge in balancing these aspects within the”
  • HELM acknowledged the challenge of accounting for linguistic diversity but did not provide a solution, focusing on English and limited consideration of other languages.

    “The study acknowledged the challenge of accounting for linguistic diversity in benchmarks, but did not provide a solution, exemplified by its focus on English and limited consideration of other languages, which indicates an unaddressed and unresolved inadequacy regarding linguistic differences and embedded logics.”
  • HELM acknowledged the challenge of ensuring uniform evaluation methods across all models, highlighting ongoing efforts to refine and adapt benchmarking practices, without explicitly stating that this inadequacy has been fully resolved.

    “The study acknowledged the challenge of ensuring uniform evaluation methods across all models, highlighting ongoing efforts to refine and adapt benchmarking practices to maintain relevancy and accuracy in assessments, without explicitly stating that this inadequacy has been fully resolved.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. V. P. Dwivedi, L. Rampášek, M. Galkin, A. Parviz, G. Wolf, A. T. Luu, and D. Beaini. Long range graph benchmark. In Thirty-sixth Conference on Neural In- formation Processing Systems Datasets and Bench- marks Track, 2022.
  2. C. L. Giles, K. D. Bollacker, and S. Lawrence. Citeseer: An automatic citation indexing system. In Proceedings of the Third ACM Conference on Digital Libraries, DL 98, pages 89-98, New York, NY, USA, 1998. ACM.
  3. S. Gui, X. Li, L. Wang, and S. Ji. GOOD: A graph out- of-distribution benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  4. X. He, X. Bresson, T. Laurent, and B. Hooi. Explana- tions as features: Llm-based features for text-attributed graphs. arXiv preprint arXiv:2305.19523, 2023. · arXiv
  5. G. Li, M. Müller, B. Ghanem, and V. Koltun. Train- ing graph neural networks with 1000 layers. In Inter- national conference on machine learning, pages 6437- 6449. PMLR, 2021.
  6. C. Ra↵el, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Explor- ing the limits of transfer learning with a unified text- to-text transformer. Journal of Machine Learning Re- search, 21(140):1-67, 2020.
  7. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
  8. H. Wang, Y. Gao, X. Zheng, P. Zhang, H. Chen, and J. Bu. Graph neural architecture search with gpt-4. arXiv preprint arXiv:2310.01436, 2023. · arXiv

Cited by

Sorted by citations

Authors
Chen, Z., Mao, H., Li, H., Jin, W., Wen, H., Wei, X., Wang, S., Yin, D., Fan, W., Liu, H., & Tang, J.
Affiliations
  • Michigan State University (US)
  • Emory University (US)
  • Baidu (China) (CN)
  • Hong Kong Polytechnic University (HK)
Venue
ACM SIGKDD Explorations Newsletter
Published
2024-03-26
Volume
25
Issue
2
Pages
42-61
ISSN
  • 1931-0145 (ISSN-L)
  • 1931-0153
OpenAlex
W4393277515
Cited by
209
References
57
Citations by year
2026: 57 · 2025: 104 · 2024: 46 · 2023: 2
FWCI
30.67
Type
journal article
Language
en
Open access
Locations
  • ACM SIGKDD Explorations Newsletter (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Graph Theory and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, graph neural networks, node classification, text embeddings, semantic understanding, knowledge improvement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review