research-article

GraphGPT: Graph Instruction Tuning for Large Language Models

Authors:

Chao HuangAuthors Info & Claims

SIGIR '24: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 491 - 500

https://doi.org/10.1145/3626772.3657775

Published: 11 July 2024 Publication History

Abstract

Graph Neural Networks (GNNs) have evolved to understand graph structures through recursive exchanges and aggregations among nodes. To enhance robustness, self-supervised learning (SSL) has become a vital tool for data augmentation. Traditional methods often depend on fine-tuning with task-specific labels, limiting their effectiveness when labeled data is scarce. Our research tackles this by advancing graph model generalization in zero-shot learning environments. Inspired by the success of large language models (LLMs), we aim to create a graph-oriented LLM capable of exceptional generalization across various datasets and tasks without relying on downstream graph data. We introduce the GraphGPT framework, which integrates LLMs with graph structural knowledge through graph instruction tuning. This framework includes a text-graph grounding component to link textual and graph structures and a dual-stage instruction tuning approach with a lightweight graph-text alignment projector. These innovations allow LLMs to comprehend complex graph structures and enhance adaptability across diverse datasets and tasks. Our framework demonstrates superior generalization in both supervised and zero-shot graph learning tasks, surpassing existing benchmarks. The open-sourced model implementation of our GraphGPT is available at https://github.com/HKUDS/GraphGPT.

References

[1]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, et al. 2022. Constitutional AI: Harmlessness from AI Feedback. CoRR, Vol. abs/2212.08073 (2022).

[2]

Zhikai Chen, Haitao Mao, Hang Li, et al. 2023. Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs. CoRR, Vol. abs/2307.03393 (2023).

[3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1). Association for Computational Linguistics, 4171--4186.

[4]

Yushun Dong, Ninghao Liu, Brian Jalaian, et al. 2022. EDITS: Modeling and Mitigating Data Bias for Graph Neural Networks. In WWW. ACM, 1259--1269.

[5]

Jiayan Guo, Lun Du, and Hengyu Liu. 2023 a. GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking. CoRR, Vol. abs/2305.15066 (2023).

[6]

Zhichun Guo, Kehan Guo, Bozhao Nan, Yijun Tian, Roshni G. Iyer, et al. 2023 b. Graph-based Molecular Representation Learning. In IJCAI. 6638--6646.

[7]

William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In NeurIPS. 1024--1034.

[8]

Xiaoxin He, Xavier Bresson, et al. 2023. Explanations as Features: LLM-Based Features for Text-Attributed Graphs. CoRR, Vol. abs/2305.19523 (2023).

[9]

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR. ACM, 639--648.

[10]

Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, et al. 2023. GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In WWW. 737--746.

[11]

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Jie Tang, et al. 2022. Graphmae: Self-supervised masked graph autoencoders. In KDD. 594--604.

[12]

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, et al. 2020c. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In NeurIPS.

[13]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020b. Gpt-gnn: Generative pre-training of graph neural networks. In KDD. 1857--1867.

Digital Library

[14]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020a. Heterogeneous Graph Transformer. In WWW. ACM / IW3C2, 2704--2710.

[15]

Yangqin Jiang, Chao Huang, and Lianghao Huang. 2023. Adaptive graph contrastive learning for recommendation. In KDD. 4252--4261.

[16]

Baoyu Jing, Chanyoung Park, and Hanghang Tong. 2021. Hdmi: High-order deep multiplex infomax. In WWW. 2414--2424.

[17]

Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR (Poster). OpenReview.net.

[18]

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In NeurIPS.

[19]

Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, et al. 2023. RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. CoRR, Vol. abs/2309.00267 (2023).

[20]

Bolian Li, Baoyu Jing, and Hanghang Tong. 2022. Graph communal contrastive learning. In WWW. 1203--1213.

[21]

Guohao Li, Matthias Mü ller, Bernard Ghanem, and Vladlen Koltun. 2021. Training Graph Neural Networks with 1000 Layers. In ICML. 6437--6449.

[22]

Mingkai Lin, Wenzhong Li, Ding Li, Yizhou Chen, and Sanglu Lu. 2022. Resource-Efficient Training for Large Graph Convolutional Networks with Label-Centric Cumulative Sampling. In WWW. ACM, 1170--1180.

[23]

Haotian Liu, Chunyuan Li, et al. 2023 a. Visual Instruction Tuning.

[24]

Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and S Yu Philip. 2022. Graph self-supervised learning: A survey. TKDE, Vol. 35, 6 (2022), 5879--5900.

[25]

Yunchao Liu, Yu Wang, Oanh Vu, Rocco Moretti, et al. 2023 b. Interpretable Chirality-Aware Graph Neural Network for Quantitative Structure Activity Relationship Modeling in Drug Discovery. In AAAI. 14356--14364.

[26]

Zemin Liu, Xingtong Yu, et al. 2023 c. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In WWW. 417--428.

[27]

Xiaojun Ma, Qin Chen, et al. 2022. Meta-Weight Graph Neural Network: Push the Limits Beyond Global Homophily. In WWW. ACM, 1270--1280.

[28]

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. In EMNLP. 11048--11064.

[29]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS.

[30]

Alec Radford, Jong Wook Kim, Chris Hallacy, et al. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML). PMLR, 8748--8763.

[31]

Zezhi Shao et al. 2022. Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting. In KDD. ACM, 1567--1577.

[32]

Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling Reasoning Capabilities into Smaller Language Models. In ACL. 7059--7073.

[33]

Mingchen Sun, Kaixiong Zhou, et al. 2022. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In KDD. 1717--1727.

[34]

Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. 2023. All in One: Multi-Task Prompting for Graph Neural Networks. In KDD.

[35]

Qiaoyu Tan, Ninghao Liu, Xiao Huang, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. 2023. S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking. In WSDM. 787--795.

[36]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothé e Lacroix, Baptiste Roziè re, et al. 2023 a. LLaMA: Open and Efficient Foundation Language Models. CoRR, Vol. abs/2302.13971 (2023).

[37]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, et al. 2023 b. Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRR, Vol. abs/2307.09288 (2023).

[38]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, et al. 2017. Attention is all you need. In NeurIPS, Vol. 30.

[39]

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, et al. 2018. Graph Attention Networks. In ICLR (Poster). OpenReview.net.

[40]

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, et al. 2019. Deep Graph Infomax. In ICLR (Poster). OpenReview.net.

[41]

Kuansan Wang, Zhihong Shen, et al. 2020. Microsoft Academic Graph: When experts are not enough. Quant. Sci. Stud., Vol. 1, 1 (2020), 396--413.

[42]

Xiang Wang, Tinglin Huang, Dingxian Wang, et al. 2021. Learning Intents behind Interactions with Knowledge Graph for Recommendation. In WWW. 878--887.

[43]

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, et al. 2019. Heterogeneous Graph Attention Network. In WWW. ACM, 2022--2032.

Digital Library

[44]

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, et al. 2023 a. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. CoRR, Vol. abs/2306.04751 (2023).

[45]

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023 b. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In ACL. 13484--13508.

[46]

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Jeff Dean, William Fedus, et al. 2022a. Emergent Abilities of Large Language Models. Trans. Mach. Learn. Res., Vol. 2022 (2022).

[47]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022b. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS.

[48]

Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2023. LLMRec: Large Language Models with Graph Augmentation for Recommendation. CoRR, Vol. abs/2311.00423 (2023).

[49]

Zhihao Wen and Yuan Fang. 2023. Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting. In SIGIR.

[50]

Qitian Wu, Chenxiao Yang, et al. 2023 a. DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion. In ICLR.

[51]

Qitian Wu, Wentao Zhao, et al. 2023 b. NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classification. CoRR, Vol. abs/2306.08385 (2023).

[52]

Jun Xia, Lirong Wu, Jintao Chen, et al. 2022. Simgrace: A simple framework for graph contrastive learning without data augmentation. In WWW. 1070--1079.

[53]

Lianghao Xia, Chao Huang, Tao Yu, Ben Kao, et al. 2023. Automated Self-Supervised Learning for Recommendation. In WWW. 992--1002.

[54]

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, et al. 2023. Baichuan 2: Open Large-scale Language Models. CoRR, Vol. abs/2309.10305 (2023).

[55]

Chenxiao Yang, Qitian Wu, and Junchi Yan. 2022b. Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks. In NeurIPS.

[56]

Haoran Yang, Hongxu Chen, Shirui Pan, Lin Li, Philip S Yu, and Guandong Xu. 2022a. Dual space graph contrastive learning. In WWW. 1238--1247.

[57]

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. CoRR, Vol. abs/2305.10601 (2023).

[58]

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. 2021. Graph contrastive learning automated. In ICML. PMLR, 12121--12132.

[59]

Yuning You, Tianlong Chen, Yongduo Sui, et al. 2020. Graph contrastive learning with augmentations. In NeurIPS, Vol. 33. 5812--5823.

[60]

Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim. 2019a. Graph Transformer Networks. In NeurIPS. 11960--11970.

[61]

Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. 2019b. Graph transformer networks. In NeurIPS, Vol. 32.

[62]

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, et al. 2023. GLM-130B: An Open Bilingual Pre-trained Model. In ICLR.

[63]

Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. 2022b. Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation. In ICLR.

[64]

Wen Zhang, Yushan Zhu, Mingyang Chen, et al. 2023. Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer. In WWW. 2581--2590.

[65]

Yanfu Zhang et al. 2022a. Robust Self-Supervised Structural Graph Neural Network for Social Network Prediction. In WWW. ACM, 1352--1361.

[66]

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304.10592 (2023).

[67]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2021. Graph contrastive learning with adaptive augmentation. In WWW. 2069--2080.

Cited By

Liu TJiang YWei YWang XHuang SDai L(2024)Educational Practices and Algorithmic Framework for Promoting Sustainable Development in Education by Identifying Real-World Learning PathsSustainability10.3390/su1616687116:16(6871)Online publication date: 10-Aug-2024
https://doi.org/10.3390/su16166871
Da LLiou KChen TZhou XLuo XYang YWei H(2024)Open-ti: open traffic intelligence with augmented language modelInternational Journal of Machine Learning and Cybernetics10.1007/s13042-024-02190-815:10(4761-4786)Online publication date: 9-May-2024
https://doi.org/10.1007/s13042-024-02190-8
Ben JSun QLiu KYang XZhang F(2024)Multi-head multi-order graph attention networksApplied Intelligence10.1007/s10489-024-05601-z54:17-18(8092-8107)Online publication date: 20-Jun-2024
https://doi.org/10.1007/s10489-024-05601-z

Index Terms

GraphGPT: Graph Instruction Tuning for Large Language Models
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Language models
  2. Information systems applications
    1. Data mining
2. Mathematics of computing
  1. Discrete mathematics
    1. Graph theory
      1. Graph algorithms

Recommendations

Large Language Models for Graph Learning
WWW '24: Companion Proceedings of the ACM Web Conference 2024

Graphs are widely applied to encode entities with various relations in web applications such as social media and recommender systems. Meanwhile, graph learning-based technologies, such as graph neural networks, are demanding to support the analysis, ...
Large Language Models for Graphs: Progresses and Directions
WWW '24: Companion Proceedings of the ACM Web Conference 2024

Graph neural networks (GNNs) have emerged as fundamental methods for handling structured graph data in various domains, including citation networks, molecule prediction, and recommender systems. They enable the learning of informative node or graph ...
GraphWiz: An Instruction-Following Language Model for Graph Computational Problems
KDD '24: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Large language models (LLMs) have achieved impressive success across various domains, but their capability in understanding and resolving complex graph problems is less explored. To bridge this gap, we introduce GraphInstruct, a novel instruction-tuning ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '24: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2024

3164 pages

ISBN:9798400704314

DOI:10.1145/3626772

General Chairs:
Grace Hui Yang
Georgetown University, USA
,
Hongning Wang
Tsinghua University, China
,
Sam Han
The Washington Post, USA
,
Program Chairs:
Claudia Hauff
Spotify, Netherlands
,
Guido Zuccon
The University of Queensland, Australia
,
Yi Zhang
University of California Santa Cruz, USA

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR 2024

Sponsor:

SIGIR

SIGIR 2024: The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 14 - 18, 2024

Washington DC, USA

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
440
Total Downloads

Downloads (Last 12 months)440
Downloads (Last 6 weeks)268

Reflects downloads up to 15 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu TJiang YWei YWang XHuang SDai L(2024)Educational Practices and Algorithmic Framework for Promoting Sustainable Development in Education by Identifying Real-World Learning PathsSustainability10.3390/su1616687116:16(6871)Online publication date: 10-Aug-2024
https://doi.org/10.3390/su16166871
Da LLiou KChen TZhou XLuo XYang YWei H(2024)Open-ti: open traffic intelligence with augmented language modelInternational Journal of Machine Learning and Cybernetics10.1007/s13042-024-02190-815:10(4761-4786)Online publication date: 9-May-2024
https://doi.org/10.1007/s13042-024-02190-8
Ben JSun QLiu KYang XZhang F(2024)Multi-head multi-order graph attention networksApplied Intelligence10.1007/s10489-024-05601-z54:17-18(8092-8107)Online publication date: 20-Jun-2024
https://doi.org/10.1007/s10489-024-05601-z
Wang YHou WSheng NZhao ZLiu JHuang LWang J(2024)Graph pooling in graph neural networks: methods and their applications in omics studiesArtificial Intelligence Review10.1007/s10462-024-10918-957:11Online publication date: 16-Sep-2024
https://doi.org/10.1007/s10462-024-10918-9

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents