Abstract
Knowledge Graph (KG) reasoning plays a crucial role in knowledge graph completion, as it involves the reasoning of unknown information based on the existing knowledge in the graph. Most current reasoning methods in reinforcement learning use a single-agent random walk. However, relying on a single agent is not sufficient, and training multi-agent to solve this problem is challenging. To overcome this obstacle, we propose an Adversary and Attention Guided Knowledge Graph Reasoning based on reinforcement learning framework (\({\textbf {A}}^2{\textbf {GKGR}}\)). Utilizing the Adversarially Guided Actor-Critic (AGAC) reinforcement learning architecture, we create an adversary for the agent that learns from the agent’s historical data. The agent gains the ability to discern its prediction region from that of its opponent by leveraging Kullback-Leibler (KL) divergence. This allows for a more extensive exploration of each path within the knowledge graph, ultimately enhancing the model’s effectiveness. At the same time, we add a self-attention mechanism to trim the action space, which solves the problem of large action space of knowledge graph and improves the effectiveness and efficiency of agent action selection. We performed experiments on multiple KG reasoning benchmarks, and the results show that our method achieves good performance and has good interpretability.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bordes, A., Usunier, N., Garcia-Duran, A., et al.: Translating embeddings for modeling multi-relational data. In: NeurIPS, pp. 2787–2795 (2013)
Chami, I., Wolf, A., Juan, D.C., et al.: Low-dimensional hyperbolic knowledge graph embeddings. In: ACL, pp. 6901–6914 (2020)
Cui, H., Peng, T., Han, R., et al.: Path-based multi-hop reasoning over knowledge graph for answering questions via adversarial reinforcement learning. Knowl. Based Syst. 276, 110760 (2023)
Das, R., Dhuliawala, S., Zaheer, M., et al.: Go for a walk and arrive at the answer: reasoning over paths in knowledge bases using reinforcement learning. In: ICLR (2018)
Dettmers, T., Minervini, P., Stenetorp, P., et al.: Convolutional 2D knowledge graph embeddings. In: AAAI, pp. 1811–1818 (2018)
Feng, J.: Knowledge graph embedding by translating on hyperplanes. In: AAAI, pp. 1112–1119 (2014)
Flet-Berliac, Y., Ferret, J., Pietquin, O., et al.: Adversarially guided actor-critic. In: ICLR (2021)
Jiang, C., Zhu, T., Zhou, H., et al.: Path spuriousness-aware reinforcement learning for multi-hop knowledge graph reasoning. In: EACL, pp. 3173–3184 (2023)
Lao, N., Mitchell, T., Cohen, W.: Random walk inference and learning in a large scale knowledge base. In: EMNLP, pp. 529–539 (2011)
Li, R., Cheng, X.: DIVINE: a generative adversarial imitation learning framework for knowledge graph reasoning. In: EMNLP, pp. 2642–2651 (2019)
Lin, X.V., Socher, R., Xiong, C.: Multi-hop knowledge graph reasoning with reward shaping. In: EMNLP, pp. 3243–3253 (2018)
Qi, P., Sun, Y., Luo, H.: Scratch-RL: a preference-driven adversarial reinforcement reasoning framework over knowledge graphs for explainable recommendation of scratch. Int. J. Intell. Syst. 37(10), 8113–8138 (2022)
Schulman, J., Moritz, P., Levine, S., et al.: High-dimensional continuous control using generalized advantage estimation. In: ICLR (2016)
Shang, B., Zhao, Y., Liu, Y., et al.: Attention-based exploitation and exploration strategy for multi-hop knowledge graph reasoning. Inf. Sci. 653, 119787 (2024)
Toutanova, K., Chen, D.Q., Pantel, P., et al.: Representing text for joint embedding of text and knowledge bases In: EMNLP, pp. 1499–1509 (2015)
Trouillon, T., Welbl, J., Riedel, S., et al.: Complex embeddings for simple link prediction. In: ICML, pp. 2071–2080 (2016)
Wang, H., Li, S., Pan, R., et al.: Incorporating graph attention mechanism into knowledge graph reasoning based on deep reinforcement learning. In: EMNLP, pp. 2623–2631 (2019)
Wang, Q., Hao, Y., Cao, J.: ADRL: an attention-based deep reinforcement learning framework for knowledge graph reasoning. Knowl. Based Syst. 197, 105910 (2020)
Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992)
Xiong, W.H., Hoang, T., Wang, W.Y.: DeepPath: a reinforcement learning method for knowledge graph reasoning. In: EMNLP, pp. 564–573 (2017)
Yang, B., Yih, W.T., He, X., et al.: Embedding entities and relations for learning and inference in knowledge bases. In: ICLR (2014)
Zhang, D., Yuan, Z., Liu, H., et al.: Learning to walk with dual agents for knowledge graph reasoning. In: AAAI, vol. 36, no. 5, pp. 5932–5941 (2022)
Zheng, M., Zhou, Y., Cui, Q.: Hierarchical policy network with multi-agent for knowledge graph reasoning based on reinforcement learning. In: KSEM, pp. 445–457 (2021)
Zhang, Y., Yao, Q.: Knowledge graph reasoning with relational digraph. In: WWW, pp. 912–924 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yu, Y. et al. (2024). Adversary and Attention Guided Knowledge Graph Reasoning Based on Reinforcement Learning. In: Cao, C., Chen, H., Zhao, L., Arshad, J., Asyhari, T., Wang, Y. (eds) Knowledge Science, Engineering and Management. KSEM 2024. Lecture Notes in Computer Science(), vol 14888. Springer, Singapore. https://doi.org/10.1007/978-981-97-5489-2_1
Download citation
DOI: https://doi.org/10.1007/978-981-97-5489-2_1
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5488-5
Online ISBN: 978-981-97-5489-2
eBook Packages: Computer ScienceComputer Science (R0)