Transformer for Graphs: An Overview from Architecture Perspective
read the original abstract
Recently, Transformer model, which has achieved great success in many artificial intelligence fields, has demonstrated its great potential in modeling graph-structured data. Till now, a great variety of Transformers has been proposed to adapt to the graph-structured data. However, a comprehensive literature review and systematical evaluation of these Transformer variants for graphs are still unavailable. It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks. In this survey, we provide a comprehensive review of various Graph Transformer models from the architectural design perspective. We first disassemble the existing models and conclude three typical ways to incorporate the graph information into the vanilla Transformer: 1) GNNs as Auxiliary Modules, 2) Improved Positional Embedding from Graphs, and 3) Improved Attention Matrix from Graphs. Furthermore, we implement the representative components in three groups and conduct a comprehensive comparison on various kinds of famous graph data benchmarks to investigate the real performance gain of each component. Our experiments confirm the benefits of current graph-specific modules on Transformer and reveal their advantages on different kinds of graph tasks.
This paper has not been read by Pith yet.
Forward citations
Cited by 8 Pith papers
-
Communicability-Inspired Positional Encoding (CIPE)
CIPE constructs graph positional encodings from communicability so that self-attention similarities equal the sum of all-path contributions between nodes, yielding 35.5% average gains on seven benchmarks over structur...
-
Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation
GaRA generates task-specific LoRA weight updates conditioned on graph structures to enable better whole-graph encoding in LLMs for zero-shot graph learning.
-
BiScale-GTR: Fragment-Aware Graph Transformers for Multi-Scale Molecular Representation Learning
BiScale-GTR achieves claimed state-of-the-art results on MoleculeNet, PharmaBench and LRGB by combining improved fragment tokenization with a parallel GNN-Transformer architecture that operates at both atom and fragme...
-
RL-SPH: Learning to Achieve Feasible Solutions for Integer Linear Programs
RL-SPH is a reinforcement learning start primal heuristic that independently produces feasible solutions for ILPs with non-binary integers at 100% rate and with 28.6× lower primal gap than prior start heuristics.
-
Handling Feature Heterogeneity with Learnable Graph Patches
Learnable graph patches enable domain-agnostic pre-training of graph models by decomposing heterogeneous graphs into transferable semantic units via patch encoders and aggregators.
-
Attention-based graph neural networks: a survey
The survey groups attention-based GNNs into three stages—graph recurrent attention networks, graph attention networks, and graph transformers—while reviewing architectures and future directions.
-
Real-World Challenges in Fake News Detection: Dealing with Posts by Cold Users
Cold users dominate fake news datasets, and the User Evidence Network approximates their absent behavior data from existing user interactions to enable robust misinformation detection.
-
Retrieval-Augmented Generation with Graphs (GraphRAG)
A survey proposing a holistic GraphRAG framework with components including query processor, retriever, organizer, generator, and data source, plus domain-tailored reviews, challenges, and future directions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.