A Generalization of Transformer Networks to Graphs
read the original abstract
We propose a generalization of transformer neural network architecture for arbitrary graphs. The original transformer was designed for Natural Language Processing (NLP), which operates on fully connected graphs representing all connections between the words in a sequence. Such architecture does not leverage the graph connectivity inductive bias, and can perform poorly when the graph topology is important and has not been encoded into the node features. We introduce a graph transformer with four new properties compared to the standard model. First, the attention mechanism is a function of the neighborhood connectivity for each node in the graph. Second, the positional encoding is represented by the Laplacian eigenvectors, which naturally generalize the sinusoidal positional encodings often used in NLP. Third, the layer normalization is replaced by a batch normalization layer, which provides faster training and better generalization performance. Finally, the architecture is extended to edge feature representation, which can be critical to tasks s.a. chemistry (bond type) or link prediction (entity relationship in knowledge graphs). Numerical experiments on a graph benchmark demonstrate the performance of the proposed graph transformer architecture. This work closes the gap between the original transformer, which was designed for the limited case of line graphs, and graph neural networks, that can work with arbitrary graphs. As our architecture is simple and generic, we believe it can be used as a black box for future applications that wish to consider transformer and graphs.
This paper has not been read by Pith yet.
Forward citations
Cited by 35 Pith papers
-
Spatial Support Matters: Geometry-Aware Graph Fusion for Rainfall Field Reconstruction
A geometry-aware multi-support heterogeneous GNN fuses point, line, and grid rainfall observations via cross-support message passing to reconstruct fields at arbitrary resolutions, reducing RMSE by 23.2% over baseline...
-
Communicability-Inspired Positional Encoding (CIPE)
CIPE constructs graph positional encodings from communicability so that self-attention similarities equal the sum of all-path contributions between nodes, yielding 35.5% average gains on seven benchmarks over structur...
-
One Generator, Any Process: LLM-Conditioning for the LHC
LLM embeddings condition generative networks for LHC events, yielding faster convergence, higher quality, and generalization to unseen processes.
-
Spatiotemporal Graph Transformer for 3D Neighborhood Interaction and Quality Prediction in Metal Additive Manufacturing
A dual-attention graph transformer on a weighted network representation of fusing locations models cross-layer interactions to improve quality prediction in metal additive manufacturing over image, sequence, and graph...
-
TRAPS: Therapeutic Response Analysis via Pathway-informed Stratification
Benchmark of BINN, GraphPath, and PATH on 2622 TCGA patients shows PATH best for targeted therapy, BINN for survival, none useful for radiation, with GraphPath at 0.92 AUROC on prostate targeted therapy.
-
Beyond Oversquashing: Understanding Signal Propagation in GNNs Via Observables
Quantum-inspired observables reveal poor signal routing in standard spectral GNNs and motivate Schrödinger GNNs with superior propagation capacity.
-
Rethinking Positional Encoding for Neural Vehicle Routing
A hierarchical anisometric positional encoding that combines distance-indexed in-route and depot-anchored angular cross-route components improves transformer-based solvers for vehicle routing problems over index-based...
-
Continual Learning for fMRI-Based Brain Disorder Diagnosis via Functional Connectivity Matrices Generative Replay
A structure-aware VAE generates realistic FC matrices for replay, combined with multi-level knowledge distillation and hierarchical contextual bandit sampling, to enable continual fMRI-based brain disorder diagnosis a...
-
How Attentive are Graph Attention Networks?
GAT uses static attention where neighbor rankings ignore the query node and thus cannot express some graph problems; GATv2 enables dynamic attention and outperforms GAT on 11 OGB and other benchmarks with equal parameters.
-
One Generator, Any Process: LLM-Conditioning for the LHC
LLM embeddings condition a generative transformer to enable faster convergence, better performance, and generalization to unseen LHC processes using a single model.
-
Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation
GaRA generates task-specific LoRA weight updates conditioned on graph structures to enable better whole-graph encoding in LLMs for zero-shot graph learning.
-
FLAGG: Flexible Autoregressive Graph Generation
FLAGG makes one-shot graph generators autoregressive through a learned reversal of stochastic node removal and outperforms baselines on datasets spanning different graph sizes and domains.
-
Is an Image Also Worth 16x16=256 Superpixels? A Framework for Attentional Image Classification
SPT generalizes SICGAT and ViT to arbitrary superpixel chunking with multidimensional sine-cosine encodings and shape/color-enriched patches, outperforming prior GNN superpixel methods while matching ViTs on CIFAR10, ...
-
Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?
Deep ensembles fail to capture meaningful epistemic uncertainty in message-passing GNNs due to epistemic collapse where independently trained networks converge to similar predictions.
-
Neural Point-Forms
Neural point-forms are introduced as permutation-invariant neural layers that output learned form-comparison matrices for point clouds, with a claimed consistency proof under sampling and manifold assumptions and comp...
-
Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinement
Crys-JEPA introduces a joint embedding predictive architecture that creates an energy-aware latent space, enabling embedding-based stability screening and a refinement pipeline that yields up to 72.7% gains on the V.S...
-
Mask-Morph Graph U-Net: A Generalisable Mesh-Based Surrogate for Crashworthiness Field Prediction under Large Geometric Variation
Mask-Morph Graph U-Net morphs coarse graph hierarchies with barycentric parameterization and applies masked supervised pretraining to improve generalizability of hierarchical GNN surrogates for crashworthiness predict...
-
Mask-Morph Graph U-Net: A Generalisable Mesh-Based Surrogate for Crashworthiness Field Prediction under Large Geometric Variation
MMGUNet morphs coarse graph hierarchies with feature-aligned barycentric mapping and uses masked pretraining plus frozen edge layers to improve generalisability of mesh surrogates for crashworthiness prediction under ...
-
Invariant-Based Diagnostics for Graph Benchmarks
Graph invariants serve as expressive, task-agnostic baselines that characterize structural heterogeneity and match trained models across 26 datasets, indicating that expressivity is not the primary driver of performance.
-
Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
GTokenLLMs do not fully understand graph tokens, exhibiting over-sensitivity or insensitivity to instruction changes and relying heavily on text for reasoning even when graph information is preserved.
-
Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs
A new distributed framework for graph transformer training auto-selects parallel strategies and optimizes sparse operations to deliver up to 6x speedup on 8 GPUs and 78% memory reduction.
-
Graph Transformer-Based Pathway Embedding for Cancer Prognosis
PATH gene embeddings in a graph transformer achieve 0.8766 F1 on pancancer metastasis prediction (8.8% over SOTA) and identify disease-state pathway rewiring.
-
A Structure-Preserving Graph Neural Solver for Parametric Hyperbolic Conservation Laws
A structure-preserving GNN solver for parametric hyperbolic conservation laws achieves superior long-horizon stability and orders-of-magnitude speedups over high-resolution simulations on supersonic flow benchmarks.
-
Invariant-Stratified Propagation for Expressive Graph Neural Networks
Invariant-Stratified Propagation (ISP) enhances GNN expressivity beyond 1-WL by stratifying nodes according to graph invariants and encoding structural heterogeneity in hierarchical strata.
-
A Hierarchical Quantized Tokenization Framework for Task-Adaptive Graph Representation Learning
A hierarchical quantized tokenization framework with task-adaptive routing and dual-view streams for improved graph representation learning on node classification and link prediction.
-
Detecting LLM-Generated Spam Reviews by Integrating Language Model Embeddings and Graph Neural Network
Introduces FraudSquad, a hybrid model using language model embeddings and a gated graph transformer that outperforms baselines on newly created LLM-generated spam review datasets.
-
Feature Augmentation of GNNs for ILPs: Local Uniqueness Suffices
Local d-hop uniqueness in GNN node features matches global UID expressiveness for ILP solving while providing stronger generalization.
-
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
Geometric deep learning provides a unified mathematical framework based on grids, groups, graphs, geodesics, and gauges to explain and extend neural network architectures by incorporating physical regularities.
-
Graph Grounded Cross Attention Transformer Neural Network for Structurally Constrained Full Event Sequence Generation in Predictive Process Monitoring
GGATN combines graph grounding with transformer self- and cross-attention to generate full event sequences, timestamps, length, and attributes in a single pass followed by Viterbi-style constrained decoding, outperfor...
-
Handling Feature Heterogeneity with Learnable Graph Patches
Learnable graph patches enable domain-agnostic pre-training of graph models by decomposing heterogeneous graphs into transferable semantic units via patch encoders and aggregators.
-
Modeling Spectral Energy Shifts in Spatio-Temporal Graph Anomaly Detection
Introduces a node-level spectral energy formulation and energy-aware message passing framework to detect camouflaged anomalies with decreased spectral variation in static and time-series graphs.
-
Composable Crystals: Controllable Materials Discovery via Concept Learning
VQ-VAE concept learning enables controllable recombination of crystal motifs to generate structures with reported gains in validity-stability-uniqueness-novelty metrics on MP-20 and Alex-MP-20.
-
Rethinking Generalization in Graph Neural Networks: A Structural Complexity Perspective
GNN generalization depends explicitly on graph structural complexity measured by effective edges, with a new regularization method shown to balance underfitting and overfitting.
-
Attention-based graph neural networks: a survey
The survey groups attention-based GNNs into three stages—graph recurrent attention networks, graph attention networks, and graph transformers—while reviewing architectures and future directions.
-
Explaining the Explainers in Graph Neural Networks: a Comparative Study
Benchmark study of ten GNN explainers on eight architectures and six datasets that isolates usable components and issues practical recommendations.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.