HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

GraphiT: Encoding Graph Structure in Transformers

Grégoire Mialon 1, 2 Dexiong Chen 2 Margot Selosse 2 Julien Mairal 2
1 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique - ENS Paris, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
2 Thoth - Apprentissage de modèles à partir de données massives
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann
Abstract : We show that viewing graphs as sets of node features and incorporating structural and positional information into a transformer architecture is able to outperform representations learned with classical graph neural networks (GNNs). Our model, GraphiT, encodes such information by (i) leveraging relative positional encoding strategies in self-attention scores based on positive definite kernels on graphs, and (ii) enumerating and encoding local sub-structures such as paths of short length. We thoroughly evaluate these two ideas on many classification and regression tasks, demonstrating the effectiveness of each of them independently, as well as their combination. In addition to performing well on standard benchmarks, our model also admits natural visualization mechanisms for interpreting graph motifs explaining the predictions, making it a potentially strong candidate for scientific applications where interpretation is important.
Document type :
Preprints, Working Papers, ...
Complete list of metadata

Contributor : Grégoire Mialon Connect in order to contact the contributor
Submitted on : Thursday, June 10, 2021 - 2:04:36 PM
Last modification on : Thursday, March 17, 2022 - 10:08:51 AM
Long-term archiving on: : Saturday, September 11, 2021 - 6:52:51 PM


Files produced by the author(s)


  • HAL Id : hal-03256708, version 1



Grégoire Mialon, Dexiong Chen, Margot Selosse, Julien Mairal. GraphiT: Encoding Graph Structure in Transformers. 2021. ⟨hal-03256708⟩



Record views


Files downloads