An Optimal Transport Kernel for Feature Aggregation and its Relationship to Attention

Grégoire Mialon (1) , Dexiong Chen (1) , Alexandre d'Aspremont (2) , Julien Mairal (1)

1 Thoth - Apprentissage de modèles à partir de données massives
2 SIERRA - Statistical Machine Learning and Parsimony

We introduce a kernel for sets of features based on an optimal transport distance, along with an explicit embedding function. Our approach addresses the problem of feature aggregation, or pooling, for sets that exhibit long-range dependencies between their members. More precisely, our embedding aggregates the features of a given set according to the transport plan between the set and a reference shared across the data set. Unlike traditional hand-crafted kernels, our embedding can be optimized for a specific task or data set. It also has a natural connection to attention mechanisms in neural networks, which are commonly used to deal with sets, yet requires less data. Our embedding is particularly suited for biological sequence classification tasks and shows promising results for natural language sequences. We provide an implementation of our embedding that can be used alone or as a module in larger learning models. Our code is freely available at https://github.com/claying/OTK.

Domaines: Machine Learning [stat.ML]; Apprentissage [cs.LG]

Grégoire Mialon, Dexiong Chen, Alexandre d'Aspremont, Julien Mairal. An Optimal Transport Kernel for Feature Aggregation and its Relationship to Attention. 2020. ⟨hal-02883436v1⟩