A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention

Résumé

We address the problem of learning on sets of features, motivated by the need of performing pooling operations in long biological sequences of varying sizes, with long-range dependencies, and possibly few labeled data. To address this challenging task, we introduce a parametrized representation of fixed size, which embeds and then aggregates elements from a given input set according to the optimal transport plan between the set and a trainable reference. Our approach scales to large datasets and allows end-to-end training of the reference, while also providing a simple unsupervised learning mechanism with small computational cost. Our aggregation technique admits two useful interpretations: it may be seen as a mechanism related to attention layers in neural networks, or it may be seen as a scalable surrogate of a classical optimal transport-based kernel. We experimentally demonstrate the effectiveness of our approach on biological sequences, achieving state-of-the-art results for protein fold recognition and detection of chromatin profiles tasks, and, as a proof of concept, we show promising results for processing natural language sequences. We provide an open-source implementation of our embedding that can be used alone or as a module in larger learning models at https://github.com/claying/OTK.
Fichier principal
Vignette du fichier
main_iclr.pdf (1.21 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-02883436 , version 1 (29-06-2020)
hal-02883436 , version 2 (05-10-2020)
hal-02883436 , version 3 (09-02-2021)

Identifiants

  • HAL Id : hal-02883436 , version 3

Citer

Grégoire Mialon, Dexiong Chen, Alexandre d'Aspremont, Julien Mairal. A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention. ICLR 2021 - The Ninth International Conference on Learning Representations, May 2021, Virtual, France. ⟨hal-02883436v3⟩
6459 Consultations
687 Téléchargements

Partager

Gmail Facebook X LinkedIn More