Skip to Main content Skip to Navigation
Journal articles

Riemannian metrics for neural networks II: Recurrent networks and learning symbolic data sequences

Yann Ollivier 1, 2
2 TAO - Machine Learning and Optimisation
CNRS - Centre National de la Recherche Scientifique : UMR8623, Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, LRI - Laboratoire de Recherche en Informatique
Abstract : Recurrent neural networks are powerful models for sequential data, able to represent complex dependencies in the sequence that simpler models such as hidden Markov models cannot handle. Yet they are notoriously hard to train. Here we introduce a training procedure using a gradient ascent in a Riemannian metric: this produces an algorithm independent from design choices such as the encoding of parameters and unit activities. This metric gradient ascent is designed to have an algorithmic cost close to backpropagation through time for sparsely connected networks. We use this procedure on \emph{gated leaky neural networks} (GLNNs), a variant of recurrent neural networks with an architecture inspired by finite automata and an evolution equation inspired by continuous-time networks. GLNNs trained with a Riemannian gradient are demonstrated to effectively capture a variety of structures in synthetic problems: basic block nesting as in context-free grammars (an important feature of natural languages, but difficult to learn), intersections of multiple independent Markov-type relations, or long-distance relationships such as the distant-XOR problem. This method does not require adjusting the network structure or initial parameters: the network used is a sparse random graph and the initialization is identical for all problems considered.
Complete list of metadatas
Contributor : Yann Ollivier <>
Submitted on : Wednesday, September 4, 2013 - 1:32:37 PM
Last modification on : Tuesday, April 21, 2020 - 1:09:49 AM

Links full text



Yann Ollivier. Riemannian metrics for neural networks II: Recurrent networks and learning symbolic data sequences. Information and Inference, Oxford University Press (OUP), 2015, 4 (2), pp.154-193. ⟨10.1093/imaiai/iav007⟩. ⟨hal-00857980⟩



Record views