Bilevel Learning of Deep Representations

Jordan Frecon (1) , Saverio Salzo (2) , Massimiliano Pontil (3, 2)

1 INSA Rouen Normandie - Institut national des sciences appliquées Rouen Normandie
2 IIT - Istituto Italiano di Tecnologia
3 UCL - University College of London [London]

We present a framework based on bilevel optimization for learning multilayer, deep data representations. While the lower-level problem implicitly defines the representation through the critical point of a learnable objective, the upper-level problem optimizes the representation mapping. We reformulate the problem via a majorizationminimization algorithm. On one hand, for some quadratic majorants, we show that the bilevel problem reduces down to the training of a feedforward neural network. On the other hand, for majorants based on Bregman distances, we introduce a new neural network architecture involving the inverse of the activation function. We argue theoretically that the novel architecture may have better mathematical properties than standard networks. Numerical experiments show that the proposed variant benefits from better training behaviors, resulting in more accurate prediction.

Domains: Machine Learning [cs.LG]; Optimization and Control [math.OC]; Machine Learning [stat.ML]

Jordan Frecon, Saverio Salzo, Massimiliano Pontil. Bilevel Learning of Deep Representations. 2021. ⟨hal-03132512v1⟩