Minimizing Calibrated Loss using Stochastic Low-Rank Newton Descent for large scale image classification

Wafa Bel Haj Ali (1) , Michel Barlaud (1) , Richard Nock (2)

1 Laboratoire d'Informatique, Signaux, et Systèmes de Sophia-Antipolis (I3S) / Equipe IMAGES-CREATIVE
2 CEREGMIA - Centre de Recherche en Economie, Gestion, Modélisation et Informatique Appliquée

A standard approach for large scale image classification involves high dimensional features and Stochastic Gradient Descent algorithm (SGD) for the minimization of classical Hinge Loss in the primal space. Although complexity of Stochastic Gradient Descent is linear with the number of samples these method suffers from slow convergence. In order to cope with this issue, we propose here a Stochastic Low-Rank Newton Descent SLND for minimization of any calibrated loss in the primal space. SLND approximates the inverse Hessian by the best low-rank approximation according to squared Frobenius norm. We provide core optimization for fast convergence. Theoretically speaking, we show explicit convergence rates of the algorithm using these calibrated losses, which in addition provide working sets of parameters for experiments. Experiments are provided on the SUN, Caltech256 and ImageNet databases, with simple, uniform and efficient ways to tune remaining SLND parameters. On each of these databases, SLND challenges the accuracy of SGD with a speed of convergence faster by order of magnitude.

Mots clés: Machine Learning; Calibrated Loss; Newton Descent; Large Scale Image Classification

Domaines: Apprentissage [cs.LG]; Vision par ordinateur et reconnaissance de formes [cs.CV]

Wafa Bel Haj Ali, Michel Barlaud, Richard Nock. Minimizing Calibrated Loss using Stochastic Low-Rank Newton Descent for large scale image classification. 2013. ⟨hal-00825414⟩