SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent

Antoine Bordes (1, 2) , Léon Bottou (1) , Patrick Gallinari (2)

1 NEC Laboratories America [Princeton]
2 MALIRE - Machine Learning and Information Retrieval

The SGD-QN algorithm is a stochastic gradient descent algorithm that makes careful use of second-order information and splits the parameter update into independently scheduled components. Thanks to this design, SGD-QN iterates nearly as fast as a first-order stochastic gradient descent but requires less iterations to achieve the same accuracy. This algorithm won the "Wild Track" of the first PASCAL Large Scale Learning Challenge (Sonnenburg et al., 2008).

Domaines: Apprentissage [cs.LG]

Antoine Bordes, Léon Bottou, Patrick Gallinari. SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent. Journal of Machine Learning Research, 2009, 10, pp.1737-1754. ⟨hal-00750911⟩