QuicK-means: Acceleration of K-means by learning a fast transform - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Machine Learning Année : 2021

QuicK-means: Acceleration of K-means by learning a fast transform

Luc Giffon
  • Fonction : Auteur
  • PersonId : 174103
  • IdHAL : luc-giffon
Valentin Emiya
Hachem Kadri

Résumé

K-means -- and the celebrated Lloyd algorithm -- is more than the clustering method it was originally designed to be. It has indeed proven pivotal to help increase the speed of many machine learning and data analysis techniques such as indexing, nearest-neighbor search and prediction, data compression, Radial Basis Function networks; its beneficial use has been shown to carry over to the acceleration of kernel machines (when using the Nyström method). Here, we propose a fast extension of K-means, dubbed QuicK-means, that rests on the idea of expressing the matrix of the $K$ centroids as a product of sparse matrices, a feat made possible by recent results devoted to find approximations of matrices as a product of sparse factors. Using such a decomposition squashes the complexity of the matrix-vector product between the factorized $K \times D$ centroid matrix $\mathbf{U}$ and any vector from $\mathcal{O}(K D)$ to $\mathcal{O}(A \log A+B)$, with $A=\min (K, D)$ and $B=\max (K, D)$, where $D$ is the dimension of the training data. This drastic computational saving has a direct impact in the assignment process of a point to a cluster, meaning that it is not only tangible at prediction time, but also at training time, provided the factorization procedure is performed during Lloyd's algorithm. We precisely show that resorting to a factorization step at each iteration does not impair the convergence of the optimization scheme and that, depending on the context, it may entail a reduction of the training time. Finally, we provide discussions and numerical simulations that show the versatility of our computationally-efficient QuicK-means algorithm.
Fichier principal
Vignette du fichier
main-stripped.pdf (906.49 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-02174845 , version 1 (26-08-2019)
hal-02174845 , version 2 (13-09-2019)
hal-02174845 , version 3 (27-11-2019)
hal-02174845 , version 4 (13-01-2021)

Identifiants

Citer

Luc Giffon, Valentin Emiya, Hachem Kadri, Liva Ralaivola. QuicK-means: Acceleration of K-means by learning a fast transform. Machine Learning, 2021, Machine Learning, 110, pp.881-905. ⟨10.1007/s10994-021-05965-0⟩. ⟨hal-02174845v4⟩
610 Consultations
359 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More