KD-means: clustering method for massive data based on kd-tree

Nabil El Malki (1, 2) , Franck Ravat (1, 3) , Olivier Teste (1, 4)

1 IRIT-SIG - Systèmes d’Informations Généralisées
2 Capgemini [Toulouse]
3 UT Capitole - Université Toulouse Capitole
4 UT2J - Université Toulouse - Jean Jaurès

K-means clustering is a popular unsupervised classification algorithm employed in several domains, e.g., imaging, segmentation, or compression. Nevertheless, the number of clusters k, fixed apriori, affects mainly the clustering quality. Current State-of-the-art k-means implementations could automatically set of the number of clusters. However, they result in unreasonable processing time while classifying large volumes of data. In this paper, we propose a novel solution based on kd-tree to determine the number of cluster k in the context of massive data for preprocessing data science projects or in near-real-time applications. We demonstrate how our solution outperforms current solutions in terms of clustering quality, and processing time on massive data.

Domaines: Base de données [cs.DB]

Nabil El Malki, Franck Ravat, Olivier Teste. KD-means: clustering method for massive data based on kd-tree. 22nd International Workshop On Design, Optimization, Languages and Analytical Processing of Big Data - DOLAP 2020 -, Mar 2020, Copenhagen, Denmark. ⟨hal-03080514⟩