Skip to Main content Skip to Navigation

Determining the k in k-means with MapReduce

Abstract : In this paper we propose a MapReduce implementation of G-means, a variant of k-means that is able to automatically determine k, the number of clusters. We show that our implementation scales to very large datasets and very large values of k, as the computation cost is proportional to nk. Other techniques that run a clustering algorithm with different values of k and choose the value of k that provides the " best " results have a computation cost that is proportional to nk 2. We run experiments that confirm that the processing time is proportional to k. These experiments also show that, because G-means adds new centers progressively, if and where they are needed, it reduces the probability to fall into a local minimum, and finally finds better centers than classical k-means processing.
Complete list of metadatas

Cited literature [22 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01525708
Contributor : Thibault Debatty <>
Submitted on : Tuesday, May 30, 2017 - 4:59:12 PM
Last modification on : Friday, June 14, 2019 - 3:36:02 PM
Document(s) archivé(s) le : Wednesday, September 6, 2017 - 10:00:58 AM

File

mrandbeyond2013_debatty.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01525708, version 1

Collections

Citation

Thibault Debatty, Pietro Michiardi, Wim Mees, Olivier Thonnard. Determining the k in k-means with MapReduce. EDBT/ICDT 2014 Joint Conference, Mar 2014, Athènes, Greece. ⟨hal-01525708⟩

Share

Metrics

Record views

133

Files downloads

809