Building k-nn graphs from large text data

Thibault Debatty (1, 2) , Pietro Michiardi (2) , Olivier Thonnard (3) , Wim Mees (1)

1 Royal Military Academy (RMA)
2 Eurecom [Sophia Antipolis]
3 Symantec Research Labs

In this paper we present our new design of NNCTPH, a scalable algorithm to build an approximate k-NN graph from large text datasets. The algorithm uses a modified version of Context Triggered Piecewise Hashing to bin the input data into buckets, and uses NN-Descent, a versatile graph-building algorithm , inside each bucket. We use datasets consisting of the subject of spam emails to experimentally test the influence of the different parameters of the algorithm on the number of computed similarities, on processing time, and on the quality of the final graph. We also compare the algorithm with a sequential and a MapRe-duce implementation of NN-Descent. For our datasets, the algorithm proved to be up to ten times faster than NN-Descent, for the same quality of produced graph. Moreover, the speedup increased with the size of the dataset, making NNCTPH a sensible choice for very large text datasets.

Domaines: Informatique [cs]

Thibault Debatty, Pietro Michiardi, Olivier Thonnard, Wim Mees. Building k-nn graphs from large text data. IEEE International Conference on Big Data (Big Data), 2014, Oct 2014, Washington DC, United States. ⟨10.1109/BigData.2014.7004276⟩. ⟨hal-01133324⟩