Scalable Graph Building from Text Data

Thibault Debatty (1, 2) , Pietro Michiardi (1) , Olivier Thonnard (3) , Wim Mees (2)

1 Eurecom [Sophia Antipolis]
2 Ecole Royale Militaire
3 Symantec Research Labs

In this paper we propose NNCTPH, a new MapReduce algorithm that is able to build an approximate k-NN graph from large text datasets. The algorithm uses a modified version of Context Triggered Piecewise Hashing to bin the input data into buckets, and uses an exhaustive search inside the buckets to build the graph. It also uses multiple stages to join the different unconnected subgraphs. We experimentally test the algorithm on different datasets consisting of the subject of spam emails. Although the algorithm is still at an early development stage, it already proves to be four times faster than a MapReduce implementation of NN-Descent, for the same quality of produced graph.

Mots clés: k-NN graph; Graph building; MapReduce; Text data; Context-Triggered Piecewise Hashing

Domaines: Calcul parallèle, distribué et partagé [cs.DC]; Algorithme et structure de données [cs.DS]

Thibault Debatty, Pietro Michiardi, Olivier Thonnard, Wim Mees. Scalable Graph Building from Text Data. Proceedings of Machine Learning Research, 2014, Proceedings of the 3rd International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications, 36, pp.120-132. ⟨hal-01525743⟩