Taming the curse of dimensionality for perturbed token identification

Olga Assainova (1) , Jérémy Rouot (2) , Ehsan Sedgh-Gooya (1)

1 LABISEN-LSL - Light Scatter Learning
2 LABISEN-VISION-AD - Vision et Analyse de Données

In the context of data tokenization, we model a token as a vector of a finite dimensional metric space E and given a finite subset of E, called the token set, we address the problem of deciding whether a given token is in a small neighborhood of an other token. We derive conditions to characterize the nearest token of a given one and show that these conditions are fulfilled asymptotically as the dimension of E tends to infinity. Whereas the classical nearest neighbor search is inefficient to solve such problem, we propose a new probabilistic algorithm, which becomes efficient if the dimension of E is large enough.

Mots clés: Nearest neighbor search; Big data; Algorithmic probability; Tokenization; Index Terms-Tokenization

Domaines: Intelligence artificielle [cs.AI]; Optimisation et contrôle [math.OC]

Olga Assainova, Jérémy Rouot, Ehsan Sedgh-Gooya. Taming the curse of dimensionality for perturbed token identification. 10th International Conference on Image Processing Theory, Tools and Applications, Nov 2020, Paris, France. ⟨hal-02865114v2⟩