Ziv Lempel Compression of Huge Natural Language Data Tries Using Suffix Arrays

Strahil Ristov (1) , Eric Laporte (2)

1 Laboratory of research on stochastic signals and processes
2 LIGM - Laboratoire d'Informatique Gaspard-Monge

We present a very efficient, in terms of space and access speed, data structure for storing huge natural language data sets. The structure is described as LZ (Ziv Lempel) compressed linked list trie and is a step further beyond directed acyclic word graph in automata compression. We are using the structure to store DELAF, a huge French lexicon with syntactical, grammatical and lexical information associated with each word. The compressed structure can be produced in O(N) time using suffix trees for finding repetitions in trie, but for large data sets space requirements are more prohibitive than time so suffix arrays are used instead, with compression time complexity O(N log N) for all but for the largest data sets.

Mots clés: Compression; lexicon

Domaines: Informatique et langage [cs.CL]

Strahil Ristov, Eric Laporte. Ziv Lempel Compression of Huge Natural Language Data Tries Using Suffix Arrays. Ziv Lempel Compression of Huge Natural Language Data Tries Using Suffix Arrays, 1999, Warwick, United Kingdom. pp.196-211. ⟨hal-00189726⟩