Slimming down a high-dimensional binary datatable: relevant eigen-subspace and substantial content

Alain Lelu (1, 2)

1 KIWI - Knowledge Information and Web Intelligence
2 LASELDI - Laboratoire de Semio-Linguistique, Didactique et Informatique

Determining the number of relevant dimensions in the eigen-space of a data matrix is a central issue in many data-mining applications. We tackle here the sub-problem of finding the ''right'' dimensionality of a type of data matrices often encountered in the domains of text or usage mining: large, sparse, high-dimensional binary datatables. We present here the application of a randomization test to this problem. We validate our approach first on artificial datasets, then on a real documentary data collection, i.e. 1900 documents described in a 3600 keywords dataspace, where the actual, intrinsic dimension appears to be 28 times less than the number of keywords - an important information when preparing to cluster or discriminate such data. We also present preliminary results on the problem of clearing the datatable from non-essential information bits.

Mots clés: randomization test; dimensionality reduction; data reconstitution; power-law distribution

Domaines: Méthodologie [stat.ME]

Alain Lelu. Slimming down a high-dimensional binary datatable: relevant eigen-subspace and substantial content. 19th International Conference on Computational Statistics - COMPSTAT 2010, Aug 2010, Paris, France. pp.1271-1278. ⟨hal-00462464⟩