Dimension-free Concentration Bounds on Hankel Matrices for Spectral Learning

Abstract : Learning probabilistic models over strings is an important issue for many applications. Spectral methods propose elegant solutions to the problem of inferring weighted automata from finite samples of variable-length strings drawn from an unknown target distribution $p$. These methods rely on a singular value decomposition of a matrix $\v{H}_S$, called the empirical Hankel matrix, that records the frequencies of (some of) the observed strings $S$. The accuracy of the learned distribution depends both on the quantity of information embedded in $\v{H}_S$ and on the distance between $\v{H}_S$ and its mean $\v{H}_p$. Existing concentration bounds seem to indicate that the concentration over $\v{H}_p$ gets looser with its dimensions, suggesting that it might be necessary to bound the dimensions of $\v{H}_S$ for learning. We prove new dimension-free concentration bounds for classical Hankel matrices and several variants, based on prefixes or factors of strings, that are useful for learning. Experiments demonstrate that these bounds are tight and that they significantly improve existing (dimension-dependent) bounds. One consequence of these results is that the spectral learning approach remains consistent even if all the observations are recorded within the empirical matrix.
Document type :
Journal articles
Complete list of metadatas

Cited literature [28 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01306915
Contributor : Amaury Habrard <>
Submitted on : Monday, April 25, 2016 - 7:20:35 PM
Last modification on : Thursday, June 27, 2019 - 1:36:06 PM
Long-term archiving on : Tuesday, November 15, 2016 - 11:46:24 AM

File

14-501.pdf
Publisher files allowed on an open archive

Link: http://jmlr.org/papers/v17/14-501.html

Identifiers

  • HAL Id : hal-01306915, version 1

Citation

François Denis, Mattias Gybels, Amaury Habrard. Dimension-free Concentration Bounds on Hankel Matrices for Spectral Learning. Journal of Machine Learning Research, Microtome Publishing, 2016, 17 (31), pp.1-32. ⟨http://jmlr.org/papers/v17/14-501.html⟩. ⟨hal-01306915⟩

Share

Metrics

Record views

289

Files downloads

43