Nonlinear feedforward networks with stochastic outputs: infomax implies redundancy reduction.

J. P. Nadal (1) , N. Brunel (1) , N. Parga (2)

1 LPS - Laboratoire de Physique Statistique de l'ENS
2 Departamento de Fisica Teorica

We prove that maximization of mutual information between the output and the input of a feedforward neural network leads to full redundancy reduction under the following sufficient conditions: (i) the input signal is a (possibly nonlinear) invertible mixture of independent components; (ii) there is no input noise; (iii) the activity of each output neuron is a (possibly) stochastic variable with a probability distribution depending on the stimulus through a deterministic function of the inputs (where both the probability distributions and the functions can be different from neuron to neuron); (iv) optimization of the mutual information is performed over all these deterministic functions. This result extends that obtained by Nadal and Parga (1994) who considered the case of deterministic outputs.

Domaines: Systèmes désordonnés et réseaux de neurones [cond-mat.dis-nn]; Neurobiologie; Neurosciences

J. P. Nadal, N. Brunel, N. Parga. Nonlinear feedforward networks with stochastic outputs: infomax implies redundancy reduction.. Network, 1998, 9 (2), pp.207-17. ⟨10.1088/0954-898X/9/2/004⟩. ⟨hal-00143780⟩