On the detection of low-rank signal in the presence of spatially uncorrelated noise: a frequency domain approach

Alexis Rosuel (1, 2) , Philippe Loubaton (2) , Pascal Vallet (3) , Xavier Mestre (4)

1 Université Gustave Eiffel
2 LIGM - Laboratoire d'Informatique Gaspard-Monge
3 IMS - Laboratoire de l'intégration, du matériau au système
4 CTTC - Centre Tecnològic de Telecomunicacions de Catalunya = Telecommunications Technological Centre of Catalonia

This paper analyzes the detection of a M-dimensional useful signal modeled as the output of a M ×K MIMO filter driven by a K-dimensional white Gaussian noise, and corrupted by a M-dimensional Gaussian noise with mutually uncorrelated components. The study is focused on frequency domain test statistics based on the eigenvalues of an estimate of the spectral coherence matrix (SCM), obtained as a renormalization of the frequency-smoothed periodogram of the observed signal. If N denotes the sample size and B the smoothing span, it is proved that in the high-dimensional regime where M, B, N converge to infinity while K remains fixed, the SCM behaves as a certain correlated Wishart matrix. Exploiting well-known results on the behaviour of the eigenvalues of such matrices, it is deduced that the standard tests based on linear spectral statistics of the SCM fail to detect the presence of the useful signal in the high-dimensional regime. A new test based on the SCM, which is proved to be consistent, is also proposed, and its statistical performance is evaluated through numerical simulations.

Mots clés: spectral coherence matrix; periodogram; high-dimensional statistics; Random Matrix Theory; Detection Tests

Domaines: Théorie de l'information et codage [math.IT]

Alexis Rosuel, Philippe Loubaton, Pascal Vallet, Xavier Mestre. On the detection of low-rank signal in the presence of spatially uncorrelated noise: a frequency domain approach. IEEE Transactions on Signal Processing, 2021, 69, p. 4458-4473. ⟨hal-03268782⟩