Multimodal Soft Nonnegative Matrix Co-Factorization for Convolutive Source Separation

Farnaz Sedighin (1) , Massoud Babaie-Zadeh (1) , Bertrand Rivet (2) , Christian Jutten (2)

1 EE department of SUT [Tehran] - Department of Electrical Engineering [Tehran]
2 GIPSA-VIBS - GIPSA - Vision and Brain Signal Processing

In this paper, the problem of convolutive source separation via multimodal soft Nonnegative Matrix Co-Factorization (NMCF) is addressed. Different aspects of a phenomenon may be recorded by sensors of different types (e.g., audio and video of human speech), and each of these recorded signals is called a modality. Since the underlying phenomenon of the modalities is the same, they have some similarities. Especially, they usually have similar time changes. It means that changes in one of them usually correspond to changes in the other one. So their active or inactive periods are usually similar. Assuming this similarity, it is expected that the activation coefficient matrices of their Nonnegative Matrix Factorization (NMF) have a similar form. In this paper, the similarity of the activation coefficient matrices between the modalities is considered for co-factorization. This similarity is used for separation procedure in a soft manner by using penalty terms. This results in more flexibility in the separation procedure. Simulation results and comparison with state-of-the-art algorithms show the effectiveness of the proposed algorithm.

Mots clés: multimodality; blind source separation; nonneg-ative matrix co-factorization; convolutive mixture; audiovisual speech separation.

Domaines: Traitement du signal et de l'image [eess.SP]

Farnaz Sedighin, Massoud Babaie-Zadeh, Bertrand Rivet, Christian Jutten. Multimodal Soft Nonnegative Matrix Co-Factorization for Convolutive Source Separation. IEEE Transactions on Signal Processing, 2017, 65 (12), pp.3179-3190. ⟨10.1109/TSP.2017.2679692⟩. ⟨hal-01514878⟩