Multimodal Data Fusion: An Overview of Methods, Challenges and Prospects

Dana Lahat (1) , Tülay Adali (2) , Christian Jutten (1)

1 GIPSA-VIBS - GIPSA - Vision and Brain Signal Processing
2 MLSP lab - Machine Learning for Signal Processing Laboratory

In various disciplines, information about the same phenomenon can be acquired from different types of detectors, at different conditions, in multiple experiments or subjects, among others. We use the term "modality" for each such acquisition framework. Due to the rich characteristics of natural phenomena, it is rare that a single modality provides complete knowledge of the phenomenon of interest. The increasing availability of several modalities reporting on the same system introduces new degrees of freedom, which raise questions beyond those related to exploiting each modality separately. As we argue, many of these questions, or "challenges" , are common to multiple domains. This paper deals with two key questions: "why we need data fusion" and "how we perform it". The first question is motivated by numerous examples in science and technology, followed by a mathematical framework that showcases some of the benefits that data fusion provides. In order to address the second question, "diversity" is introduced as a key concept, and a number of data-driven solutions based on matrix and tensor decompositions are discussed, emphasizing how they account for diversity across the datasets. The aim of this paper is to provide the reader, regardless of his or her community of origin, with a taste of the vastness of the field, the prospects and opportunities that it holds.

Mots clés: data fusion; multimodality; multiset data analysis; latent variables; tensor; overview

Domaines: Traitement du signal et de l'image [eess.SP]

Dana Lahat, Tülay Adali, Christian Jutten. Multimodal Data Fusion: An Overview of Methods, Challenges and Prospects. Proceedings of the IEEE, 2015, Multimodal Data Fusion, 103 (9), pp.1449-1477. ⟨10.1109/JPROC.2015.2460697⟩. ⟨hal-01179853⟩