Model-based Clustering with Missing Not At Random Data

Aude Sportisse (1, 2) , Matthieu Marbac (3, 4, 5, 6) , Fabien Laporte (7, 5, 8) , Gilles Celeux (9, 10) , Claire Boyer (11, 12, 13) , Julie Josse (14, 15) , Christophe Biernacki (16, 5, 17)

1 UniCA - Université Côte d'Azur
2 MAASAI - Modèles et algorithmes pour l’intelligence artificielle
3 UR - Université de Rennes
4 ENSAI - Ecole Nationale de la Statistique et de l'Analyse de l'Information [Bruz]
5 CNRS - Centre National de la Recherche Scientifique
6 CREST - Centre de Recherche en Economie et Statistique [Bruz]
7 Nantes Univ - Nantes Université
8 ITX-lab - ITX-lab unité de recherche de l'institut du thorax UMR1087 UMR6291
9 Université Paris-Saclay
10 CELESTE - Statistique mathématique et apprentissage
11 SU - Sorbonne Université
12 LPSM (UMR_8001) - Laboratoire de Probabilités, Statistique et Modélisation
13 MOKAPLAN - Méthodes numériques pour le problème de Monge-Kantorovich et Applications en sciences sociales
14 IDESP - Institut Desbrest de santé publique
15 PREMEDICAL - Médecine de précision par intégration de données et inférence causale
16 Université Lille Nord (France)
17 MODAL - MOdel for Data Analysis and Learning

Model-based unsupervised learning, as any learning task, stalls as soon as missing data occurs. This is even more true when the missing data are informative, or said missing not at random (MNAR). In this paper, we propose model-based clustering algorithms designed to handle very general types of missing data, including MNAR data. To do so, we introduce a mixture model for different types of data (continuous, count, categorical and mixed) to jointly model the data distribution and the MNAR mechanism, remaining vigilant to the relative degrees of freedom of each. Several MNAR models are discussed, for which the cause of the missingness can depend on both the values of the missing variable themselves and on the class membership. However, we focus on a specific MNAR model, called MNARz, for which the missingness only depends on the class membership. We first underline its ease of estimation, by showing that the statistical inference can be carried out on the data matrix concatenated with the missing mask considering finally a standard MAR mechanism. Consequently, we propose to perform clustering using the Expectation Maximization algorithm, specially developed for this simplified reinterpretation. Finally, we assess the numerical performances of the proposed methods on synthetic data and on the real medical registry TraumaBase® as well.

Mots clés: Model-based Clustering; Informative Missing Values; EM and Stochastic EM Algorithms; Medical Data

Domaines: Machine Learning [stat.ML]

Aude Sportisse, Matthieu Marbac, Fabien Laporte, Gilles Celeux, Claire Boyer, et al.. Model-based Clustering with Missing Not At Random Data. 2023. ⟨hal-03494674v4⟩