Audiovisual diarization of people in video content

Elie El Khoury (1, 2) , Christine Sénac (3) , Philippe Joly (3)

1 IDIAP Research Institute
2 LIUM - Laboratoire d'Informatique de l'Université du Maine
3 IRIT-SAMoVA - Équipe Structuration, Analyse et MOdélisation de documents Vidéo et Audio

Audio-Visual People Diarization (AVPD) is an original framework that simultaneously improves audio, video, and audiovisual diarization results. Following a literature review of people diarization for both audio and video content and their limitations, which includes our own contributions, we describe a proposed method for associating both audio and video information by using co-occurrence matrices and present experiments which were conducted on a corpus containing TV news, TV debates, and movies. Results show the effectiveness of the overall diarization system and confirm the gains audio information can bring to video indexing and vice versa.

Mots clés: People diarization; Segmentation; Unsupervised clustering; Audiovisual fusion; Video indexing

Domaines: Informatique [cs]

Elie El Khoury, Christine Sénac, Philippe Joly. Audiovisual diarization of people in video content. Multimedia Tools and Applications, 2012, 68, pp.747--775. ⟨10.1007/s11042-012-1080-6⟩. ⟨hal-03220748⟩