Skip to Main content Skip to Navigation
Journal articles

Hierarchical Multi-Label Propagation using Speaking Face Graphs for Multimodal Person Discovery

Abstract : TV archives are growing in size so fast that manually indexing becomes unfeasible. Automatic indexing techniques can be applied to overcome this issue, and this work proposes an unsupervised technique for multimodal person discovery. To achieve this goal, we propose a hierarchical label propagation technique based on quasi-flat zones theory, that learns from labeled and unlabeled data and propagates names through a multimodal graph representation. In this representation, we combine audio, video, and text processing techniques to model the data as a graph of speaking faces. In the proposed mod-eling, we extract names via optical character recognition and propagate them through the graph using audiovisual relationships between speaking faces. We also use a random walk label propagation and two graph clustering strategies to serve as baselines. The proposed label propagation techniques always outper-form the clustering baselines on the quantitative assessments. Our approach also outperforms all literature methods tested on the same dataset except for one, which uses a different preprocessing step. The proposed hierarchical label propagation and the random walk baseline produce highly equivalent results according to the Kappa coefficient, but the hierarchical propagation is parameter-free and over 9 times faster than the random walk under the same configurations.
Complete list of metadata

Cited literature [76 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02926035
Contributor : Guillaume Gravier <>
Submitted on : Monday, August 31, 2020 - 12:09:28 PM
Last modification on : Wednesday, October 28, 2020 - 10:19:28 AM
Long-term archiving on: : Tuesday, December 1, 2020 - 12:21:01 PM

File

MTAP-MPD.pdf
Files produced by the author(s)

Identifiers

Citation

Gabriel Barbosa da Fonseca, Gabriel Sargent, Ronan Sicre, Zenilton Kleber Gonçalves Do Patrocinio, Guillaume Gravier, et al.. Hierarchical Multi-Label Propagation using Speaking Face Graphs for Multimodal Person Discovery. Multimedia Tools and Applications, Springer Verlag, In press, pp.1-27. ⟨10.1007/s11042-020-09692-x⟩. ⟨hal-02926035⟩

Share

Metrics

Record views

75

Files downloads

145