IRISA at TRECVid 2017: Beyond Crossmodal and Multimodal Models for Video Hyperlinking

Mikail Demirdelen 1 Mateusz Budnik 1 Gabriel Sargent 1 Rémi Bois 1 Guillaume Gravier 1
1 LinkMedia - Creating and exploiting explicit links between multimedia fragments
Inria Rennes – Bretagne Atlantique , IRISA_D6 - MEDIA ET INTERACTIONS
Abstract : This paper presents the runs that were submitted to the TRECVid Challenge 2017 for the Video Hyperlinking task. The goal of the task is to propose a list of video segments, called targets, to complement query video segments defined as anchors. The data provided with the task encourage participants to make use of multiple modalities such as the audio track and the keyframes. In this context, we submitted four runs: 1) BiDNNFull uses a BiDNN model to combine ResNet with Word2Vec; 2) BiDNNFilter makes use of the same model and also exploits the metadata to narrow down the list of possible candidates; 3) BiDNNPinv tries to improve on the anchor keyframe fusion by using the Moore-Penrose pseudo-inverse and finally 4) noBiDNNPinv tests on the relevance of not using a BiDNN to fuse the modalities. Our runs were built based on a pre-trained model of ResNet as well as the transcripts and the metadata provided by the organizers of the task. The results show a gain in performance over the baseline BiDNN model both when the metadata filter was used and when the keyframe fusion was done with a pseudo-inverse.
Document type :
Conference papers
Liste complète des métadonnées

Cited literature [14 references]  Display  Hide  Download
https://hal.archives-ouvertes.fr/hal-01643232
Contributor : Guillaume Gravier <>
Submitted on : Tuesday, November 21, 2017 - 11:26:36 AM
Last modification on : Monday, April 1, 2019 - 12:45:51 PM

File

irisa-trecvid-lnk-2017.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01643232, version 1

Collections

UNIV-RENNES1 | INRIA | IRISA | INRIA2017 | UR1-UFR-ISTIC | UNIV-RENNES

Citation

Mikail Demirdelen, Mateusz Budnik, Gabriel Sargent, Rémi Bois, Guillaume Gravier. IRISA at TRECVid 2017: Beyond Crossmodal and Multimodal Models for Video Hyperlinking. Working Notes of the TRECVid 2017 Workshop, 2017, Gettysburg, United States. 〈hal-01643232〉

Export

BibTeX TEI DC DCterms EndNote

Share

Metrics

Record views

1083

Files downloads

80

Contact

support.ccsd.cnrs.fr
hal.support@ccsd.cnrs.fr
Logo CCSD