SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set

Abstract : This paper presents an augmentation of MSCOCO dataset where speech is added to image and text. Speech captions are generated using text-to-speech (TTS) synthesis resulting in 616,767 spoken captions (more than 600h) paired with images. Disfluencies and speed perturbation are added to the signal in order to sound more natural. Each speech signal (WAV) is paired with a JSON file containing exact timecode for each word/syllable/phoneme in the spoken caption. Such a corpus could be used for Language and Vision (LaVi) tasks including speech input or output instead of text. Investigating multimodal learning schemes for unsupervised speech pattern discovery is also possible with this corpus, as demonstrated by a preliminary study conducted on a subset of the corpus (10h, 10k spoken captions).
Type de document :
Communication dans un congrès
Grounding Language Understanding GLU2017 Workshop (Satellite of Interspeech 2017), Aug 2017, Stockholm, Sweden. Grounding Language Understanding GLU2017 Workshop (Satellite of Interspeech 2017), 〈http://www.speech.kth.se/glu2017/ 〉
Liste complète des métadonnées

Littérature citée [20 références]  Voir  Masquer  Télécharger

https://hal.archives-ouvertes.fr/hal-01580879
Contributeur : Laurent Besacier <>
Soumis le : dimanche 3 septembre 2017 - 12:01:39
Dernière modification le : jeudi 11 octobre 2018 - 08:48:03
Document(s) archivé(s) le : lundi 11 décembre 2017 - 18:00:16

Fichier

GLU2017.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01580879, version 1

Collections

Citation

William Havard, Laurent Besacier, Olivier Rosec. SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set. Grounding Language Understanding GLU2017 Workshop (Satellite of Interspeech 2017), Aug 2017, Stockholm, Sweden. Grounding Language Understanding GLU2017 Workshop (Satellite of Interspeech 2017), 〈http://www.speech.kth.se/glu2017/ 〉. 〈hal-01580879〉

Partager

Métriques

Consultations de la notice

173

Téléchargements de fichiers

78