Skip to Main content Skip to Navigation
New interface
Conference papers

ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks

Abstract : We aim at improving spoken language modeling (LM) using very large amount of automatically transcribed speech. We leverage the INA (French National Audiovisual Institute 1) collection and obtain 19GB of text after applying ASR on 350,000 hours of diverse TV shows. From this, spoken language models are trained either by fine-tuning an existing LM (FlauBERT 2) or through training a LM from scratch. New models (FlauBERT-Oral) are shared with the community and evaluated for 3 downstream tasks: spoken language understanding, classification of TV shows and speech syntactic parsing. Results show that FlauBERT-Oral can be beneficial compared to its initial FlauBERT version demonstrating that, despite its inherent noisy nature, ASR-generated text can be used to build spoken language models.
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03770506
Contributor : Valentin Pelloin Connect in order to contact the contributor
Submitted on : Tuesday, September 6, 2022 - 3:07:15 PM
Last modification on : Friday, October 7, 2022 - 4:37:07 AM

File

2207.01893.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03770506, version 1
  • ARXIV : 2207.01893

Citation

Valentin Pelloin, Franck Dary, Nicolas Hervé, Benoît Favre, Nathalie Camelin, et al.. ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks. Interspeech 2022, Sep 2022, Incheon, South Korea. ⟨hal-03770506⟩

Share

Metrics

Record views

20

Files downloads

7