Skip to Main content Skip to Navigation
New interface
Conference papers

FlauBERT : des modèles de langue contextualisés pré-entraînés pour le français

Abstract : Language models have become a key step to achieve state-of-the art results in many NLP tasks. Leveraging the huge amount of unlabeled texts available, they provide an efficient way to pretrain continuous word representations that can be fine-tuned for downstream tasks, along with theircontextualization at the sentence level. This has been widely demonstrated for English. In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. We train models of different sizes using the new CNRS Jean Zay supercomputer. We apply our French language models to several NLP tasks (text classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that they often outperform other pre-training approaches on the FLUE benchmark also presented in this article.
Document type :
Conference papers
Complete list of metadata

Cited literature [52 references]  Display  Hide  Download
Contributor : Sylvain Pogodalla Connect in order to contact the contributor
Submitted on : Tuesday, June 23, 2020 - 12:09:12 PM
Last modification on : Friday, August 5, 2022 - 11:58:04 AM


Publisher files allowed on an open archive


  • HAL Id : hal-02784776, version 3


Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, et al.. FlauBERT : des modèles de langue contextualisés pré-entraînés pour le français. 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Volume 2 : Traitement Automatique des Langues Naturelles, Jun 2020, Nancy, France. pp.268-278. ⟨hal-02784776v3⟩



Record views


Files downloads