Skip to Main content Skip to Navigation
Conference papers

On the Usability of Transformers-based models for a French Question-Answering task

Oralie Cattan 1, 2 Christophe Servan 1 Sophie Rosset 2 
2 ILES - Information, Langue Ecrite et Signée
LISN - Laboratoire Interdisciplinaire des Sciences du Numérique, STL - Sciences et Technologies des Langues
Abstract : For many tasks, state-of-the-art results have been achieved with Transformer-based architectures, resulting in a paradigmatic shift in practices from the use of task-specific architectures to the fine-tuning of pre-trained language models. The ongoing trend consists in training models with an ever-increasing amount of data and parameters, which requires considerable resources. It leads to a strong search to improve resource efficiency based on algorithmic and hardware improvements evaluated only for English. This raises questions about their usability when applied to small-scale learning problems, for which a limited amount of training data is available, especially for underresourced languages tasks. The lack of appropriately sized corpora is a hindrance to applying data-driven and transfer learning-based approaches with strong instability cases. In this paper, we establish a state-of-the-art of the efforts dedicated to the usability of Transformerbased models and propose to evaluate these improvements on the question-answering performances of French language which have few resources. We address the instability relating to data scarcity by investigating various training strategies with data augmentation, hyperparameters optimization and cross-lingual transfer. We also introduce a new compact model for French FrALBERT which proves to be competitive in low-resource settings.
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03336060
Contributor : oralie cattan Connect in order to contact the contributor
Submitted on : Monday, September 6, 2021 - 5:40:07 PM
Last modification on : Friday, August 5, 2022 - 9:27:30 AM
Long-term archiving on: : Tuesday, December 7, 2021 - 7:24:06 PM

File

RANLP_2021_transformers_usabil...
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03336060, version 1

Citation

Oralie Cattan, Christophe Servan, Sophie Rosset. On the Usability of Transformers-based models for a French Question-Answering task. Recent Advances in Natural Language Processing (RANLP), Sep 2021, Varna, Bulgaria. ⟨hal-03336060⟩

Share

Metrics

Record views

255

Files downloads

280