Skip to Main content Skip to Navigation
Conference papers

Enriching Language Models with Semantics

Tobias Mayer 1
1 WIMMICS - Web-Instrumented Man-Machine Interactions, Communities and Semantics
CRISAM - Inria Sophia Antipolis - Méditerranée , Laboratoire I3S - SPARKS - Scalable and Pervasive softwARe and Knowledge Systems
Abstract : Recent advances in language model (LM) pre-training from large-scale corpora have shown to improve various natural language processing tasks. They achieve performances comparable to non-expert humans on the GLUE benchmark for natural language understanding (NLU). While the improvement of the different contextualized representations comes from (i) the usage of more and more data, (ii) changing the types of lexical pre-training tasks or (iii) increasing the model size, NLU is more than memorizing word co-occurrences. But how much world knowledge and common sense can those language model capture? How much can those models infer from just the lexical information? To overcome this problem, some approaches include semantic information in the training process. In this paper, we highlight existing approaches to combine different types of semantics with language models during the pre-training or fine-tuning phase.
Complete list of metadata

Cited literature [9 references]  Display  Hide  Download
Contributor : Tobias Mayer Connect in order to contact the contributor
Submitted on : Tuesday, June 23, 2020 - 4:45:57 PM
Last modification on : Thursday, March 31, 2022 - 4:12:15 AM
Long-term archiving on: : Thursday, September 24, 2020 - 6:17:38 PM


Files produced by the author(s)


  • HAL Id : hal-02879286, version 1



Tobias Mayer. Enriching Language Models with Semantics. ECAI 2020 - 24th European Conference on Artificial Intelligence, Aug 2020, Santiago de Compostela / Online, Spain. ⟨hal-02879286⟩



Record views


Files downloads