CESAR: A new metric to measure the level of code-switching in corpora -Application to Maghrebian dialects

Karima Abidi (1) , Kamel Smaïli (1)

1 SMarT - Statistical Machine Translation and Speech Modelization and Text

In this paper, we are interested in a sociolinguistic phenomenon that occurs in daily conversations of Maghrebi people, commonly known as code-switching or also code-mixing. This problem consists of alternating languages during communication or writing. In this work, we measure the importance of this phenomenon in the Maghrebi languages. To this end, we harvested from YouTube, comments written in Algerian, Moroccan and Tunisian dialects. Each of which contains at least 17 million words. Although there are several metrics in the literature to measure the code-switching, to the best of our knowledge, there isn't yet a measure that takes into account the degree of mixture according to a reference language. In contrast to the existing measures, we propose a new metric named CESAR (CodE-Switching According to a Reference language) that estimates the degree of the language mixtures, in accordance with a reference language. Experiments are carried out on the three collected corpora by considering the local dialects as reference languages. Experimental results show that CESAR is well adapted to this purpose and allows to compare the three Maghrebi dialects according to their level of code-switching.. . .

Mots clés: Code-swtching; Maghrebian dialects; Reference language

Domaines: Informatique et langage [cs.CL]

Karima Abidi, Kamel Smaïli. CESAR: A new metric to measure the level of code-switching in corpora -Application to Maghrebian dialects. IntelliSys2021, Sep 2021, Amsterdam, Netherlands. ⟨hal-03139685⟩