An Unsupervised Morphological Criterion for Discriminating Similar Languages

Adrien Barbaresi (1, 2)

1 BBAW - Berlin-Brandenburgische Akademie der Wissenschaften
2 OeAW - Austrian Academy of Sciences

In this study conducted on the occasion of the Discriminating between Similar Languages shared task, I introduce an additional decision factor focusing on the token and subtoken level. The motivation behind this submission is to test whether a morphologically-informed criterion can add linguistically relevant information to global categorization and thus improve performance. The contributions of this paper are (1) a description of the unsupervised, low-resource method; (2) an evaluation and analysis of its raw performance; and (3) an assessment of its impact within a model comprising common indicators used in language identification. I present and discuss the systems used in the task A, a 12-way language identification task comprising varieties of five main language groups. Additionally I introduce a new off-the-shelf Naive Bayes classifier using a contrastive word and subword n-gram model ("Bayesline") which outperforms the best submissions.

Mots clés: Language identification; Language detection; Morphological analysis; Dialectal differences

Domaines: Linguistique; Informatique et langage [cs.CL]

Adrien Barbaresi. An Unsupervised Morphological Criterion for Discriminating Similar Languages. 3rd Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2016), Dec 2016, Osaka, Japan. pp.212-220. ⟨hal-01575653⟩