Unsupervised Creation of Normalisation Dictionaries for Micro-Blogs in Arabic, French and English

Abstract : Text normalisation is a necessity to correct and make more sense of the micro-blogs messages, for information retrieval purposes. Unfortunately, tools and resources of text normalisation are rarely shared. In this paper, an approach is presented based on an unsupervised method for text normalisation using distributed representations of words, known also as "word embedding", applied on Arabic, French and English Languages. In addition, a tool will be supplied to create dictionaries for micro-blogs normalisation, in a form of pairs of misspelled word with its standard-form word, in the languages: Arabic, French and English. The tool will be available as open source including the resources: word embedding's models (with vocabulary size of million words for Arabic language model, million words for English language model and thousand words for French language model), and also three normalisation dictionaries of thousand pairs in Arabic language, thousand pairs in French language and thousand pairs in English language. The evaluation of the tool shows an average in N ormalisation success of % for English language, .% for Arabic Language and % for French Language. Also, the results of using an English language normalisation dictionary with a sentiment analysis tool for micro-blog's messages, show an increase in f-measure from. to ..
Complete list of metadatas

Cited literature [5 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01795348
Contributor : Amal Htait <>
Submitted on : Friday, May 18, 2018 - 1:18:20 PM
Last modification on : Friday, March 22, 2019 - 11:34:05 AM
Long-term archiving on : Tuesday, September 25, 2018 - 12:47:03 PM

File

AmalHtait_CICLing2018.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01795348, version 1

Collections

Citation

Amal Htait, Sébastien Fournier, Patrice Bellot. Unsupervised Creation of Normalisation Dictionaries for Micro-Blogs in Arabic, French and English. 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2018), Mar 2018, Hanoi, Vietnam. ⟨hal-01795348⟩

Share

Metrics

Record views

97

Files downloads

39