MLWIKIR: A Python toolkit for building large-scale Wikipedia-based Information Retrieval Datasets in Chinese, English, French, Italian, Japanese, Spanish and more

Jibril Frej , Didier Schwab (1) , Jean-Pierre Chevallet (2)

1 GETALP - Groupe d’Étude en Traduction Automatique/Traitement Automatisé des Langues et de la Parole
2 MRIM - Modélisation et Recherche d’Information Multimédia [Grenoble]

Deep learning allowed for new state-of-the-art performance on ad-hoc information retrieval (IR). This approach usually requires large amounts of annotated data to be more effective than traditional baselines such as BM25. However, most standard ad-hoc IR datasets publicly available for academic research (e.g. Robust04, ClueWeb09) have at most 250 annotated queries and are usually in English only. Deep learning models for IR (e.g. DUET, Conv-KNRM) perform poorly on such datasets as they are trained and evaluated on large scale datasets collected from commercial search engines, not publicly available for academic research. This is a problem for reproducibility and the advancement of research. Moreover, most datasets are in English or Chinese only and deep learning models for ad-hoc IR are not evaluated on other languages. In this paper, we propose MLWIKIR: an open-source toolkit to automatically build large-scale information retrieval datasets based on Wikipedia in 10 different languages that can be adapted to any Wikipedia language given a tokenizer.

Mots clés: Information Retrieval; Open Source; Dataset; Deep Learning

Domaines: Intelligence artificielle [cs.AI]

Jibril Frej, Didier Schwab, Jean-Pierre Chevallet. MLWIKIR: A Python toolkit for building large-scale Wikipedia-based Information Retrieval Datasets in Chinese, English, French, Italian, Japanese, Spanish and more. Joint Conference of the Information Retrieval Communities in Europe, Jul 2020, Toulouse, France. ⟨hal-03263816⟩