Capitalizing on a TREC Track to Build a Tweet Summarization Dataset

Alexis Dusart (1) , Karen Pinel-Sauvagnat (1) , Gilles Hubert (1)

1 IRIT-IRIS - Recherche d’Information et Synthèse d’Information

Today there is a lack of standard collection for automatic tweet summarization evaluation. The construction of such a large dataset is very tedious. In this paper, we check whether the dataset proposed for the TREC Incident Streams track, which was not created for automatic summary generation, could be used in this way. Indeed, when filtering the TREC Incident Streams (IS) dataset with the assessors' annotations, it appears to respect the citeria identified in the literature related to automatic summarization. For this, we studied the TREC IS dataset and then proposed a subset summarizing each event, based on the assessors' annotations. This subset is evaluated according to the criteria previously mentioned. Several widely used state-of-the-art models for automatic text summarization, adapted to tweet summarization, were finally tested on the proposed dataset. The code, the annotations and the results are provided on our Github.

Domaines: Recherche d'information [cs.IR]; Réseaux sociaux et d'information [cs.SI]

Alexis Dusart, Karen Pinel-Sauvagnat, Gilles Hubert. Capitalizing on a TREC Track to Build a Tweet Summarization Dataset. Joint Conference of the Information Retrieval Communities in Europe (CIRCLE 2020), Université de Toulouse, France, Jul 2020, Samatan, Gers, France. pp.1-9. ⟨hal-03095613⟩