Corpus-Based methods for Short Text Similarity

Prajol Shrestha (1)

1 LINA - Laboratoire d'Informatique de Nantes Atlantique

This paper presents corpus-based methods to find similarity between short text (sentences, paragraphs, ...) which has many applications in the field of NLP. Previous works on this problem have been based on supervised methods or have used external resources such as WordNet, British National Corpus etc. Our methods are focused on unsupervised corpus-based methods. We present a new method, based on Vector Space Model, to capture the contextual behavior, senses and correlation, of terms and show that this method performs better than the baseline method that uses vector based cosine similarity measure. The performance of existing document similarity measures, Dice and Resemblance, are also evaluated which in our knowledge have not been used for short text similarity. We also show that the performance of the vector-based baseline method is improved when using stems instead of words and using the candidate sentences for computing the parameters rather than some external resource.

Mots clés: Similarity; Vector Space Model; Similarity metric

Domaines: Informatique et langage [cs.CL]

Prajol Shrestha. Corpus-Based methods for Short Text Similarity. Rencontre des Étudiants Chercheurs en Informatique pour le Traitement automatique des Langues, Jun 2011, Montpellier, France. pp.297. ⟨hal-00609909⟩