Filtering and Measuring the Intrinsic Quality of Human Compositionality Judgments

Carlos Ramisch (1) , Silvio Cordeiro (1) , Aline Villavicencio (2)

1 LIF - Laboratoire d'informatique Fondamentale de Marseille
2 Instituto de Informática [Porto Alegre]

This paper analyzes datasets with numerical scores that quantify the semantic compositionality of MWEs. We present the results of our analysis of crowdsourced compositionality judgments for noun compounds in three languages. Our goals are to look at the characteristics of the annotations in different languages; to examine intrinsic quality measures for such data; and to measure the impact of filters proposed in the literature on these measures. The cross-lingual results suggest that greater agreement is found for the extremes in the compositionality scale, and that outlier annotation removal is more effective than outlier annotator removal.

Domaines: Informatique et langage [cs.CL]

Carlos Ramisch, Silvio Cordeiro, Aline Villavicencio. Filtering and Measuring the Intrinsic Quality of Human Compositionality Judgments. 12th Workshop on Multiword Expressions, Aug 2016, Berlin, Germany. pp.32--37, ⟨10.18653/v1/W16-1804⟩. ⟨hal-01459913⟩