#hardtoparse: POS Tagging and Parsing the Twitterverse

Jennifer Foster (1) , Özlem Çetinoglu (1) , Joachim Wagner (1) , Joseph Le Roux (2) , Stephen Hogan (1) , Joakim Nivre (3) , Deirdre Hogan (1) , Josef van Genabith (1)

1 NCLT - National Centre for Language Technology
2 LIF - Laboratoire d'informatique Fondamentale de Marseille - UMR 6166
3 Uppsala University

We evaluate the statistical dependency parser, Malt, on a new dataset of sentences taken from tweets. We use a version of Malt which is trained on gold standard phrase structure Wall Street Journal (WSJ) trees converted to Stanford labelled dependencies. We observe a drastic drop in performance moving from our in-domain WSJ test set to the new Twitter dataset, much of which has to do with the propagation of part-of-speech tagging er- rors. Retraining Malt on dependency trees produced by a state-of-the-art phrase structure parser, which has itself been self-trained on web material, results in a sig- nificant improvement. We analyse this improvement by examining in detail the effect of the retraining on indi- vidual dependency types.

Domaines: Informatique et langage [cs.CL]

Jennifer Foster, Özlem Çetinoglu, Joachim Wagner, Joseph Le Roux, Stephen Hogan, et al.. #hardtoparse: POS Tagging and Parsing the Twitterverse. AAAI 2011 Workshop On Analyzing Microtext, 2011, United States. pp.20-25. ⟨hal-00702445⟩