For a fistful of blogs: Discovery and comparative benchmarking of republishable German content

Adrien Barbaresi (1) , Kay-Michael Würzner (1)

1 BBAW - Berlin-Brandenburgische Akademie der Wissenschaften

We introduce two corpora gathered on the web and related to computer-mediated communication: blog posts and blog comments. In order to build such corpora, we addressed following issues: website discovery and crawling, content extraction constraints, and text quality assessment. The blogs were manually classified as to their license and content type. Our results show that it is possible to find blogs in German under Creative Commons license, and that it is possible to perform text extraction and linguistic annotation efficiently enough to allow for a comparison with more traditional text types such as newspaper corpora and subtitles. The comparison gives insights on distributional properties of the processed web texts on token and type level. For example, quantitative analysis reveals that blog posts are close to written language, while comments are slightly closer to spoken language.

Mots clés: Corpus Construction; Web Crawling; Computer-Mediated Communication CMC; Creative Commons; Blogosphere; Text Extraction; Quality Assessment; Visualization

Domaines: Linguistique; Informatique et langage [cs.CL]; Web

Adrien Barbaresi, Kay-Michael Würzner. For a fistful of blogs: Discovery and comparative benchmarking of republishable German content. KONVENS 2014, NLP4CMC workshop, Oct 2014, Hildesheim, Germany. pp.2-10. ⟨hal-01083750⟩