CEHugeWebCorpus

  Authors

Rüdiger, Jan Oliver

  Item identifier

http://hdl.handle.net/11372/LRT-2638

 Project URL

https://notes.jan-oliver-ruediger.de/korpora/

 Date issued

2018-03-01

 Type

corpus, text

 Size

3021000000 tokens

 Language(s)

German

 Description

This corpus was originally created for performance testing (server infrastructure CorpusExplorer - see: diskurslinguistik.net / diskursmonitor.de). It includes the filtered database (German texts only) of CommonCrawl (as of March 2018). First, the URLs were filtered according to their top-level domain (de, at, ch). Then the texts were classified using NTextCat and only uniquely German texts were included in the corpus. The texts were then annotated using TreeTagger (token, lemma, part-of-speech). 2.58 million documents - 232.87 million sentences - 3.021 billion tokens. You can use CorpusExplorer (http://hdl.handle.net/11234/1-2634) to convert this data into various other corpus formats (XML, JSON, Weblicht, TXM and many more).

 Publisher

Rüdiger, Jan Oliver

 Subject(s)

corpus German Germanistik Web corpus web corpora CorpusExplorer

 Collection(s)

LRT + Open Submissions Data & Tools

Show full item record