Plaintext Wikipedia dump 2018

Rosa, Rudolf

dc.contributor.author	Rosa, Rudolf
dc.date.accessioned	2018-05-09T09:25:05Z
dc.date.available	2018-05-09T09:25:05Z
dc.date.issued	2018-02-25
dc.identifier.uri	http://hdl.handle.net/11234/1-2735
dc.description	Wikipedia plain text data obtained from Wikipedia dumps with WikiExtractor in February 2018. The data come from all Wikipedias for which dumps could be downloaded at [https://dumps.wikimedia.org/]. This amounts to 297 Wikipedias, usually corresponding to individual languages and identified by their ISO codes. Several special Wikipedias are included, most notably "simple" (Simple English Wikipedia) and "incubator" (tiny hatching Wikipedias in various languages). For a list of all the Wikipedias, see [https://meta.wikimedia.org/wiki/List_of_Wikipedias]. The script which can be used to get new version of the data is included, but note that Wikipedia limits the download speed for downloading a lot of the dumps, so it takes a few days to download all of them (but one or a few can be downloaded fast). Also, the format of the dumps changes time to time, so the script will probably eventually stop working one day. The WikiExtractor tool [http://medialab.di.unipi.it/wiki/Wikipedia_Extractor] used to extract text from the Wikipedia dumps is not mine, I only modified it slightly to produce plaintext outputs [https://github.com/ptakopysk/wikiextractor].
dc.language.iso	abk
dc.language.iso	ace
dc.language.iso	ady
dc.language.iso	afr
dc.language.iso	aka
dc.language.iso	als
dc.language.iso	amh
dc.language.iso	ang
dc.language.iso	ara
dc.language.iso	arc
dc.language.iso	arg
dc.language.iso	arz
dc.language.iso	asm
dc.language.iso	ast
dc.language.iso	atj
dc.language.iso	ava
dc.language.iso	aym
dc.language.iso	azb
dc.language.iso	aze
dc.language.iso	bak
dc.language.iso	bam
dc.language.iso	bar
dc.language.iso	bcl
dc.language.iso	bel
dc.language.iso	ben
dc.language.iso	bis
dc.language.iso	bjn
dc.language.iso	bod
dc.language.iso	bos
dc.language.iso	bpy
dc.language.iso	bre
dc.language.iso	bug
dc.language.iso	bul
dc.language.iso	bxr
dc.language.iso	cat
dc.language.iso	cdo
dc.language.iso	ceb
dc.language.iso	ces
dc.language.iso	cha
dc.language.iso	che
dc.language.iso	chr
dc.language.iso	chu
dc.language.iso	chv
dc.language.iso	chy
dc.language.iso	ckb
dc.language.iso	cor
dc.language.iso	cos
dc.language.iso	cre
dc.language.iso	crh
dc.language.iso	csb
dc.language.iso	cym
dc.language.iso	dan
dc.language.iso	deu
dc.language.iso	din
dc.language.iso	diq
dc.language.iso	div
dc.language.iso	dsb
dc.language.iso	dzo
dc.language.iso	ell
dc.language.iso	eng
dc.language.iso	epo
dc.language.iso	est
dc.language.iso	eus
dc.language.iso	ewe
dc.language.iso	ext
dc.language.iso	fao
dc.language.iso	fas
dc.language.iso	fij
dc.language.iso	fin
dc.language.iso	fra
dc.language.iso	frp
dc.language.iso	frr
dc.language.iso	fry
dc.language.iso	ful
dc.language.iso	fur
dc.language.iso	gag
dc.language.iso	gan
dc.language.iso	gla
dc.language.iso	gle
dc.language.iso	glg
dc.language.iso	glk
dc.language.iso	glv
dc.language.iso	gom
dc.language.iso	got
dc.language.iso	grn
dc.language.iso	guj
dc.language.iso	hak
dc.language.iso	hat
dc.language.iso	hau
dc.language.iso	haw
dc.language.iso	hbs
dc.language.iso	heb
dc.language.iso	her
dc.language.iso	hif
dc.language.iso	hin
dc.language.iso	hmo
dc.language.iso	hrv
dc.language.iso	hsb
dc.language.iso	hun
dc.language.iso	hye
dc.language.iso	ibo
dc.language.iso	ido
dc.language.iso	iku
dc.language.iso	ile
dc.language.iso	ilo
dc.language.iso	ina
dc.language.iso	ind
dc.language.iso	ipk
dc.language.iso	isl
dc.language.iso	ita
dc.language.iso	jam
dc.language.iso	jav
dc.language.iso	jbo
dc.language.iso	jpn
dc.language.iso	kaa
dc.language.iso	kab
dc.language.iso	kal
dc.language.iso	kan
dc.language.iso	kas
dc.language.iso	kat
dc.language.iso	kau
dc.language.iso	kaz
dc.language.iso	kbd
dc.language.iso	kbp
dc.language.iso	khm
dc.language.iso	kik
dc.language.iso	kin
dc.language.iso	kir
dc.language.iso	koi
dc.language.iso	kom
dc.language.iso	kon
dc.language.iso	kor
dc.language.iso	krc
dc.language.iso	ksh
dc.language.iso	kur
dc.language.iso	lad
dc.language.iso	lao
dc.language.iso	lat
dc.language.iso	lav
dc.language.iso	lbe
dc.language.iso	lez
dc.language.iso	lij
dc.language.iso	lim
dc.language.iso	lin
dc.language.iso	lit
dc.language.iso	lmo
dc.language.iso	lrc
dc.language.iso	ltg
dc.language.iso	ltz
dc.language.iso	lug
dc.language.iso	lzh
dc.language.iso	mah
dc.language.iso	mai
dc.language.iso	mal
dc.language.iso	mar
dc.language.iso	mdf
dc.language.iso	mhr
dc.language.iso	min
dc.language.iso	mkd
dc.language.iso	mlg
dc.language.iso	mlt
dc.language.iso	mon
dc.language.iso	mri
dc.language.iso	mrj
dc.language.iso	msa
dc.language.iso	mus
dc.language.iso	mwl
dc.language.iso	mya
dc.language.iso	myv
dc.language.iso	mzn
dc.language.iso	nan
dc.language.iso	nap
dc.language.iso	nau
dc.language.iso	nav
dc.language.iso	ndo
dc.language.iso	nds
dc.language.iso	nep
dc.language.iso	new
dc.language.iso	nld
dc.language.iso	nno
dc.language.iso	nor
dc.language.iso	nov
dc.language.iso	nso
dc.language.iso	nya
dc.language.iso	oci
dc.language.iso	olo
dc.language.iso	ori
dc.language.iso	orm
dc.language.iso	oss
dc.language.iso	pag
dc.language.iso	pam
dc.language.iso	pan
dc.language.iso	pap
dc.language.iso	pcd
dc.language.iso	pdc
dc.language.iso	pfl
dc.language.iso	pih
dc.language.iso	pli
dc.language.iso	pms
dc.language.iso	pnb
dc.language.iso	pnt
dc.language.iso	pol
dc.language.iso	por
dc.language.iso	pus
dc.language.iso	que
dc.language.iso	rmy
dc.language.iso	roh
dc.language.iso	ron
dc.language.iso	rue
dc.language.iso	run
dc.language.iso	rup
dc.language.iso	rus
dc.language.iso	sag
dc.language.iso	sah
dc.language.iso	san
dc.language.iso	scn
dc.language.iso	sco
dc.language.iso	sgs
dc.language.iso	sin
dc.language.iso	slk
dc.language.iso	slv
dc.language.iso	sme
dc.language.iso	smo
dc.language.iso	sna
dc.language.iso	snd
dc.language.iso	som
dc.language.iso	sot
dc.language.iso	spa
dc.language.iso	sqi
dc.language.iso	srd
dc.language.iso	srn
dc.language.iso	srp
dc.language.iso	ssw
dc.language.iso	stq
dc.language.iso	sun
dc.language.iso	swa
dc.language.iso	swe
dc.language.iso	szl
dc.language.iso	tah
dc.language.iso	tam
dc.language.iso	tat
dc.language.iso	tcy
dc.language.iso	tel
dc.language.iso	ten
dc.language.iso	tet
dc.language.iso	tgk
dc.language.iso	tgl
dc.language.iso	tha
dc.language.iso	tir
dc.language.iso	ton
dc.language.iso	tpi
dc.language.iso	tsn
dc.language.iso	tso
dc.language.iso	tuk
dc.language.iso	tum
dc.language.iso	tur
dc.language.iso	twi
dc.language.iso	tyv
dc.language.iso	udm
dc.language.iso	uig
dc.language.iso	ukr
dc.language.iso	urd
dc.language.iso	uzb
dc.language.iso	vec
dc.language.iso	ven
dc.language.iso	vep
dc.language.iso	vie
dc.language.iso	vls
dc.language.iso	vol
dc.language.iso	vro
dc.language.iso	war
dc.language.iso	wln
dc.language.iso	wol
dc.language.iso	wuu
dc.language.iso	xal
dc.language.iso	xho
dc.language.iso	xmf
dc.language.iso	yid
dc.language.iso	yor
dc.language.iso	yue
dc.language.iso	zea
dc.language.iso	zha
dc.language.iso	zho
dc.language.iso	zul
dc.language.iso	dty
dc.publisher	Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL)
dc.rights	Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0)
dc.rights.uri	http://creativecommons.org/licenses/by-sa/3.0/
dc.subject	Wikipedia
dc.subject	text corpora
dc.subject	monolingual corpus
dc.title	Plaintext Wikipedia dump 2018
dc.type	corpus
metashare.ResourceInfo#ContentInfo.mediaType	text
dc.rights.label	PUB
has.files	yes
branding	LINDAT / CLARIAH-CZ
contact.person	Rudolf Rosa rosa@ufal.mff.cuni.cz Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics (UFAL)
sponsor	Ministerstvo školství, mládeže a tělovýchovy České republiky CZ.02.1.01/0.0/0.0/16_013/0001781 LINDAT/CLARIN - Výzkumná infrastruktura pro jazykové technologie - rozšíření repozitáře a výpočetní kapacity nationalFunds
size.info	21 gb
size.info	297 files
size.info	8503209631 words
files.size	21947585957
files.count	299