OAGS Title Generation Dataset
===============================

OAGS is a title generation dataset consisting of 34993700 abstracts 
and titles from scientific articles. Texts were lowercased and 
tokenized with Stanford CoreNLP tokenizer. No other preprocessing
steps were applied in this release version. Dataset records 
(samples) are stored as JSON lines in each text file. 

The data is derived from OAG data collection 
(https://aminer.org/open-academic-graph) which was released 
under ODC-BY licence. 

This data (OAGS Title Generation Dataset) is released under 
CC-BY licence (https://creativecommons.org/licenses/by/4.0/). 


Download
--------

This dataset can be download from LINDAT/CLARIN repository
http://hdl.handle.net/11234/1-3043


Publications
------------

If using it, please cite the following paper:

Çano, Erion and Bojar, Ondřej, 2019, "Efficiency Metrics for 
Data-Driven Models: A Text Summarization Case Study", INLG 2019, 
The 12th International Conference on Natural Language Generation, 
November 2019, Tokyo, Japan.

To reproduce the experiments in the above paper, you can use 
oags_train1.txt, oags_train2.txt, oags_train3.txt, oags_test.txt
and oags_val.txt files. If you need more data samples you can get 
them from oags_train_backup.txt and oags_val-test_backup.txt.


Acknowledgements
----------------

This research work was [partially] supported by OP RDE project No. 
CZ.02.2.69/0.0/0.0/16_027/0008495, International Mobility of 
Researchers at Charles University.


Statistics of Full OAGS:
------------------------

Total records: 34993700
Titles:
Total tokens: 479882266
Min length: 1 tokens
Max length: 821 tokens
Avg length: 12.3 tokens
Abstracts:
Total tokens: 7861867117
Min length: 1 tokens
Max length: 321610 tokens
Avg length: 189.7 tokens