Giters

bigscience-workshop / data_tooling

Tools for managing datasets for governance and training.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Create license-compliant version of the Pile: EuroParl

albertvillanova opened this issue · comments

commented
Create license-compliant version of the Pile: EuroParl
commented

DONE:
https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_en_the_pile_europarl
https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_es_the_pile_europarl
https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_fr_the_pile_europarl
https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_pt_the_pile_europarl

Sample:

{
  'text': "The placing of biocidal products on the market - New biocides revision proposal (debate) \nPresident\nThe next item is the joint debate on the following:\nMrs Sârbu's report, on behalf of the Committee on the Environment, Public He..."
}