Create dataset opus_100

Question

Create dataset opus_100

albertvillanova opened this issue 2 years ago · comments

Albert Villanova del Moral commented 2 years ago

uid: opus_100
type: processed
description:
- name: OPUS-100
- description: OPUS-100 is an English-centric multilingual corpus covering 100 languages.
- homepage: https://github.com/EdinburghNLP/opus-100-corpus
- validated: True
languages:
- language_names:
  - Niger-Congo
  - Arabic
  - Catalan
  - Chinese
  - English
  - French
  - Indonesian
  - Portuguese
  - Spanish
- language_comments:
- language_locations:
  - Netherlands
  - Switzerland
  - Scotland
- validated: False
custodian:
- name: Jörg Tiedemann
- in_catalogue:
- type: A private individual
- location: Turkey
- contact_name:
- contact_email: jorg.tiedemann@helsinki.fi
- contact_submitter: False
- additional: https://opus.nlpl.eu/
- validated: False
availability:
- procurement:
  - for_download: Yes - it has a direct download link or links
  - download_url: https://opus.nlpl.eu/opus-100.php
  - download_email:
- licensing:
  - has_licenses: Unclear
  - license_text: Quoting the website "In the OPUS project we try to convert and align free online data, to add linguistic annotation, and to provide the community with a publicly available parallel corpus.", I believe the data can be used.
  - license_properties:
  - license_list:
- pii:
  - has_pii: Yes
  - generic_pii_likely: very likely
  - generic_pii_list:
    - names
    - physical addresses
  - numeric_pii_likely: unlikely
  - numeric_pii_list:
  - sensitive_pii_likely: unlikely
  - sensitive_pii_list:
  - no_pii_justification_class:
  - no_pii_justification_text:
- validated: False
processed_from_primary:
- from_primary: Taken from primary source
- primary_availability: Yes - their documentation/homepage/description is available
- primary_license: Unclear / I don't know
- primary_types:
  - news articles
  - web | wiki
  - web | other
- validated: False
- from_primary_entries:
media:
- category:
  - text
- text_format:
  - .TXT
- audiovisual_format:
- image_format:
- database_format:
  - .GZ
  - .TAR
- text_is_transcribed: No
- instance_type: sentence pair
- instance_count: 1M<n<1B
- instance_size: 10<n<100
- validated: False
fname: opus_100.json

Albert Villanova del Moral · Answer 1 · Thu Jan 20 2022 00:18:14 GMT+0800 (China Standard Time)

It already exists: https://huggingface.co/datasets/opus100

Need to pass the language pair: en + ?

Albert Villanova del Moral · Answer 2 · Wed Jan 26 2022 18:22:09 GMT+0800 (China Standard Time)

DONE:

https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_nigercongo-ig_opus100
https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_nigercongo-rw_opus100
https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_nigercongo-xh_opus100
https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_nigercongo-yo_opus100
https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_nigercongo-zu_opus100