oscar-project / download_oscar

Downloading all files of a language from the OSCAR (Open Super-large Crawled Aggregated coRpus)

Downloading language data from OSCAR automated

Features

• Adds dodc and dodg as command line tools
• dodc: command line variant provided with arguments to download data
• dodg: gui variant to download data, put arguments into input fields

Usage

dodc

To get help with the command line tool use dodc -h from a shell.

The command line tool needs to be supplied with multiple arguments:

• user: The user used to login to the site providing OSCAR.
• password: The password used to login to the site providing OSCAR.
• base_url: The url where the language files are hosted.
  • e.g. https://oscar-public.huma-num.fr/shuff-dedup/sv/
  • or https://oscar-prive.huma-num.fr/2109/packaged/sv/
• out: The folder where files should be downloaded to.
• chunk_size (optional): Defaults to 4096. The size of the chunks files are downloaded in.

dodg

The gui tool internally calls the command line tool dodc. Instead of providing arguments to the command line you can enter these into input fields directly and they will be passed downward to the command line tool.

Installation

Simple Installation

pip install download-oscar will install the requirements and the tool with one command.

Installing from source

Requirements

• Requires Python in version 3.
• Requires Requests
• Requires html5lib
• Requires BeautifulSoup
• Requires PySimpleGUI
• Requires tqdm

Building

• install Python
• git clone https://github.com/xamm/download_oscar.git
• cd download_oscar
• (optional) create a virtual enironment
• pip install -r requirements.txt
• pip install -e . will install the tool in development mode.

Release a new version

• All pushed git commits and pull requests on the main branch trigger an automatic build and packaging for pypi
  • commits without a tag only trigger packaging for TestPyPi
  • commits with a tag will also push to PyPi
  • A new version number must be specified in setup.py in order for publishing to work
    • publishing is trigerred on creation of a tag on the main branch
    • e.g. git tag -a v0.0.1 -m 'Release 0.1' and git push origin v0.0.1`
    • easiest procedure:
      • work on your code
      • add & commit changes
      • push changes
      • create tag
      • push tag

Licence

This tool was originally started during a student project at the Database Systems Group Dresden.