FlyPythons / hwget

Download data with HuaWei cloud

hwget

Hwget is used to download data to HuaWei cloud OBS.

Why to develop

Too slow to download bio-data from sites abroad.

WARNING!!!

It's cheap to download data to HuaWei cloud, but is expensive to get data from HuaWei cloud.

Requirement

• python 2.7+ or python3.5+
• requests
• huaweicloud-sdk-python
• huaweicloud-sdk-python-obs

Install

pip install git+https://github.com/FlyPythons/hwget.git

Usage

import sys
import logging
from hwget import Hwget

logging.basicConfig(
    level=logging.INFO, 
    stream=sys.stderr,
    format='%(asctime)s [%(levelname)s] %(message)s',
    datefmt='%d %b %Y %H:%M:%S'
)

ak = "your_ak"
sk = "your_sk"
region = "ap-southeast-1"  # Hong Kong
project_id = "my_project_id"
bucket = "bucket_you_want_to_save_to"
image = "1ab5c293-81e8-46c9-9bc8-f6d6fa464cd0"

cloud = Hwget(
    ak=ak,
    sk=sk, 
    region=region, 
    project_id=project_id,
    bucket=bucket,
    image=image
)

cloud.get([
    "https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-5/SRR1609905/SRR1609905.2",
    "https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos2/sra-pub-run-7/SRR1609906/SRR1609906.2",
    "https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-5/SRR1609907/SRR1609907.2",
])