ameerezae / Media-Downloader

Python Script to download media with live reporting using Prometheus and Grafana

Media Downloader

Description

A Multi-Thread Media downloader for small size files like image.

Getting Started

Dependencies

Dependencies are listed in requirements.txt file and automatically installed by Docker.

Executing

Environment Variables can set/change in docker-compose.yml

environment:
      DOWNLOADER_MODE: "VPN"
      DOWNLOADER_THREAD_NO: "40"
      REQUEST_TIMEOUT: "30"
      CONNECTION_ESTABLISHMENT_TIMEOUT: "1"
      DOWNLOADING_PATH: "media"
      DOWNLOADER_MAX_RETRY: "1"

run downloader by following command:

sudo docker-compose up

after successfully run this command logs should be like this:

Prometheus Metrics are accessible in localhost:5000

Metrics are:

1. File_Average_Size
2. Total count
3. Faild count
4. Success count
5. Permissoin_Denied errors count
6. Not_Found errors count
7. Internal_Server errors count
8. Timeout errors count
9. Not_Valid errors count
10. File_Average_Time count
11. Average_Rate of downloading files
12. File_Per_Second count

Result

All downloaded file saved with a unique uuid in DOWNLOADING_PATH folder specified name in environment variable in docker-compose.yml.

You can clone prometheus from here and track live data reporting by Gragana panel running on localhost:3000

Notice: you should change targets in prometheus/prometheus/prometheus.yml to:

scrape_configs:
  - job_name: my-service
    static_configs:
      - targets:
        - {YOUR-SYSTEM-IP-ADDRESS}:5000

Grafana panel should be something like this:

Authors

Amir Rezaei @ameerezae