Welcome to AgDownloader! This library is a simple cmd tool to download images concurrently and thread-safe
Please Note: System uses ruby 3.2.2 syntax, if you run it in lower version of ruby it will raise syntax error.
System Divided into modules and classes and each component has a single responsibility and will be used as interface it the system.
AgDownloader::Util
helpers for handling file related concerns like reading and writing files.AgDownloader::HTTP
For handling HTTP requests all HTTP request will be initiated by this class.AgDownloader::Validations
For validating url and file content, this module providesvalidates
method as interface for the system.Agdownloader::Download
Main class which handles all the logic and has a public methodsbatch_download
which will be used as interface for the system.AgDownloader::AgDownloaderError
Base class for all errors in the system.AgDownloader::CLI
is a interface in cmd and your command will be handled by this class.AgDownloader::Logger
is a simple logger class which will be used for logging in the system.AgDownloader::Logging
interface for logger class and will be used in the system.
- Reason not using
open-uri
is memory usage, system reads downloaded file content in chunks and will not load all data in memory - For HTTP request its better to use
Net::HTTP
instead ofOpenURI
because it has more control over the request and response and also it has more options to set for the request, also its possible to useFaraday
orTyphoeus
but they are more complex but they coming with dependencies and not necessary for this task. - All HTTP requests are concurrent and thread-safe and will be handled by
Net::HTTP
andThread
classes - Since system is IO-intensive there is no need to parallelize the process and concurrent requests are enough to handle the task especially its the case in bigger files.
- Ruby v3.2.2
Start a new Bash shell in login mode:
$ /bin/bash --login
Login shell loads system-wide settings from /etc/profile and then looks for the first readable file among ~/.bash_profile
, ~/.bash_login
, and ~/.profile
, loading its settings.
This way you can make sure all executables are available and environment variables are set.
Clone the repository and for build run:
$ gem build ag_downloader.gemspec
Then to install the gem run:
$ gem install ag_downloader-0.1.0.gem
You are all set to use the application. refer to Usage section for more information.
After installing the gem, you can use it in your terminal. The gem has two commands:
This app uses ruby as interpreter and has executable file and will be installed in gem path is your system which makes it usable in any path in your terminal and does not need to run it in specific path.
$ ag -u https://www.example.com/links_file.txt
This command will check url and validates it and after that will validates and download all images in the file and save them in the current directory.
or you can pass local file to the command line:
$ ag -f path/to/file/links_file.txt
This way all the links in the file will be validated and files will be downloaded.
After checking out the repo, to install dependencies run:
$ bin/setup
Then, to run the tests run:
$ bundle exec rspec
- Adding ability to capture errors and retry the request
- Adding ability to pick up uncompleted downloads
- Adding ability to set number of concurrent requests
- Adding ability to set number of retries for requests
- System Errors are implemented in and all errors are inherited from
AgDownloader::AgDownloaderError
and class could be extend for more functionality like formatting error and as template for subclasses. - Adding format to logger to customize the output