neilsmurphy / sharadar-download-manager

a Python tool for downloading sharadar data from Quandl.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Memory management and full datafile downloads

neilsmurphy opened this issue · comments

Currently the downloads can fail due to memory isssues when downloading some of the larger files.

Need to implement method to batch downloads on date columns and then also on a sub column, likely tickers?

Then if this is done natural to progress to downloading all files for initial installation.

See if there's an existing library for managing batch size. Or perhaps time the batches and adjust size accordingly.

Created downloading for large files in zip and then creating new database. This is not in the repo. Try to have auto update happen if possilbe with a auto running script.