durner / AnyBlob

AnyBlob - A Universal Cloud Object Storage Download Manager Built For Cost-Throughput Optimal Analytics!

Home Page:http://anyblob.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Make it easier to set-up the required amount of retrievers

durner opened this issue · comments

  • Use default throughput per core values, that can be adapted by the user
  • Use default requests per retriever values, that can be adapted by the user
  • Use the network throughput information from the instance to calculate concurrency and outstanding requests
  • Build pipeline to automatically get the information for the cloud vendors instances
  • Integrate that into a config helper class that can be used easily, e.g.,
    - config.requiredThreadCount() to give the user the number of retriever threads that need to started
    - New constructor TaskSendReceiverGroup(config) to initialize the values more easily