lyft / cartography

Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.

Home Page:https://lyft.github.io/cartography/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

AWS: Make fetches concurrent: ECR and S3

achantavy opened this issue · comments

Discussed in #1201

Description:

Describe your idea. Please be detailed. If a feature request, please
describe the desired behavior, what scenario it enables, and how it
would be used.

As a proof of concept for parallel gets, implement async fetching for AWS ECR and S3. In initial testing we've seen speedup improvements from 15 minutes to 2 minutes so this is worth pursuing.

Ensure that

  • The solution is easy to apply for other modules
  • We have a plan on how to handle rate limits being exceeded - e.g. with backoff + retry + giveup
  • We have documentation on how new developers can employ the pattern for their own intel modules.