lyft / cartography

Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.

Home Page:https://lyft.github.io/cartography/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ECS Ingestion Improvements

tmsteere opened this issue · comments

Title: ECS Ingestion currently repeats the ecs_task_defintion and ingests unused task definitions

Description:
ECS relies on task definitions for each service running on the platform. These task definitions are unique across ECS and not dependent on clusters. Currently, Cartography has two issues with ingestion task definitions:

  1. Cartography repeats the ingestion process for each Cluster although the data is the same. This adds significant time to the sync process
  2. Cartography syncs all task definitions, even if they are not currently in use. Since a common ECS configuration creates a new task definition on each deployment, this results in hundreds or thousands of unused task definitions. Syncing all of these task definitions adds significant time to the sync process and results in task definition nodes that do not link to anything in the graph.

I'm providing a merge request that makes the following changes:

  1. Updates the process to only import the actively used task definitions by parsing the tasks list for the taskDefinitionArn instead of ingesting all task definitions
  2. Creating a relationship between the task definition and the task that definition is used by
  3. Updates the tests for importing task definitions

[optional Relevant Links:]

Any extra documentation required to understand the issue.