lyft / cartography

Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.

Home Page:https://lyft.github.io/cartography/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Cartography does not handle Okta API limits

tmsteere opened this issue · comments

Title: Cartography does not check Okta API rate limits

Description:
Okta enforces strict rate limits and includes the current rate limit status in response headers. Cartography will crash during groups sync if the Okta API rate limit is exceeded. In applications sync, there are catch statements that discard errors at debug level. When these are hit, data is not collected for that request, resulting in missing data.

To Reproduce:
Run Okta sync on an account with a large number of groups and users

Logs:
ERROR:cartography.intel.okta.groups:OktaError while listing members of group redacted
ERROR:cartography.sync:Unhandled exception during sync stage 'okta'
Traceback (most recent call last):
File "/srv/cartography/cartography/sync.py", line 85, in run
stage_func(neo4j_session, config)
File "/srv/cartography/cartography/util.py", line 128, in timed
result = method(*args, **kwargs)
File "/srv/cartography/cartography/intel/okta/init.py", line 67, in start_okta_ingestion
groups.sync_okta_groups(neo4j_session, config.okta_org_id, config.update_tag, config.okta_api_key, state)
File "/srv/cartography/cartography/util.py", line 128, in timed
result = method(*args, **kwargs)
File "/srv/cartography/cartography/intel/okta/groups.py", line 298, in sync_okta_groups
sync_okta_group_membership(neo4_session, api_client, group_list_info, okta_update_tag)
File "/srv/cartography/cartography/util.py", line 128, in timed
result = method(*args, **kwargs)
File "/srv/cartography/cartography/intel/okta/groups.py", line 268, in sync_okta_group_membership
members_data: List[Dict] = get_okta_group_members(api_client, group_id)
File "/srv/cartography/cartography/util.py", line 128, in timed
result = method(*args, **kwargs)
File "/srv/cartography/cartography/intel/okta/groups.py", line 80, in get_okta_group_members
paged_response = api_client.get_path(f'/{group_id}/users', params)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 82, in get_path
return self.get(self.base_url + url_path, params)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 48, in get
return self.get(url, params, attempts)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 48, in get
return self.get(url, params, attempts)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 48, in get
return self.get(url, params, attempts)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 45, in get
if self.__check_response(resp, attempts):
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 102, in __check_response
raise OktaError(json.loads(resp.text))
okta.framework.OktaError.OktaError: API call exceeded rate limit due to too many requests.
Traceback (most recent call last):
File "/usr/local/bin/cartography", line 33, in
sys.exit(load_entry_point('cartography', 'console_scripts', 'cartography')())
File "/srv/cartography/cartography/cli.py", line 581, in main
sys.exit(CLI(default_sync, prog='cartography').main(argv))
File "/srv/cartography/cartography/cli.py", line 561, in main
return cartography.sync.run_with_config(self.sync, config)
File "/srv/cartography/cartography/sync.py", line 163, in run_with_config
return sync.run(neo4j_driver, config)
File "/srv/cartography/cartography/sync.py", line 85, in run
stage_func(neo4j_session, config)
File "/srv/cartography/cartography/util.py", line 128, in timed
result = method(*args, **kwargs)
File "/srv/cartography/cartography/intel/okta/init.py", line 67, in start_okta_ingestion
groups.sync_okta_groups(neo4j_session, config.okta_org_id, config.update_tag, config.okta_api_key, state)
File "/srv/cartography/cartography/util.py", line 128, in timed
result = method(*args, **kwargs)
File "/srv/cartography/cartography/intel/okta/groups.py", line 298, in sync_okta_groups
sync_okta_group_membership(neo4_session, api_client, group_list_info, okta_update_tag)
File "/srv/cartography/cartography/util.py", line 128, in timed
result = method(*args, **kwargs)
File "/srv/cartography/cartography/intel/okta/groups.py", line 268, in sync_okta_group_membership
members_data: List[Dict] = get_okta_group_members(api_client, group_id)
File "/srv/cartography/cartography/util.py", line 128, in timed
result = method(*args, **kwargs)
File "/srv/cartography/cartography/intel/okta/groups.py", line 80, in get_okta_group_members
paged_response = api_client.get_path(f'/{group_id}/users', params)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 82, in get_path
return self.get(self.base_url + url_path, params)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 48, in get
return self.get(url, params, attempts)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 48, in get
return self.get(url, params, attempts)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 48, in get
return self.get(url, params, attempts)
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 45, in get
if self.__check_response(resp, attempts):
File "/usr/local/lib/python3.10/dist-packages/okta/framework/ApiClient.py", line 102, in __check_response
raise OktaError(json.loads(resp.text))
okta.framework.OktaError.OktaError: API call exceeded rate limit due to too many requests.

Please complete the following information::

  • Cartography release version or commit hash 0.74.0
  • Python version: 3.10.6

Additional context:

I have a working fix for this and will create a PR after my existing PR is merged