Downloading fresh JSON (update) loops unnecessarily?
xakepnz opened this issue · comments
I want to utilise local config as much as I can, to help with speed. When you had the .update() method I would utilise that. Now with the latest update, I need to build my own "update function" to overwrite the existing config, and any normal call, would just use the local json.
I've tried updating as below (fresh virtualenv) using pyattck-4.0.3 on Python 3.9.2 from pip:
>>> from pyattck import Attck
>>> import logging
>>>
>>> logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(message)s',level = 'DEBUG')
>>>
>>> attack_update = Attck(
... use_config=False,
... save_config=True,
... data_path='/home/xakep/derp/'
... )
>>>
>>> if attack_update:
... print('yes')
...
yes
>>>
>>> if attack_update.enterprise:
... print('yes')
...
Only now begins the download of the JSON files, and appears to download the same files 6 times:
2021-07-21 12:19:24,522 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:26,339 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:27,191 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:27,419 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:27,464 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:27,628 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:27,748 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:27,990 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:28,117 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:28,350 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:28,564 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:28,567 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:29,957 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:31,359 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:31,577 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:31,778 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:31,781 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:32,390 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:32,704 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:32,734 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:32,743 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:32,745 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:32,759 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:33,259 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:33,277 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:34,169 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:34,185 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:34,229 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:34,243 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:34,359 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:34,375 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:34,496 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:34,535 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:34,549 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:34,551 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:34,575 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:36,646 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:36,673 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:36,679 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:36,682 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:36,696 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:37,041 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:37,057 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:37,090 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:37,114 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:38,014 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:38,029 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:38,071 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:38,086 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:38,173 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:38,188 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:38,293 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:38,321 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:38,330 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:38,333 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:38,348 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:40,455 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:40,489 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:40,500 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:40,503 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:40,517 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:40,836 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:40,866 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:40,874 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:40,877 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:40,890 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:41,157 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:41,175 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:42,030 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:42,046 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:42,154 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:42,179 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:42,283 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:42,299 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:42,419 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:42,450 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:42,460 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:42,463 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:42,479 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:44,938 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:44,963 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:44,970 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:44,974 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:44,995 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:45,334 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:45,350 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:45,421 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:45,437 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
2021-07-21 12:19:46,304 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:46,318 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/pre-attack/pre-attack.json HTTP/1.1" 200 85794
2021-07-21 12:19:46,365 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:46,379 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/mobile-attack/mobile-attack.json HTTP/1.1" 200 228573
2021-07-21 12:19:46,483 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:46,499 - DEBUG - https://raw.githubusercontent.com:443 "GET /center-for-threat-informed-defense/attack-control-framework-mappings/master/frameworks/nist800-53-r4/stix/nist800-53-r4-controls.json HTTP/1.1" 200 331153
2021-07-21 12:19:46,643 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:46,675 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/generated_attck_data.json?raw=True HTTP/1.1" 302 138
2021-07-21 12:19:46,687 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/generated_attck_data.json HTTP/1.1" 302 157
2021-07-21 12:19:46,690 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:46,704 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/generated_attck_data.json HTTP/1.1" 200 31396583
2021-07-21 12:19:48,808 - DEBUG - Starting new HTTPS connection (1): github.com:443
2021-07-21 12:19:48,840 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/blob/master/attck_to_nist_controls.json?raw=True HTTP/1.1" 302 140
2021-07-21 12:19:48,847 - DEBUG - https://github.com:443 "GET /swimlane/pyattck/raw/master/attck_to_nist_controls.json HTTP/1.1" 302 159
2021-07-21 12:19:48,850 - DEBUG - Starting new HTTPS connection (1): media.githubusercontent.com:443
2021-07-21 12:19:48,869 - DEBUG - https://media.githubusercontent.com:443 "GET /media/swimlane/pyattck/master/attck_to_nist_controls.json HTTP/1.1" 200 9105826
2021-07-21 12:19:49,200 - DEBUG - Starting new HTTPS connection (1): raw.githubusercontent.com:443
2021-07-21 12:19:49,230 - DEBUG - https://raw.githubusercontent.com:443 "GET /mitre/cti/master/enterprise-attack/enterprise-attack.json HTTP/1.1" 200 2910652
yes
>>>
I believe the issue may be from the iteration over the JSON file names, and instead of a single-download it's downloading multiple? Am I wrong? And how do I try to implement an "update" func/method to only overwrite the json that exists using the new version 4.
Ref:
pyattck/pyattck/configuration.py
Line 50 in 719e358
for json_data in ['enterprise_attck_json', 'pre_attck_json',
'mobile_attck_json', 'nist_controls_json',
'generated_attck_json', 'generated_nist_json']:
if cls._check_if_url(getattr(cls, json_data)):
path = os.path.join(cls.data_path, "{json_data}.json".format(json_data=json_data))
data = cls.__download_url_data(getattr(cls, json_data))
cls.__write_to_disk(path, data)
PS: Great framework!
Thanks
@xakepnz interesting and thanks for reporting this! I’ll take a look at why it’s downloading multiple times - it shouldn’t be.
As far as the update functionality, I can definitely add that back in. Please create another issue for that and I’ll add it in the next release.
@xakepnz I believe I have a fix for this and should be pushing a change soon.