KeyError: epss
brainstorm opened this issue · comments
Today's clean re-installation of emba (rm -rf ./external) yields the CveExplorer error: KeyError: epss:
emba$ git reflog
95d469d4 (HEAD -> master, origin/master, origin/HEAD) HEAD@{0}: pull: Fast-forward
7b5ae5b5 HEAD@{1}: pull: Fast-forward
bc8f63e9 HEAD@{2}: clone: from https://github.com/e-m-b-a/emba.git
emba$ sudo ./installer.sh -d
(...)
2023-11-23 13:14:21,853 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:06:03.180899
2023-11-23 13:14:21,854 - CveXplore.database.maintenance.Sources_process - INFO - Finished VIA4 database update
Traceback (most recent call last):
File "/home/rvalls/dev/personal/emba/external/emba_venv/bin/cvexplore", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
rv = super().invoke(ctx)
^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/cli_cmds/db_cmds/commands.py", line 28, in initialize_cmd
ctx.obj["data_source"].database.initialize()
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 187, in initialize
self.update()
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 72, in update
up = source["updater"]()
^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 1051, in __init__
self.feed_url = Configuration.getFeedURL(self.feed_type.lower())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/common/config.py", line 113, in getFeedURL
return cls.SOURCES[source]
~~~~~~~~~~~^^^^^^^^
KeyError: 'epss'
------------------------------------------------------------------------------------------------
Error detected - status code 1
Command: echo -e "Command: ${ORANGE}${BASH_COMMAND}${NC}"
Location: ./installer/IF20_cve_search.sh, line 203
Stack Trace:
[1] IF20_cve_search(): ./installer/IF20_cve_search.sh, line 203 -> IF20_cve_search
[2] main(): ./installer.sh, line 358 -> main -d
Important: Consider filling out a bug report at https://github.com/e-m-b-a/emba/issues
------------------------------------------------------------------------------------------------
Could you tell us which OS you are using and when you updated your local git repo?
Ubuntu Linux 23.10.
I have also seen multiple instabilities of the NIST API today. Please try to update your cve database manually or use the latest EMBA master for another try. EMBA now tries the update multiple times ....
Dropped the entire MongoDB for good measure and re-tried, exactly the same epss error today.
Here is a bit of context for the mongodb.log if that helps?:
{"t":{"$date":"2023-11-24T10:18:01.431+11:00"},"s":"I", "c":"INDEX", "id":20440, "ctx":"conn107","msg":"Index build: waiting for index build to complete","attr":{"buildUUID":{"uuid":{"$uuid":"d1b8bd03-8aaf-4c23-a844-2e90d7ec33eb"}},"deadline":{"$date":{"$numberLong":"9223372036854775807"}}}}
{"t":{"$date":"2023-11-24T10:18:01.432+11:00"},"s":"I", "c":"INDEX", "id":20447, "ctx":"conn107","msg":"Index build: completed","attr":{"buildUUID":{"uuid":{"$uuid":"d1b8bd03-8aaf-4c23-a844-2e90d7ec33eb"}}}}
{"t":{"$date":"2023-11-24T10:18:01.432+11:00"},"s":"I", "c":"INDEX", "id":20438, "ctx":"conn107","msg":"Index build: registering","attr":{"buildUUID":{"uuid":{"$uuid":"7411fb82-d670-4e21-bf66-4b8caa24af3f"}},"namespace":"cvedb.cves","collectionUUID":{"uuid":{"$uuid":"e106ee13-22d5-4025-844b-c9353f6328fb"}},"indexes":1,"firstIndex":{"name":"epss"}}}
{"t":{"$date":"2023-11-24T10:18:01.437+11:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"conn107","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"cvedb.cves","index":"epss","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2023-11-24T10:18:01.437+11:00"},"s":"I", "c":"INDEX", "id":20440, "ctx":"conn107","msg":"Index build: waiting for index build to complete","attr":{"buildUUID":{"uuid":{"$uuid":"7411fb82-d670-4e21-bf66-4b8caa24af3f"}},"deadline":{"$date":{"$numberLong":"9223372036854775807"}}}}
{"t":{"$date":"2023-11-24T10:18:01.437+11:00"},"s":"I", "c":"INDEX", "id":20447, "ctx":"conn107","msg":"Index build: completed","attr":{"buildUUID":{"uuid":{"$uuid":"7411fb82-d670-4e21-bf66-4b8caa24af3f"}}}}Welp, later on during an:
emba$ ./external/cve-search/sbin/db_updater.py -f
I was greeted with:
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=420000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=420000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=430000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=430000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=440000 failed....
2023-11-24 10:40:32,657 - CveXplore.database.maintenance.Sources_process - ERROR - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cpes/2.0/?startIndex=440000 failed....
And that's after registering a NIST API key (not anonymous)... I'd guess that they'd be more lenient with registered API key users?
Sigh, anyway, I guess I'll try another day :-S
Small observation... why are those messages repeated twice for every action?:
emba$ ./external/cve-search/sbin/db_updater.py -f
2023-11-24 10:42:56,305 - DBUpdater - INFO - Dropping metadata
2023-11-24 10:42:56,320 - CveXplore.database.maintenance.main_updater - INFO - Starting Database initialization....
2023-11-24 10:42:56,320 - CveXplore.database.maintenance.main_updater - INFO - Starting Database initialization....
2023-11-24 10:42:56,325 - CveXplore.database.maintenance.Sources_process - INFO - CPE Database population started
2023-11-24 10:42:56,325 - CveXplore.database.maintenance.Sources_process - INFO - CPE Database population started
2023-11-24 10:42:56,326 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-24 10:42:56,326 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-24 10:43:17,086 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 1199455 CPE entries
2023-11-24 10:43:17,086 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 1199455 CPE entries
Downloading and processing content: 0%| | 0/1199455 [00:00<?, ?it/s]
Adding a print where it fails to see the contents of cls.SOURCES:
cls.SOURCES is: {'cwe': 'https://cwe.mitre.org/data/xml/cwec_latest.xml.zip', 'capec': 'https://capec.mitre.org/data/xml/capec_latest.xml', 'via4': 'https://www.cve-search.org/feeds/via4.json'}
Context:
emba$ ./external/cve-search/sbin/db_updater.py -f
(...)
2023-11-24 21:28:45,632 - CveXplore.database.maintenance.Sources_process - INFO - VIA4 database update started
2023-11-24 21:28:45,632 - CveXplore.database.maintenance.Sources_process - INFO - VIA4 database update started
2023-11-24 21:28:45,633 - CveXplore.database.maintenance.Sources_process - INFO - Downloading files (max 10 workers)
2023-11-24 21:28:45,633 - CveXplore.database.maintenance.Sources_process - INFO - Downloading files (max 10 workers)
Downloading files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [03:41<00:00, 221.42s/it]
Processing downloaded files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02<00:00, 2.41s/it]
Transferring queue to database: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [11:46<00:00, 47.12s/it]
2023-11-24 21:44:20,594 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:15:34.961643
2023-11-24 21:44:20,594 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:15:34.961643
2023-11-24 21:44:20,691 - CveXplore.database.maintenance.Sources_process - INFO - Finished VIA4 database update
2023-11-24 21:44:20,691 - CveXplore.database.maintenance.Sources_process - INFO - Finished VIA4 database update
getFeedURL source argument contains: epss
and cls.SOURCES is: {'cwe': 'https://cwe.mitre.org/data/xml/cwec_latest.xml.zip', 'capec': 'https://capec.mitre.org/data/xml/capec_latest.xml', 'via4': 'https://www.cve-search.org/feeds/via4.json'}
Traceback (most recent call last):
File "/home/rvalls/dev/personal/emba/./external/cve-search/sbin/db_updater.py", line 122, in <module>
cvex.database.initialize()
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 187, in initialize
self.update()
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 72, in update
up = source["updater"]()
^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 1051, in __init__
self.feed_url = Configuration.getFeedURL(self.feed_type.lower())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rvalls/dev/personal/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/common/config.py", line 114, in getFeedURL
return cls.SOURCES[source]
~~~~~~~~~~~^^^^^^^^
KeyError: 'epss'
@m-1-k-3 Would it make sense to download the EPSS part(s) from FIRST instead of NIST or are there disparities between those two sources of information?:
I have reopened the original issue here #725
Check on this #913
Seems to work @m-1-k-3, analysis finished after 19h (:-!) and 12 cores with this on the MIPS router firmware:
(...)
[+] Identified 1274 CVE entries.](file:///home/rvalls/dev/personal/emba/results/html-report/f20_vul_aggregator.html#anchor_collectcveandexploitdetails)
Identified 981 High rated CVE entries / Exploits: 114
Identified 249 Medium rated CVE entries / Exploits: 7
Identified 44 Low rated CVE entries /Exploits: 41
[ 162 possible exploits available (63 Metasploit modules).](file:///home/rvalls/dev/personal/emba/results/html-report/f20_vul_aggregator.html#anchor_minimalreportofexploitsandcves)
Remote exploits: 47 / Local exploits: 41 / DoS exploits: 32 / Github PoCs: 0 / Known exploited vulnerabilities: 0 / Verified Exploits: 0
Intense use of disk space, btw:
results$ du -hs .
6.1G .
s99_grepit module is very disk intense. If you are runnning the default profile this module should not run. The firmware is also stored and could be removed automatically via the -r option