some error occured when I import the vulnerability database
chenkehao1998 opened this issue · comments
I generated the steady.sh by kaybee, and ran the script .
I found a lot of 404 and 403 in log. The server is run in local. So there is no mistake of the uri.
And after the script run out .
I use mvn -Dsteady compile steady:app to scan .
And I could find the dependencies ,but found there is no vulnerabilities at all.
The project is download at https://github.com/srcclr/efda,which has the vulnerabilities.So the list cannot be empty.
I wonder how to deal with this.
I hope I can get your help!
Thank you very much!
Hi @chenkehao1998
the 403 response codes in the log occur while populating the backend with the vulnerabilities, so that explains why no vulnerabilities are shown for the project you analyzed. However this should not happen provided that you have BACKEND_BUGS_TOKEN configured in docker/.env (see https://eclipse.github.io/steady/admin/tutorials/docker/#setup). In fact haproxy checks that PUT and POST requests for the /bugs endpoint contain such token. Can you check that BACKEND_BUGS_TOKEN is correctly configured in your docker/.env ?
I never modified the docker/.env. I just install the backend as the document(https://eclipse.github.io/steady/admin/tutorials/docker/) said.
The config of BACKEND_BUGS_TOKEN is BACKEND_BUGS_TOKEN=FD262258-9E0E-494F-AB98-10352914D7A6
What should I do to solve this problem?
Could you bash into the container steady-kb-importer docker exec -ti steady-kb-importer bash and check if the environment variable is set according to the .env file 'echo $BACKEND_BUGS_TOKEN' ?
Could you also share where are you running the composition ? Until now it was mainly tested on Ubuntu.
Thank you!
I checked it in container, the environment variable is exist.
And my running environment is ubuntu ,too. (Ubuntu 20.04LTS)
And It's wsl2 which is virtual machine in windows.
I have a server with centos. I run it in centos! The problem is also existed!
Thanks for the additional details. The 403 errors in the screenshot you pasted above are due to the fact that haproxy is expecting a different token than the one passed from the script 'steady.sh' you run.
The easiest way to populate the database is to rely on the container 'steady-kb-importer' that automatically starts the import upon creation. it will import the ~720 vulnerabilities available at https://github.com/SAP/project-kb/tree/vulnerability-data/statements and it will run daily to keep the steady backend up to date with respect to the vulnerabilities available in project-kb. Note that the initial import takes ~1h. So you only need to run the steps [1] and [2] from the documentation and wait ~1h to have all the vulnerabilities imported. The documentation will be updated shortly to reflect this simplified process.
[1] https://eclipse.github.io/steady/admin/tutorials/docker/#setup
[2] https://eclipse.github.io/steady/admin/tutorials/docker/#run
I did deploy as described above. I used Kaybee to manually import data only because the deployed program couldn't scan for vulnerabilities.
Could you check if the import from steady-kb-importer worked as expected running docker logs steady-kb-importer?
It should contain POST requests to http://localhost:8033/backend/bugs (assuming it's already up and running since 1h+). If those are successful you should also be able to see the imported bugs running GET requests to http://localhost:8033/backend/bugs
Hello!
I run docker logs steady-kb-importer checked and found no recent log.
But you said that it will run daily to keep the steady backend up to date with respect to the vulnerabilities available in project-kb.
There is no POST requests to http://localhost:8033/backend/bugs in logs
From docker logs steady-kb-importer you get the logs of the initial import. The logs of the cron job performing the updates (daily by default but configurable from .env) are only accessible inside the container (cron.log).
The script steady.sh is failing even before running the POST requests, while writing the files metadata.json using the cat command. It looks like the issue is related to running the container on windows as we never observed this issue when running the script on 'native' Ubuntu (no WSL). We will investigate further.
Thank you!
As steady-kb-importer is currently using a volume for the /tmp folder, the cat command is returning the error you see in your last screenshot due to the underlying windows filesystem.
If you already want to test the system before we release a final fix, you can checkout the branch docker-win (where we removed the volume) and restart the composition docker-compose -f docker-compose.yml up -d.
Thank you very much! Maybe there are too many trouble with the wsl . Recently, I will not use wsl to do my research.