ERRO[3779] {HORUSEC_CLI} Error while running tool HorusecEngine
MarkLee131 opened this issue · comments
What happened: Encountered an error when running the horusec against OWASP benchmark.
What you expected to happen: get the result file by running horusec against OWASP benchmark.
How to reproduce it (as minimally and precisely as possible): you can reproduce it by running horusec against the owasp benchmark, the repo I post on above.
Anything else we need to know?: I think the size of this benchmark causes it, so maybe there can be a solution to scan a large project like this benchmark?
Environment:
- Horusec version (use
horusec version): v2.8.0 - Operating System: WSL2
- Others: the detailed error log on output:
/mnt/e/OSSPERT/BenchmarkJava$ horusec start -p /mnt/e/OSSPERT/BenchmarkJava -t 60000 -o="json" -O=horusec.json
WARN[0000] {HORUSEC_CLI} Config file not found
WARN[0053] {HORUSEC_CLI} When starting the analysis WE SKIP A TOTAL OF 75 FILES that are not considered to be analyzed. To see more details use flag --log-level=debug
WARN[0344] Horusec will return a timeout after 60000 seconds. This time can be customized in the cli settings.
WARN[0344] {HORUSEC_CLI} PLEASE DON'T REMOVE ".horusec" FOLDER BEFORE THE ANALYSIS FINISH! Don’t worry, we’ll remove it after the analysis ends automatically! Project sent to folder in location: [/mnt/e/OSSPERT/BenchmarkJava/.horusec/8907e2ef-e60f-40a1-9194-29b952b33459]
INFO[3779] {HORUSEC_CLI} Writing output JSON to file in the path: /mnt/e/OSSPERT/BenchmarkJava/horusec.json
==================================================================================
WARN[3779] {HORUSEC_CLI} No authorization token was found, your code it is not going to be sent to horusec. Please enter a token with the -a flag to configure and save your analysis
WARN[3779] {HORUSEC_CLI} 26 VULNERABILITIES WERE FOUND IN YOUR CODE SENT TO HORUSEC, TO SEE MORE DETAILS USE THE LOG LEVEL AS DEBUG AND TRY AGAIN
WARN[3779] {HORUSEC_CLI} Horusec not show info vulnerabilities in this analysis, to see info vulnerabilities add option "--information-severity=true". For more details use (horusec start --help) command.
==================================================================================
WARN[3779] {HORUSEC_CLI} During execution we found some problems:
ERRO[3779] {HORUSEC_CLI} Error while running tool HorusecEngine: open /mnt/e/OSSPERT/BenchmarkJava/.horusec/8907e2ef-e60f-40a1-9194-29b952b33459/target/benchmark/xss-04: too many open files
ERRO[3779] {HORUSEC_CLI} Error while running tool HorusecEngine: open /mnt/e/OSSPERT/BenchmarkJava/.horusec/8907e2ef-e60f-40a1-9194-29b952b33459/src/main/webapp/js/js.cookie.js: too many open files
ERRO[3779] {HORUSEC_CLI} Error while running tool HorusecEngine: open /mnt/e/OSSPERT/BenchmarkJava/.horusec/8907e2ef-e60f-40a1-9194-29b952b33459/src/main/java/org/owasp/benchmark/testcode/BenchmarkTest01941.java: too many open files
WARN[3779] {HORUSEC_CLI} Error while running tool YarnAudit: file yarn.lock file was not found in your Javascript project. If you use Yarn to handle your dependencies, it would be a good idea to commit it so Horusec can check for vulnerabilities
WARN[3779] {HORUSEC_CLI} Error while running tool NpmAudit: file package-lock.json was not found in your Javascript project. If you use NPM to handle your dependencies, it would be a good idea to commit it so Horusec can check for vulnerabilities
Hi,
Depending on the size of the project, you need to adjust the maximum number of open file descriptors. For this, the ulimit utility is used.
Thank you. I tried it on a ubuntu device, and modify the open files num with ulimit. And now, it is set to 1048576.
lab@lab-MS-7C82:~/Desktop/BenchmarkJava/.horusec/d7359263-fa85-4d08-91c2-2bb76a623a9c$ ulimit -Hn 1048576 lab@lab-MS-7C82:~/Desktop/BenchmarkJava/.horusec/d7359263-fa85-4d08-91c2-2bb76a623a9c$ ulimit -Sn 1048576
And, the config I customed is as follows:
"horusecCliTimeoutInSecondsAnalysis": 60000, "horusecCliTimeoutInSecondsRequest": 30000,
leaving the others by default.
I think it is enough to scan, but my scan was killed by the horusec, can you give me some guide to keep me from this issue when scanning? Thanks!
lab@lab-MS-7C82:~/Desktop/BenchmarkJava$ sudo horusec start -p ./ - t -o json -O ./horusec.json
time="2022-08-13T11:09:33+08:00" level=warning msg="{HORUSEC_CLI} When starting the analysis WE SKIP A TOTAL OF 80 FILES that are not considered to be analyzed. To see more details use flag --log-level=debug"
time="2022-08-13T11:09:33+08:00" level=warning msg="Horusec will return a timeout after 60000 seconds. This time can be customized in the cli settings."
time="2022-08-13T11:09:33+08:00" level=warning msg="{HORUSEC_CLI} PLEASE DON'T REMOVE \".horusec\" FOLDER BEFORE THE ANALYSIS FINISH! Don’t worry, we’ll remove it after the analysis ends automatically! Project sent to folder in location: [/home/lab/Desktop/BenchmarkJava/.horusec/9a3959ed-3d34-4747-b29f-17ab10b8fb6c]"
⣾ Scanning code ...Killed
@MarkLee131 were you able to fix this issue?
Hhh anyway, thanks for your team!
Actually, I solved it on my new machine according to @xotohop's advice. It seems to be a problem of my previous machine.
Hi @jtabone16. Another thing is, maybe the documentation of Horusec can be improved, I found there are some typos in the docs about its rules. For example, it starts from No.99, with missing No.104 and so on.
https://docs.horusec.io/docs/cli/analysis-tools/open-source-horusec-engine/horusec-java/
@MarkLee131 what did you have to set your ulimit to? and agreed the docs are lacking at times :/
I set them super high and now getting failed to create new OS thread
I checked it. The details are as follows:
lab@****:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 771296
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 771296
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited