When scanning large projects, the memory usage is too large, causing the node process to be stuck
jsmgg opened this issue · comments
Describe the bug
When the number of project code lines reaches 300000, the error of insufficient memory often occurs. The error code is ERR_ WORKER_ OUT_ OF_ MEMORY
To Reproduce
Steps to reproduce the behavior:
Find a huge project, scan it and try it. Pay attention to the changes in the memory occupied by nodes. I see that the peak value exceeds 4GB in my mind.
Expected behavior
It supports real asynchronous scanning and piecemeal scanning of large projects.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: macOs Monterey
- OS Version : 12.4
- NodeJS Version :12.22.12
- jscpd version:3.4.2
Additional context
Add any other context about the problem here.
Hi, thank you for the issue, I can suggest you use custom data store like https://www.npmjs.com/package/@jscpd/leveldb-store