Bloop off-heap memory usage
q-benwillis opened this issue · comments
We're using Bloop as a build server for a shared UI based application to compile generated code on the fly for multiple users. We've found that as the number of workspaces in Bloop increases we run in to memory issues.
Bloop is running in its own container with 4GB of memory. We've set the maximum heap size for the Bloop JVM to 3GB leaving what we think should be pleanty of space for any off-heap requirements. Unfortuntaly we're seeing memory usage rise above the 4GB limit which leads to the container being killed and restarted. The main thing we've noticed is that the off-heap memory is never garbage collected so seems to grow until it exceeds the limit and gets killed.
We've tried using the cleanCache request to free up memory but that just seems to delete the class files rather than freeing up any of the used memory.
We have a couple of questions around this:
- What could Bloop be using this off-heap memory for and how can we limit it?
- Has anyone else deployed Bloop in a containerized environment in this manner?
Thanks for reporting! I think the off-heap memory is used by the compiler. It needs to load classfiles etc. which is neccessary for the compilation. I've noticed that it can grow quite heavily for Bloop since it has to deal with a lot of jars. I am not sure if we can make it use it less or GC that.
I wonder if setting -XX:MaxPermSize would limit the memory usage?
Or maybe I was suggested that setting MaxMetaspaceSize might help in your case.
Gave MaxMetaspaceSize a go (set it to 256MB) but didn't seem to help. Interesting that you mentioned the compiler using using the off-heap memory. I guess I would have expected the resources to be have freed up once the compilation was completed but seems they are not.
Having read the docs I noticed that there were some recommendations on setting the max heap size. You mentioned that you had also seen the off-heap grow quite heavily for Bloop, do you have a rough idea of how it grew vs the allocated heap? Perhaps we just need to allow for more off heap-usage.
It's hard to say with your specific usage, we usually use it locally and we could limit Bloop heap to even 1 GB without issues. 🤔 From what I understand, that off heap usage should also be freed at some point.
Probably the best scenario would be to profile the memory usage and see which parameter causes it to go out of ram. Or alternatively, limit the heap to 2 GB, maybe that will be enough.