High RAM Consumption when activating the WAF on multiple sites
skixmix opened this issue · comments
Hello,
I'm encountering a problem with this module, specifically, the RAM usage keeps rising consistently whenever I incorporate a new Caddyfile configuration (or when I activate the WAF on an existing config).
My main configuration looks like this one:
{ Caddy global options... }
import *.caddy
In each Caddyfile, I manage a distinct site. I have approximately 200 Caddyfiles in total. Prior to installing the module, the loaded configuration utilized merely 1 GB of memory. However, once the module is installed and the WAF directive is enabled like this:
coraza_waf {
include /waf/coraza/coraza.conf-recommended
include /waf/coreruleset/crs-setup.conf.example
include /waf/coreruleset/rules/*.conf
include /var/sites_waf/specific_site_configuration.conf
}
I observe an increment of nearly 200 MB of RAM per site. Upon restarting the Caddy server, the RAM usage amplifies further and appears to remain unreleased. This does not occur when I deactivate the module and the WAF directive.
Do you have any idea on what is happening?
Thank you,
Simone
Hey! Could you provide other metrics, like concurrent users and traffic information? Each WAF instance is just a few KBs of memory, it shouldn't create that much overhead.
It could be related to the garbage collector.
Hello,
Thank you, as of now I'm testing it on a VM running Ubuntu 20.04.6 LTS and not exposed to the internet, so there are no clients and no traffic. On the other VMs in production, I have Caddy without the WAF module installed and the same configurations occupy more or less 1,3 GB of RAM. Whenever I perform a Caddy reload or restart, it takes up some additional memory, goes up to 2GB and then releases it back to 1,3 GB.
This behaviour changes in the test VM with the coraza module installed, and what happens is what I described before.
I've been checking this issue but could not get my head around it.
Locally
When I build the binary and run it in local with the example caddyfile
# run httpbin in terminal 1
go run github.com/mccutchen/go-httpbin/v2/cmd/go-httpbin@v2.9.0 -port 8081
# run caddy in terminal 2, make sure you change the host in caddyfile from httpbin to localhost
mage buildCaddy
./build/caddy run --config example/Caddyfile --adapter caddyfile
# run ab in terminal 3
ab -n 100000 -c 100 http://localhost:8080/I get a caddy server going from 40MB to 93MB (checked with Activity Monitor)
Docker
If I run the same ab command but running the example in a container the memory can get up from 40MB to 5GB and slowly go down to 2.87GB (checked with docker stats), actually every curl localhost:8080/ increments the memory usage in 1.1MB. Both tests are similar, however the local one built the mac version and docker one uses the linux version (GOOS=linux).
# run example
go run mage.go buildExample runExample
# run ab
ab -n 100000 -c 100 http://localhost:8080/
To know for sure, you'll need to capture a memory profile.
You can get one from :2019/debug/pprof on your server and viewing the allocations.
You can use go tool pprof to view them interactively, though sometimes it can be obvious just looking at the raw dump. I like generating an SVG to see what call stacks are allocating the most memory.
Here's an example tutorial (skip the top part that talks about adding the pprof handlers, Caddy already does this for you, hence the pprof endpoint you can load): https://www.freecodecamp.org/news/how-i-investigated-memory-leaks-in-go-using-pprof-on-a-large-codebase-4bec4325e192/
Not really a fair comparison since the two are completely different platforms/OSes. Will need a profile to be sure.
Hello,
Thank you for your support. If it might be useful, what I'm observing is that each time I execute a reboot of the machine or a systemctl restart of the Caddy service, the RAM usage reverts to approximately 1 GB. After this operation, even if I execute multiple 'curl' requests on various sites, it remains stable and the usage does not increase by 1MB as it does in your situation. Additionally, as mentioned previously, I'm observing a rise in RAM utilization whenever I add a new site (or enable the WAF on a site where the directive was previously absent) and subsequently execute the rebuild/reload of Caddy configurations.
Caddy version: v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
Coraza Caddy version: 1.2.2
The VM is operating on vMware, not Docker, and on the same host there are two additional VMs without the Coraza module installed. All VMs share the same Caddy configurations (excluding the WAF directive, which is absent in the production VMs), and the same system:
- OS: Ubuntu 20.04
- RAM: 4 GB (1.4-1.6 GB used in production, same on the test machine, but increasing like said before)
- CPU: 4 vCores (0.15 - 0.20 load average on production, 0.01 on the test machine)
You mean without the site-specific configuration?
Currently, I have this configuration on every site (I'm not truly customizing it, it was simply to have something where I could place a custom configuration if necessary):
# ============================== WAF STATUS ====================================
# Detection Only Mode
SecRuleEngine DetectionOnly
# ============================== LOGGING =====================================
# Enable Audit Engine - use RelevantOnly to avoid logging everything
SecAuditEngine RelevantOnly
# Log file
SecAuditLogStorageDir /waf/coraza/audit/
SecAuditLogFormat JSON
SecAuditLog /waf/coraza/audit/202.log
# Log transaction
SecAuditLogParts ABIFHZ
# Use a single file for logging.
SecAuditLogType Serial
# ======================== DISABLED OWASP RULES =============================
SecRuleRemoveByID 920430
SecRuleRemoveByID 932236
SecRuleRemoveByID 942421
SecRuleRemoveByID 920272
SecRuleRemoveByID 901340
I tried replacing every site coraza_waf config with this one
coraza_waf {
include /waf/coraza/coraza.conf-recommended
include /waf/coreruleset/crs-setup.conf.example
include /waf/coreruleset/rules/*.conf
}
without the site-specific configuration, and the problem persists even after reloading. The initial RAM usage was of 1.64 GB, which increased to 2.33 GB. After a few minutes, it released the memory, bringing it down to 1.88 GB. However, I'm unsure why it continues to retain that extra 200 MB of RAM.
Hello, is there any update on this issue?
Got the profiling (thanks @mholt for the heads up on the endpoint)
I think one issue here is that we are initializing one instance of WAF per site, meaning that regexes and dicts (used by aho-corasick) are being initialized many times even when it is the same content and it is immutable. One idea to overcome this could be to use a regex map and dictionary map, however we need to be careful as tinygo won't support synchronization (hence non concurrent map for tinygo). This way, if you span 100 WAF with CRS, the regexes are going to use the same one in memory cc @anuraaga
I think a map for caching by pattern can work ok. Since TinyGo only runs on one thread, no worries about concurrency. One issue is all regex then are memory leaks - a pattern of replacing the waf instance with a different one with different rules becomes questionable. It's not common so maybe it doesn't need to be supported, but I guess making cache opt-in with a build tag is safer
Hello,
If I got it right, there are two problems here:
- When using the WAF module with multiple sites, it consumes a lot of memory because of regex allocation (duplicate rules saved in separate memory blocks for each site).
- Every time we reload the Caddy configuration, the module appears to use up some RAM and then doesn't release all of it, even if nothing changed in the config.
Concerning the first problem, I can consider allocating additional RAM to the VM as a temporary solution. However, I'm unsure how to address the second issue. In the long run, it will take up all the RAM...
Do you have any updates or suggestions?
Thank you,
Simone
Hello and thank you for the detailed analysis. Indeed we are working on a solution, as described by JC, we will implement cache mechanisms to avoid duplication on dictionaries for pm and regexes for rx. It will take some time to design this as it is a sensitive change. We will use this issue as reference so we will keep you posted.
Multiple hosts in a single web server is an important use case for coraza.
Having a Close is important, SGTM
No, perhaps we could document it better (similar to what wazero does) but our interfaces are not for outside implementations. It's fine to keep that up and we can make it more explicit it needed.
I don't know the exact situation in the code, of course, but there's two simple rules I try to abide by:
- Store all state in the module's struct itself; this will get garbage collected after a config reload. If it doesn't, then not "all state" is being stored in the module's struct.
- If you do need global state, or state that persists across reloads, but is based on configuration that may or may not be needed after a reload, use a
UsagePool: https://pkg.go.dev/github.com/caddyserver/caddy/v2#UsagePool -- often in conjunction with a module implementing the CleanerUpper: https://pkg.go.dev/github.com/caddyserver/caddy/v2#CleanerUpper -- this allows modules to say when they're done with values as configs are cycled through, and Caddy & Go's GC will take care of the rest.
Hello everyone,
thanks for the prompt and diligent responses. Will the upcoming 2.0 module version include a resolution for this issue? If yes, do you have any timeline in mind?
Currently, I have Caddy deployed in production for various sites, with the WAF temporarily disabled. As a workaround, I've implemented a server-side (Node.js Express) WAF. However, I'm eager to integrate Coraza on Caddy as soon as possible.
Thank you,
Simone
Hi,
If you could give this fix higher priority, it would be fantastic 👍
Alternatively, I would be willing to test a custom build tag. If we choose that route, will I need to compile Caddy in a specific manner?
I believe you can do:
XCADDY_GO_BUILD_FLAGS="-ldflags '-w s' -trimpath -tags my_custom_tag" xcaddy build --with github.com/corazawaf/coraza-caddy/v2cc @mholt
Let me know how I can help :)
Store all state in the module's struct itself; this will get garbage collected after a config reload. If it doesn't, then not "all state" is being stored in the module's struct.
Is there any callback for when a module struct is going to be dismissed?
Good question: yes there is! :)
a module implementing the CleanerUpper: https://pkg.go.dev/github.com/caddyserver/caddy/v2#CleanerUpper -- this allows modules to say when they're done with values as configs are cycled through, and Caddy & Go's GC will take care of the rest.
Awesome! Thanks @mholt
@skixmix I opened this PR corazawaf/coraza#836. Please use that commit to give it a try and we see. You don't need to use a build tag.
Use something like:
xcaddy build --with github.com/corazawaf/coraza-caddy/v2@24ab5d92e9d71ebde31a34e0e86c595b568e79d3Great! Thank you so much. I'll give it a try as soon as possible and provide you with an update.
Have a nice day,
Simone
Hello,
I have an update. What I did was to start with a clean Caddy setup, no site configured and no traffic. Then, I added 23 sites one by one, each one with its own Caddyfile and WAF configuration + OWASP CRS.
- RAM consumption before starting: 340MB
- RAM consumption after 10 sites: 1.20 GB
- RAM consumption after 23 sites: 2.06 GB
Throughout the process, I noticed that the RAM usage was having peaks at 3 GB and gradually dropping successively.
- to 1.8 GB (with 10 sites) and
- to 2.8 GB (with 23 sites)
before reaching a stable consumption.
This is what happens when I add a new site or when I perform a reload of a site configuration (in this specific case, I added a new site).
Initial RAM consumption
During configuration reload
After configuration reload
After more or less 3 minutes
After 30 minutes
Production comparison
This is the current state of one of the VMs that I have running on one of my DCs in production, with more than 100 site configurations, same as the test ones, but without the WAF module:
--
Recap
The problem persists in such a manner that each configuration uses over 100 MB of RAM instead of the 8/9 MB used in production without the WAF, which is 10 times higher. Nevertheless, I believe that one of the issues has been resolved as the memory usage now decreases rather than continuously leaking indefinitely. 😄
You can find out what's using memory by capturing a profile. Caddy exposes HTTP endpoints for this at :2019/debug/pprof -- this is a simple HTML document with links to collect various profiles, either instantaneously or over a time period if you add query params. Capture the heap profile then use go tool pprof ... to open the profile; type top to see the top allocations, or svg to generate a vector image you can view in your browser.
According to #76 (comment) we might need to memoize the aho-corasick dictionaries too.