Growing memory usage
mike-burns opened this issue · comments
Hi there. good_job is a cool project. I like the way it uses the Rails hooks.
Anyway, we've been seeing growing memory usage in our app and traced it down to good_job itself (we think). I created a fresh Rails app and only ran good_job, and watched the Resident Set Size (RSS) over 10 minutes. It increased, quickly at first, and then more slowly -- but it kept increasing.
Ruby: 3.1.0p0
Rails: 7.0.8.1
good_job: 3.26.1
My script (GNU/Linux):
#!/bin/sh
pid="$1"
while true; do
printf "%s\t%skb\n" "$(date -Iseconds)" "$(ps -o rss= "$pid")"
sleep 5
done
rails new example -d postgres
&&cd example
bundle add good_job
bin/rails db:create
&&bin/rails db:migrate
bundle exec good_job start
And then in another shell:
ps ax | grep good_job
./memory-watcher $that_pid > minimal-app-memory-use
(I know it's possible to get fancier but that wasn't the goal of the exercise.)
I've attached the full output to this ticket, but in summary:
2024-03-06T12:30:33-08:00 106808kb 2024-03-06T12:30:38-08:00 107024kb 2024-03-06T12:30:43-08:00 107084kb 2024-03-06T12:30:48-08:00 107272kb 2024-03-06T12:30:54-08:00 107300kb 2024-03-06T12:30:59-08:00 107692kb 2024-03-06T12:31:04-08:00 107720kb 2024-03-06T12:31:09-08:00 107940kb 2024-03-06T12:31:19-08:00 108156kb 2024-03-06T12:31:24-08:00 108184kb 2024-03-06T12:31:29-08:00 108444kb 2024-03-06T12:31:34-08:00 108472kb 2024-03-06T12:31:39-08:00 108724kb 2024-03-06T12:31:45-08:00 108780kb 2024-03-06T12:31:50-08:00 108968kb 2024-03-06T12:31:55-08:00 108996kb 2024-03-06T12:32:00-08:00 109376kb 2024-03-06T12:32:05-08:00 109400kb 2024-03-06T12:32:10-08:00 109612kb 2024-03-06T12:32:15-08:00 109652kb 2024-03-06T12:32:20-08:00 109844kb 2024-03-06T12:32:25-08:00 109876kb 2024-03-06T12:32:30-08:00 110212kb 2024-03-06T12:32:36-08:00 110240kb 2024-03-06T12:32:41-08:00 110424kb 2024-03-06T12:32:46-08:00 110452kb 2024-03-06T12:32:51-08:00 110652kb 2024-03-06T12:32:56-08:00 110684kb 2024-03-06T12:33:01-08:00 111016kb 2024-03-06T12:33:06-08:00 111200kb 2024-03-06T12:33:11-08:00 111220kb 2024-03-06T12:33:16-08:00 111416kb 2024-03-06T12:33:21-08:00 111440kb 2024-03-06T12:33:26-08:00 111648kb 2024-03-06T12:33:31-08:00 111808kb 2024-03-06T12:33:37-08:00 112012kb 2024-03-06T12:33:42-08:00 112036kb 2024-03-06T12:33:47-08:00 112228kb 2024-03-06T12:33:52-08:00 112260kb 2024-03-06T12:33:57-08:00 112452kb 2024-03-06T12:34:02-08:00 112616kb 2024-03-06T12:34:07-08:00 112816kb 2024-03-06T12:34:12-08:00 112856kb 2024-03-06T12:34:17-08:00 113016kb 2024-03-06T12:34:28-08:00 113188kb 2024-03-06T12:34:48-08:00 113448kb 2024-03-06T12:35:29-08:00 113712kb 2024-03-06T12:35:59-08:00 113976kb 2024-03-06T12:36:30-08:00 114240kb 2024-03-06T12:41:00-08:00 114504kb
Download the full 11-ish minutes (133 lines): minimal-app-memory-use.txt
That's weird! I can try setting that up and doing some heap dumps on that and give an explanation.
Here's some notes from my previous explorations if you wanted to try it yourself on your own app.: https://island94.org/2024/01/the-answer-is-in-your-heap-debugging-big-rails-memory
Hmm, I'm having trouble reproducing this. Albeit I'm using slightly different versions (Ruby 3.2.2, Rails 7.0.8.1)
I ran it for an hour and did some sampling and I only seem memory of:
409374016
409374208
409374656
409117776
409117776
409121968
When you were doing your sampling, were you running with RAILS_ENV=production
, specifically code autoloading disabled / eager loading enabled? That would definitely affect your local debugging/profiling. (I ran my code in production mode).
But none of that is to disbelieve that you're seeing bad memory growth in production. I can totally believe that, though can't pinpoint the cause without more digging.
Here's my calendar if you wanted to look at it together: https://calendly.com/bensheldon/office-hours
My ask to prepare would be:
- Read the blog post about digging through the heap
- Installing derailed_benchmarks
- having a local
bin/profile
file (described in the blog post) with all of your environment variables/configuration to be able to runbin/profile rails s
and have a version of your app boot locally
Thanks for offering to chat about this. I've set up bin/profile
and scheduled some time.
I can confirm that with RAILS_ENV=production
, the memory usage does level off. We are still seeing suspicious memory usage on the actual production job service so we might need help tracking it down.
Closing since the memory issue is likely elsewhere in our app. We did not successfully track it down, though.
Bummer that you couldn't track down the culprit