Problem with log rotation
Simbul opened this issue · comments
I've had some very weird crashes while using spawn. I've tested high and low, and all I could find out is that it seems related to log rotation.
A consistent behavior I experienced: when the log rotates, either the parent or the child process becomes unable to write on the new logfile (maybe because of a stale file handler?).
Sometimes this seems connected to the crash, sometimes not. Anyway, this is the backtrace:
Read error: #<NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.each>
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/utils.rb:196:in initialize' /Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/content_length.rb:14:innew'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/content_length.rb:14:in call' /Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/chunked.rb:15:incall'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/handler/mongrel.rb:64:in process' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:159:inorig_process_client'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:158:in each' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:158:inorig_process_client'
/Users/simbul/Sites/spawnlog/vendor/plugins/spawn/lib/patches.rb:61:in process_client' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:inrun'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in initialize' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:innew'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in run' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:ininitialize'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:in new' /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:inrun'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/handler/mongrel.rb:34:in run' /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/commands/server.rb:111 /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:ingem_original_require'
/Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
script/server:3
I'm willing to help with further testing, if needed. At the moment, I don't seem able to make head nor tail of it.
No idea ... what version of rails are you using?
Rails 2.3.5
What are you using to do log rotation? We probably need to close/reopen a log file handler in the child. Is your child running past the time where the log is rolled? Or does it seem to happen in cases where you are spawning something at the log roll time?
It doesn't seem to be connected to processes spawned at log-roll time.
I'm just relying on the standard rails Logger to do rotation. This is what I have in my environment.rb:
config.logger = Logger.new(config.log_path, 10, 10485760)
I created a test application which just spawns a child and logs lines of text (both from the parent and from the child).
With a clean log/ folder, this creates 3 files:
- development.log.1, maxed out at 10M, containing messages from both parent and child
- development.log.0, containing only messages from the child.
- development.log, containing only messages from the parent.
It would seem everything goes smoothly up to the first rotation. Afterwards, each one of the processes takes hold of its own logfile. This is a strange behavior but it doesn't mean the server crashed, in itself.
What seems to be causing a crash is when one of the processes cannot take hold of a logfile. After a crash, the log/ folder contains just 2 files:
- development.log.0, maxed out at 10M, containing messages from both parent and child
- development.log.1, containing only messages from the child.
My guess is that the logger for the parent process cannot write on its file for whatever reason.
Running another request after the one which crashed the server returns the same error as above, plus:
Error during failsafe response: Shifting failed. closed stream
There have been lots of changes on "edge" branch lately. Maybe it's worth trying that branch to see if this problem still exists.
Closing to clean up as this in an old issue, if this is still valid on the latest version of spawnling then please re-open the issue and I will address it.