Can't use this on production with unicorn!

Question

Can't use this on production with unicorn!

zeitnot opened this issue 9 years ago · comments

This gem is working like a charm on localhost, but I could not use it on production which served by unicorn web server. How can use it easily on production?

Adam Crownoble · Answer 1 · Thu Dec 10 2015 11:00:42 GMT+0800 (China Standard Time)

I can't think of any way in which Turnout would work differently in production than it would in development.

Could you give me some more information, maybe some code examples, of how you are using it and have it configured?

Adam Crownoble · Answer 2 · Sun Feb 28 2016 14:53:28 GMT+0800 (China Standard Time)

No activity on this for a while so I'm going to close it. Let me know if you have any more info and I'll reopen it.

Grzegorz Kaczorek · Answer 3 · Tue May 10 2016 17:27:37 GMT+0800 (China Standard Time)

I believe I might just have the same issue. Tournout is working fine on my local enviroment, but it is not doing anything in the production setup.

First thing to note: this is not dependent on the RAILS_ENV, so both of the below work fine:

bundle exec unicorn -p 3000 -c ./config/unicorn.rb
RAILS_ENV=production bundle exec unicorn -p 3000 -c ./config/unicorn.rb

Second: My production setup is a complicated kubernetes/docker based setup which is hard to replicate locally and share here.

Third: When I ssh into the machine and run the rake tasks, the file is created and the command works fine. I can cat the file and so on. The only problem is that the server doesn't really do anything in response to the file.

I realize that this is impossible to solve for you so my question is: is there any verbose mode is debug trick that I could use to determine what the problem might be?

 # unicorn.rb
worker_processes(ENV['UNICORN_WORKERS'].nil? ? 2 : ENV['UNICORN_WORKERS'].to_i)
timeout 120
preload_app true

before_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
    Rails.logger.info('Disconnected from ActiveRecord')
  end

  sleep 1
end

after_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
    Rails.logger.info('Connected to ActiveRecord')
  end
end

Adam Crownoble · Answer 4 · Wed May 11 2016 17:24:22 GMT+0800 (China Standard Time)

If it's writing the file correctly but you're not getting the maintenance page then it's probably having trouble finding the maintenance file. That is most likely due to it having trouble determining what the app root is.

I'd recommend doing something like this:

Turnout.configure do |config|
  config.named_maintenance_file_paths = {test: '/tmp/turnout_test.yml'}
end

Then run touch /tmp/turnout_test.yml and see if it triggers maintenance mode. Let me know how it goes I'd like either write a fix or add some documentation for any of these sort of problems. Thanks.

Grzegorz Kaczorek · Answer 5 · Wed May 11 2016 20:39:10 GMT+0800 (China Standard Time)

Tried exactly as stated but still will not work (tried both with rake maintenance:test:start and touch /tmp/turnout_test.yml (the file was written correctly even in the first case).

My exact config is:

Turnout.configure do |config|
  config.default_reason = 'An error occurred in the application and your page could not be served.</br>Please try again in a few moments.'
  # config.maintenance_pages_path = config.app_root.join('public').join('maintenance').to_s
  config.named_maintenance_file_paths = {test: '/tmp/test.yml'}
  config.default_maintenance_page = Turnout::MaintenancePage::HTML
  config.default_allowed_paths = []
  config.default_retry_after = 600
end

Adam Crownoble · Answer 6 · Wed May 11 2016 20:41:14 GMT+0800 (China Standard Time)

Hmm. Could you run rake middleware in production for me and post the output here?

Grzegorz Kaczorek · Answer 7 · Wed May 11 2016 22:04:16 GMT+0800 (China Standard Time)

use Rack::Deflater
use Rack::Cors
use ActionDispatch::Static
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use RequestStore::Middleware
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::DebugExceptions
use Rollbar::Middleware::Rails::RollbarMiddleware
use ActionDispatch::RemoteIp
use ActionDispatch::Callbacks
use ActiveRecord::ConnectionAdapters::ConnectionManagement
use ActiveRecord::QueryCache
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use ActionDispatch::ParamsParser
use Rack::Head
use Rack::ConditionalGet
use Rack::ETag
use Warden::Manager
use PDFKit::Middleware
use ActionDispatch::Flash
use PDFKit::Middleware
use Rack::Turnout
run App::Application.routes

Adam Crownoble · Answer 8 · Wed May 11 2016 22:10:51 GMT+0800 (China Standard Time)

Strange. Everything seems to be fine so far. Like it really should be working.

One other thing you could do to narrow down the problem is to create the tmp/maintenance.yml and /tmp/test.yml files, then start up a production environment console and run Turnout::MaintenanceFile.find. That should at least tell us if it's finding the maintenance files or not.

Sorry this is turning out to be a hassle. And thanks for sticking with this. Hopefully we'll get to the root of the problem soon.

Grzegorz Kaczorek · Answer 9 · Wed May 11 2016 22:19:35 GMT+0800 (China Standard Time)

Created both files (the second one by touch tmp/maintenance.yml).

irb(main):003:0> Turnout::MaintenanceFile.find                          
=> #<Turnout::MaintenanceFile:0x0055f2b7818008 @path="/tmp/test.yml", @reason="An error occurred in the application and your page could not be served.</br>Please try again in a few moments.", @allowed_paths=[], @allowed_ips=[], @response_code=503, @retry_after=600>

Thanks for the help so far Adam, you're a top-notch OSS maintainer!

Adam Crownoble · Answer 10 · Wed May 11 2016 23:04:52 GMT+0800 (China Standard Time)

Well it looks like the rake task is working, the middleware is loading and the maintenance file is being found and parsed. The next place to look would probably be the Turnout::Request#allowed? method. I looked through it and I can't see any reason you'd get a false positive with it but I'm pretty much out of other ideas so it's probably the best place to look next.

You'll need to make some changes to the code so you can either make edits in the gem code itself or you can monkey patch it by including something like the following in config/initializers/

module Turnout
  class Request
    def allowed?(settings)
      false
    end
end

Also, another option just occurred to me. Are you running any sort of load balancing for your app? Often times load balancers will pick up on a 503 error and automatically fail over to another instance. You could try setting response_code: 200 in one of the .yml files and see if it works.

Grzegorz Kaczorek · Answer 11 · Thu May 12 2016 22:38:16 GMT+0800 (China Standard Time)

Am I right in assuming that the above should trigger maintenance mode regardless of file state?

If so - then it's not working even on localhost. I've put it into the turnout config initializer:

# config/initializers/turnout.rb

Turnout.configure do |config|
  config.default_reason = 'An error occurred in the application and your page could not be served.</br>Please try again in a few moments.'
  # config.maintenance_pages_path = config.app_root.join('public').join('maintenance').to_s
  config.named_maintenance_file_paths = {test: '/tmp/test.yml'}
  config.default_maintenance_page = Turnout::MaintenancePage::HTML
  config.default_allowed_paths = []
  config.default_retry_after = 600
end

module Turnout
  class Request
    def allowed?(settings)
      false
    end
  end
end

Adam Crownoble · Answer 12 · Fri May 13 2016 22:22:30 GMT+0800 (China Standard Time)

Sorry it took me a while to get to this. I had a tight deadline I was working on.

There are two things that keep a maintenance page from being shown. One is a missing maintenance.yml file. The other is a request that's allowed because of it's IP or path. This is taking the IP or path part out of the equation. So you'll still have to have a maintenance.yml for it to find.

If it's still not working for you, let me know. It may be a legitimate Unicorn issue. I may need to fire up an environment to debug this.

Grzegorz Kaczorek · Answer 13 · Tue May 17 2016 00:22:49 GMT+0800 (China Standard Time)

Ok tested this. Still not working. Any ideas on how to further debug this?

Adam Crownoble · Answer 14 · Tue May 17 2016 10:35:24 GMT+0800 (China Standard Time)

I'm honestly pretty confused about what it could possibly be at this point. I think my next step is to setup an environment with Unicorn and see if I can reproduce it. I can't promise I'll have time to get to it too soon. If you'd like to try to push a repository that can reproduce this problem, that would be very helpful. If not I'll get to it when I can.

Rada Bogdan · Answer 15 · Thu May 19 2016 16:24:53 GMT+0800 (China Standard Time)

I think you're touching the wrong file. Your setup says that the file that should be touched to enable maintenance is /tmp/test.yml and not the maintenance.yml from application directory.

So please try to do the following:

cd /tmp
touch test.yml

I tested the exact configuration you have with Unicorn and a Sinatra application and it works, just that you probably were using the tmp directory from within the application, which was wrong, since your configuration says /tmp directory.

In case you want though to use the tmp directory from within application please consider doing the following

  config.named_maintenance_file_paths = {test: config.app_root.join('tmp', 'test.yml').to_s }

Hope it helps. For me works perfectly in both situations. I tested it with the 2.3.0 version of turnout.

It could be possible though you're using a older version?

If you're using thought the latest version i think the only problem there is with whitelisting ip addresses.

Grzegorz Kaczorek · Answer 16 · Tue Jun 07 2016 20:41:08 GMT+0800 (China Standard Time)

@bogdanRada Sorry for the delay - I was on holiday.

Please note that on localhost this also works for me without a hitch.
The issue is not with the filepaths - I've tried both configurations (relative, absolute) with touching all the filepaths I could think of each time.
Version is 2.3.0

I'm leaning towards that this is something related to docker with different processes seeing different states of the filesystem.

Rada Bogdan · Answer 17 · Tue Jun 07 2016 22:13:05 GMT+0800 (China Standard Time)

i personally haven't tested it with Docker, but i did tested your exact configuration with a Sinatra aplication and Unicorn web-server and having a load balancer. and for me it works fine.

Maybe if you show me how you use Docker to enable the maintenance mode, i might be able to help.

Rada Bogdan · Answer 18 · Thu Jun 23 2016 14:40:30 GMT+0800 (China Standard Time)

maybe can you provide more details on how you have setup docker? That would probably help us identify the problem.

Adam Crownoble · Answer 19 · Sat Sep 17 2016 18:13:29 GMT+0800 (China Standard Time)

There hasn't been any activity on this for a while so I'm going to close it. If Anybody has any new information I'll be happy to reopen it.