nginx proxy can't connect to service

[Davide Bertola]

I don't get how nginx is supposed to know which address to use to contact the service instance.

In yout example the services run on port 3000 which is not mapped to the host os. The VIRTUAL_HOST env var tells nginx-proxy to contact the service on host localost. In my configuration (docker on osx, all up to date to stable releases) this does not work because the services are not listening in localhost. Infact each container has its own private ip addres that is not known by nginx.

[Vince Tse]

@davibe:
I see the reason why it doesn't work--you may need to set up port-forwarding to get this to work on a Mac. First of all, can you tell me if you are using Docker Machine or Boot2Docker on your Mac? If so, are you using it with VirtualBox or VMWare? If not, is the Docker daemon on a remote host?

[Davide Bertola]

docker-machine + virtual box (i use dinghy)

Imho there is no way nginx proxy container can contact another container (service) using 127.0.0.1
It might work if the other service exposed the port on the host and the proxy was --privileged net=host

or.. i am missing something, which is quite possible :)

[Davide Bertola]

@vincetse ?

[Vince Tse]

I use DM+VB too, but I haven't heard of Dinghy. Very cool, and thanks for introducing me to it. First of all, I am going to share what I do with my DM+VB setup without Dinghy, then move on to the Dinghy setup, so please bear with me.

Docker Machine + VirtualBox without Dinghy

If you use this setup, you will need to add a port-forwarding rule to get curl running with my example working from your laptop. The following command, which assumes your Docker Machine name is local, will forward the Docker Machine's TCP port 8000 to your laptop so that you can curl that port directly. Run it after shutting now the machine, tne restart it after.

# Replace "local" with the name of your Docker Machine name.
VBoxManage modifyvm "local" --natpf1 "web-8000,tcp,127.0.0.1,8000,,8000"

Alternatively, you can also ssh into the Docker Machine host before running curl.

# Again, replace "local" with your Docker machine name
docker-machine ssh local

Docker Machine + VirtualBox with Dinghy

Dinghy uses a similar image as the nginx-proxy image that is used in my sample, so you should be able to reuse its setup to get this going. Here are the high-level steps first.

1. Check out my repo, but use a short name for the directory cos the Dinghy Nginx config, by default, only allows hostnames of less than 32 characters. Using a long directory name will cause the Docker container names to be really long and trip the Nginx setup. This tripped me up a lot.
2. There is no need to start the proxy with the docker-compose up -d proxy command since we can reuse the Dinghy proxy.
3. Update my docker-compose.yml file ane replace localhost with the Dinghy proxy name, dinghy_http_proxy.docker.
4. Instead of curling localhost:3000, run curl dinghy_http_proxy.docker since the Dinghy host listens on TCP port 80.

Here are the steps in detail.

# Check out repo with short name. It is *important* to use a short name.
git clone [e-mail]:vincetse/docker-compose-zero-downtime-deployment.git dc0dd
cd dc0dd

Replace the docker-compose.yml file with this.

app:
  build: service
  restart: always
  ports:
    - "3000"
  environment:
    - VIRTUAL_HOST=dinghy_http_proxy.docker
    - VIRTUAL_PORT=3000

service_a:
  extends:
    service: app

service_b:
  extends:
    service: app

Run the ./deploy-and-restart.sh script, then curl dinghy_http_proxy.docker.

./deploy-and-restart.sh
curl dinghy_http_proxy.docker

Alternatively, you can also use the method above w/o Dinghy.

[Vince Tse]

Hey @davibe, any luck?

[Davide Bertola]

Yes it works. The problem was the project name too long. Now the nginx configuration is generated and it does make sense. Before nginx was trying to contact the services using 127.0.0.1, now it uses the correct internal docker ip.

Now I am looking at the generated nginx configuration and i see that the server_name aginx directive is set to the VIRTUAL_HOST value of the container(s). I think what i want si to have a generic server name so that no matter what host/ip the user will connect to and nginx will proxy the connections to the available service. Any idea ?

[Davide Bertola]

Also, looking at the nginx.conf again i see that all services are added to the "upstream" group. This means that when you activate the second service some connections may still be routed to the first one and will get dropped badly when the first service gets closed. Right ?

The correct thing to do would be to just route connection to the second (newer) container and let the connections to the old one drain before closing it (a timeout is ok for most applications)

[Vince Tse]

Yes, you can definitely set VIRTUAL_HOST to a DNS hostname instead of the Docker container name. I am only illustrating the point with the container name since we don't have a hostname to work with. Assuming you have app.container.com mapped to your Dinghy IP address, you can certainly set VIRTUAL_HOST to app.container.com. Have you review @jwilder's Automated Nginx Reverse Proxy for Docker?

The connection shouldn't get dropped if your application handles signals correctly and doesn't just crash when it gets a SIGTERM. I don't believe my server.sh script handles signals correctly since I didn't add signal-handling in there. It would be ideal if we can get the nginx-proxy to mark upstreams as down so that Nginx will stop sending requests to do that. I think we might be able to get it going figuring out the order to change nginx.tmpl to add down to upstreams that we want to take out of service and force a configure regen, or maybe we can send @jwilder a pull request for nginx-proxy.

[Vince Tse]

@davibe:
I just had a duh moment--you could actually just add a fake hostname to your /etc/hosts file and map it to the Dinghy IP address. Duh. :)

[Davide Bertola]

Ok but can you set nginx so that it answers to ANY address and not a specific virtual host ? something like 0.0.0.0.. I don't want my application to know where its published.

[Vince Tse]

Right now, the only way you can do this with nginx-proxy is to use the VIRTUAL_HOST env var to set the DNS name, which triggers the nginx conf file to be regenerated from the template. You could update the nginx.tmpl to reflect what you want to do.

[Davide Bertola]

Indeed. Which brings us to customizing nginx-proxy again. Ok.

[Vince Tse]

Yeah, unfortunately.

Btw, I also connected with @codekitchen about the long server name issue, and he has already addressed it. I think we should see the issue addressed in a future Dingly release.

[Davide Bertola]

I think it would be easy to modify the proxy so that it answers to any vhost. I think it would be more difficult to stop it from redirecting traffic to the "old" service.

I understand the "old" service may wait for connections to drain before closing. But even if it did that (which may be bad for other reasons) if the proxy keeps routing requests to it it may never shut down :P

[Vince Tse]

I cannot agree more with you. I am actually trying to solve the cutover problem myself today. I'll keep you posted how it goes. :)

[Davide Bertola]

News?

[Vince Tse]

Nothing yet. I went down a rabbit hole trying to get it working 2 days ago, but didn't have any luck cos I misunderstand how Docker events work. Gonna try again this weekend.