Portainer with reverse proxy (caddy) can't connect to edge agent but gets heartbeats

Question

Portainer with reverse proxy (caddy) can't connect to edge agent but gets heartbeats

BitWuehler opened this issue 2 years ago · comments

Since a while I try now to connect my portainer instance at home to the home server at my mothers house. For that I red, the safest way to use is edge agent.

At home I run portainer behind a caddy reverse proxy on a raspberry pi 4.
Caddy is configured to route https://portainer.mydomain.de:443 to 192.168.178.3:9233 and tcp://portainer.mydomain.de:8000 to port 192.168.178.3:8111. In Docker I configured, 8111:8000 and 9233:9000 in the portainer compose file on my server.
I opened up port 8000 and 443 in my router (tcp and udp). Also in ufw I allowed port 443 and 8000.

Portainer is working well so far.

On my mothers server I opened up port 9001 in the router. Ufw is also configured so far.

Now I tried to set up edge agent. I used:

sudo docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  -v /:/host \
  -v portainer_agent_data:/data \
  --restart always \
  -e EDGE=1 \
  -e EDGE_ID=----------------------------------- \
  -e EDGE_KEY=-------------------------------------------------------------- \
  -e EDGE_INSECURE_POLL=1 \
  --name portainer_edge_agent \
  portainer/agent:2.15.0

I now can see a heartbeat under Environments but if I try to connect it says Failed loading environment Environment is unreachable.

The portainer logs say:

time="2022-09-15T22:18:25+02:00" level=info msg="2022/09/15 22:18:25 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 7.157875] [message: environment tunnel monitoring]"
time="2022-09-15T22:18:28+02:00" level=info msg="2022/09/15 22:18:28 http: proxy error: dial tcp 127.0.0.1:51018: connect: connection refused"
time="2022-09-15T22:23:30+02:00" level=info msg="2022/09/15 22:23:30 http error: Unable to find the container (err=Error: No such container: 3bfdd889277c8539ed7f13f4df61339c6821c53ad3a5a404730793545eab88c6) (code=404)"
time="2022-09-15T22:23:30+02:00" level=info msg="2022/09/15 22:23:30 http error: Unable to find the container (err=Error: No such container: dae984b1b0af5e2ab7d8a7d4a8f4d04f8d278091412641c87250d3700a5d10dd) (code=404)"
time="2022-09-15T22:34:45+02:00" level=info msg="2022/09/15 22:34:45 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 3.817940] [message: environment tunnel monitoring]"
time="2022-09-15T22:34:50+02:00" level=info msg="2022/09/15 22:34:50 http: proxy error: dial tcp 127.0.0.1:64692: connect: connection refused"
time="2022-09-15T22:37:04+02:00" level=info msg="2022/09/15 22:37:04 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 8.822090] [message: environment tunnel monitoring]"
time="2022-09-15T22:37:05+02:00" level=info msg="2022/09/15 22:37:05 http: proxy error: dial tcp 127.0.0.1:55147: connect: connection refused"
time="2022-09-15T22:41:24+02:00" level=info msg="2022/09/15 22:41:24 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: REQUIRED] [status_time_seconds: 0.182232] [message: environment tunnel monitoring]"
time="2022-09-15T22:41:34+02:00" level=info msg="2022/09/15 22:41:34 http: proxy error: dial tcp 127.0.0.1:65013: connect: connection refused"
time="2022-09-15T23:12:44+02:00" level=info msg="2022/09/15 23:12:44 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: REQUIRED] [status_time_seconds: 1.361693] [message: environment tunnel monitoring]"
time="2022-09-15T23:12:53+02:00" level=info msg="2022/09/15 23:12:53 http: proxy error: dial tcp 127.0.0.1:60140: connect: connection refused"
time="2022-09-15T23:13:34+02:00" level=info msg="2022/09/15 23:13:34 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 2.269864] [message: environment tunnel monitoring]"
time="2022-09-15T23:13:41+02:00" level=info msg="2022/09/15 23:13:41 http: proxy error: dial tcp 127.0.0.1:61949: connect: connection refused"
time="2022-09-15T23:14:34+02:00" level=info msg="2022/09/15 23:14:34 [DEBUG] [chisel,monitoring] [endpoint_id: 8] [status: ACTIVE] [status_time_seconds: 2.271884] [message: environment tunnel monitoring]"
time="2022-09-15T23:14:41+02:00" level=info msg="2022/09/15 23:14:41 http: proxy error: dial tcp 127.0.0.1:60159: connect: connection refused"

The Agent logs:

2022/09/16 08:59:53 [INFO] [main] [message: Agent running on Docker platform]
2022/09/16 08:59:53 [INFO] [edge] [message: Edge key loaded from options]
2022/09/16 08:59:53 [INFO] [edge,registry] [message: Starting registry credential server]
2022/09/16 08:59:53 [INFO] [http] [server_addr: 172.01.02.03] [server_port: 9001] [use_tls: false] [api_version: 2.15.0] [message: Starting Agent API server]
2022/09/16 09:00:38 client: Connecting to ws://portainer.mydomain.de:8000
2022/09/16 09:00:38 client: Connection error: websocket: bad handshake
2022/09/16 09:00:38 client: Give up
2022/09/16 09:01:38 client: Connecting to ws://portainer.mydomain.de:8000
2022/09/16 09:01:38 client: Connection error: websocket: bad handshake
2022/09/16 09:01:38 client: Give up
2022/09/16 09:02:38 client: Connecting to ws://portainer.mydomain.de:8000
2022/09/16 09:02:38 client: Connection error: websocket: bad handshake
2022/09/16 09:02:38 client: Give up
2022/09/16 09:03:38 client: Connecting to ws://portainer.mydomain.de:8000
2022/09/16 09:03:38 client: Connection error: websocket: bad handshake

I googled a lot, tried a lot but nothing changes something in a better way.
Maybe it could be a problem with caddy? Also here I tried a lot. That's my config at the moment:

portainer.{$DOMAIN}:443 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:9233
}

tcp://portainer.{$DOMAIN}:8000 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:8111
}

And for the sake of completeness here also my portainer docker-compose.yml:

version: '3'

networks:
  caddy:
    external: true

services:
  portainer:
    image: portainer/portainer-ce:latest
    command: -H unix:///var/run/docker.sock
    container_name: portainer
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./portainer-data:/data
    ports:
      - 9233:9000
      - 8111:8000
    networks:
      caddy:
        ipv4_address: 192.168.112.8
        ipv6_address: 2001:ab12::8

Im not sure if it is a problem with the agent, portainer or caddy but I hope some of you has an idea!

Anthony Lapenna · Answer 1 · Fri Sep 16 2022 14:21:02 GMT+0800 (China Standard Time)

Hey @BitWuehler

Do you mind sharing the agent logs with us?

Also have you tried to update the Caddy config to remove the tcp protocol from the Edge specific proxy?

portainer.{$DOMAIN}:8000 {
  tls {$EMAIL}
  reverse_proxy 192.168.178.3:8111
}

I don't think the tcp bit is actually required as the agent will initiate the communications over web socket.

BitWuehler · Answer 2 · Fri Sep 16 2022 16:53:28 GMT+0800 (China Standard Time)

@deviantony

Do you mind sharing the agent logs with us?

Sure! I added it above.

Yes, I tried it without the tcp://. Was just the last state, after I tried a lot.

BitWuehler · Answer 3 · Wed Sep 21 2022 17:52:28 GMT+0800 (China Standard Time)

Nobody an Idea? I will add to main as portainer issue too...

Ma · Answer 4 · Mon Dec 05 2022 01:16:10 GMT+0800 (China Standard Time)

portainer/portainer-compose#24 (comment)

you should look at this