Redis daemonized containers cannot see each other.
Muzietto opened this issue · comments
I am unable to run the redis-nodejs example because the daemonized containers cannot see each other: redis_replica errors with:
[10] 19 Sep 15:57:21.915 * Connecting to MASTER redis-master:6379
[10] 19 Sep 15:57:20.913 # Unable to connect to MASTER: No such file or directory
By narrowing down the case, I have eventually found out that on my laptop daemonized containers (i.e. launched with -d) cannot ever see each other. Interactive containers (i.e. launched with -ti) can.
All container have their IP neatly configured and visible with sudo docker network inspect express; therefore I can attach a daemonized container, start a shell on it and update manually its /etc/hosts. After that everything is hunky dory, but of course this is not a passable solution in the real world.
Did I encounter a problem due to a new version of the networking software?
Is there something wrong with my Docker installation? It is plain 1.12.1, build 23cf638.
Please assist. Thank you.
I'll try to look at the issue soon but I am a bit busy right now. But I see no reason why this shouldn't work. There is NO difference between -ti and -d in this case.
Can you show me the exact commands and output from each that you are running please?
Thank you for your support.
Dockerfiles are in https://github.com/Muzietto/dockerfiles/tree/master/node-redis
Commands go as follows:
$ sudo docker run -d -h redis-primary --net express --name redis_primary muzietto/redis_primary
e2cbd4e741b92e27f999e0035aad4534692fe918d11d305a53d0dab693ad584d
$ sudo docker run -ti --rm --volumes-from redis_primary ubuntu cat /var/log/redis/redis-server.log
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.0.7 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1:M 22 Sep 12:11:28.418 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 22 Sep 12:11:28.418 # Server started, Redis version 3.0.7
1:M 22 Sep 12:11:28.418 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 22 Sep 12:11:28.418 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 22 Sep 12:11:28.418 * The server is now ready to accept connections on port 6379
$ sudo docker run -d -h redis-replica1 --net express --name redis_replica1 muzietto/redis_replica
f4e3709aec77765267fbc224af225420ed0aaae3bd3a520a4ba0156b4e043103
$ sudo docker run -ti --rm --volumes-from redis_replica1 ubuntu cat /var/log/redis/redis-replica.log
1:S 22 Sep 12:16:59.298 * Connecting to MASTER redis-primary:6379
1:S 22 Sep 12:16:59.366 # Unable to connect to MASTER: No such file or directory
1:S 22 Sep 12:17:00.369 * Connecting to MASTER redis-primary:6379
1:S 22 Sep 12:17:00.425 # Unable to connect to MASTER: No such file or directory
1:S 22 Sep 12:17:01.427 * Connecting to MASTER redis-primary:6379
1:S 22 Sep 12:17:01.535 # Unable to connect to MASTER: No such file or directory
1:S 22 Sep 12:17:02.536 * Connecting to MASTER redis-primary:6379
1:S 22 Sep 12:17:02.604 # Unable to connect to MASTER: No such file or directory
1:S 22 Sep 12:17:03.606 * Connecting to MASTER redis-primary:6379
1:S 22 Sep 12:17:03.665 # Unable to connect to MASTER: No such file or directory
1:S 22 Sep 12:17:04.667 * Connecting to MASTER redis-primary:6379
1:S 22 Sep 12:17:04.713 # Unable to connect to MASTER: No such file or directory
1:S 22 Sep 12:17:05.715 * Connecting to MASTER redis-primary:6379
1:S 22 Sep 12:17:05.783 # Unable to connect to MASTER: No such file or directory
$ sudo docker network inspect express
[
{
"Name": "express",
"Id": "70a8cab21d4ddea5ec88fd5067a01b265070cb8b7c6636bd04ed77331cf27003",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.20.0.0/16",
"Gateway": "172.20.0.1/16"
}
]
},
"Internal": false,
"Containers": {
"e2cbd4e741b92e27f999e0035aad4534692fe918d11d305a53d0dab693ad584d": {
"Name": "redis_primary",
"EndpointID": "41f734a65d8f528bcebde16b903306297b36a95d684f4050e16668bc098cda31",
"MacAddress": "02:42:ac:14:00:02",
"IPv4Address": "172.20.0.2/16",
"IPv6Address": ""
},
"f4e3709aec77765267fbc224af225420ed0aaae3bd3a520a4ba0156b4e043103": {
"Name": "redis_replica1",
"EndpointID": "33638d4e88c7e7e245cda339d62b64ef2115b4ceee157001ec8ae3854b8bfbef",
"MacAddress": "02:42:ac:14:00:03",
"IPv4Address": "172.20.0.3/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
$ sudo docker exec -ti redis_replica1 /bin/bash
root@redis-replica1:/# ping redis-primary
ping: unknown host redis-primary
root@redis-replica1:/#
root@redis-replica1:/# ping 172.20.0.2
PING 172.20.0.2 (172.20.0.2) 56(84) bytes of data.
64 bytes from 172.20.0.2: icmp_seq=1 ttl=64 time=0.211 ms
64 bytes from 172.20.0.2: icmp_seq=2 ttl=64 time=0.077 ms
64 bytes from 172.20.0.2: icmp_seq=3 ttl=64 time=0.115 ms
64 bytes from 172.20.0.2: icmp_seq=4 ttl=64 time=0.126 ms
^C
--- 172.20.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2997ms
rtt min/avg/max/mdev = 0.077/0.132/0.211/0.049 ms
root@redis-replica1:/#
NB - containers are able to ping because I added one RUN instruction in the Dockerfile of redis_base.
Works perfectly for me on OSX:
docker run -d -h redis-primary \
--net express --name redis_primary jamtur01/redis_primary
docker network inspect express
[
{
"Name": "express",
"Id": "941dc5d99720b5e8cac678b433dfa03eb5d67b3ae54a121866a3e19e55bd7f41",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1/16"
}
]
},
"Internal": false,
"Containers": {
"23bc810a742cfd1a2d76a2636c824a56172842104a13c83bb6c3ba7d1bfd891a": {
"Name": "redis_primary",
"EndpointID": "e476b40a0ffec8312d21f196a5640d3ddec4804ee2a5fa30ad692630fc1fc22f",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
docker run -ti --rm --volumes-from redis_primary \
ubuntu cat /var/log/redis/redis-server.log
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.0.7 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1:M 22 Sep 13:08:56.453 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 22 Sep 13:08:56.460 # Server started, Redis version 3.0.7
1:M 22 Sep 13:08:56.460 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 22 Sep 13:08:56.460 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 22 Sep 13:08:56.460 * The server is now ready to accept connections on port 6379
docker run -d -h redis-replica1 \ 3036ms
--name redis_replica1 \
--net express \
jamtur01/redis_replica
docker run -ti --rm --volumes-from redis_replica1 \ ubuntu cat /var/log/redis/redis-replica.log
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.0.7 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1:S 22 Sep 13:10:14.865 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:S 22 Sep 13:10:14.865 # Server started, Redis version 3.0.7
1:S 22 Sep 13:10:14.865 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:S 22 Sep 13:10:14.865 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:S 22 Sep 13:10:14.865 * The server is now ready to accept connections on port 6379
1:S 22 Sep 13:10:14.865 * Connecting to MASTER redis_primary:6379
1:S 22 Sep 13:10:14.996 * MASTER <-> SLAVE sync started
1:S 22 Sep 13:10:14.997 * Non blocking connect for SYNC fired the event.
1:S 22 Sep 13:10:14.997 * Master replied to PING, replication can continue...
1:S 22 Sep 13:10:14.997 * Partial resynchronization not possible (no cached master)
1:S 22 Sep 13:10:14.998 * Full resync from master: b6d5e0d68c4650d7b48693634c5c5d65906d23b5:1
1:S 22 Sep 13:10:15.115 * MASTER <-> SLAVE sync: receiving 18 bytes from master
1:S 22 Sep 13:10:15.115 * MASTER <-> SLAVE sync: Flushing old data
1:S 22 Sep 13:10:15.115 * MASTER <-> SLAVE sync: Loading DB in memory
1:S 22 Sep 13:10:15.115 * MASTER <-> SLAVE sync: Finished with success
Docker version:
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.7.1
Git commit: 6f9534c
Built: Thu Sep 8 10:31:18 2016
OS/Arch: darwin/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 17:52:38 2016
OS/Arch: linux/amd64
Running that from Docker for Mac.
How are you running Docker?
How do you get that complete output? I am on Ubuntu 16.04; I give sudo docker --version and I get:
Docker version 1.12.1, build 23cf638
Run:
docker version
I've also tested on Ubuntu 16.04 and it works for me also.
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64
:-(
Example with compose at ch. 7 works fine. I'll try to run the nodejs app with it.
Trying with the following docker-compose.yml: still same errors
web:
image: muzietto/nodejs
ports:
- "3000:3000"
redis_primary:
image: muzietto/redis_primary
hostname: redis-primary
redis_replica1:
image: muzietto/redis_replica
hostname: redis-replica1
redis_replica2:
image: muzietto/redis_replica
hostname: redis-replica2
Will try using your very images. I assume they are downloadable from Docker Hub. Let's check...
Using your images I see everything OK.
I am pulling my hair: Dockerfile's are equal. Will study your images better, up to the sources.
Thank you for your patience.