--internal bridge networked containers cannot expose ports
yajo opened this issue · comments
Description
When using an --internal network, the container does not expose the requested ports.
I have a development environment which imitates the production one but uses an --internal network. It's very useful to avoid the inner processes contacting outside world and sending/receiving emails, etc.
However, if I publish with -p 127.0.0.1:80:80 to be able to browse that app in http://localhost, I simply cannot.
Steps to reproduce the issue:
docker network create testnet --internaldocker container run --name testcon --net testnet -dp 127.0.0.1:80:80 stackatoapps/phpinfo
Describe the results you received:
Cannot browse http://localhost.
Describe the results you expected:
I shouldn't be able to connect with the outside world from within testcon (that happens and is 👍), although the outside world should be able to connect testcon's port 80, and in such cases it should be able to answer the call.
If I cannot do that, if I need to un-isolate testcon to be able to develop it, then it makes not much sense the purpose of --internal, since basically means that it is either isolated from everybody (even me), or from nobody. But what if I need it isolated from everybody except me?
Additional information you deem important (e.g. issue happens only occasionally):
Similar to moby/libnetwork#1151 (comment).
docker network inspect testnet
[
{
"Name": "testnet",
"Id": "c952572231efe71ea025df259b0a4f7b1c2d4a03e5cf15cf1e62e839379078a3",
"Created": "2018-02-01T09:19:04.151226862Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": true,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"55cf54b814fe1fc06bdb12e6c065f07938b9f2163e87932ffaf38ada8b4f5ccd": {
"Name": "testcon",
"EndpointID": "c5b9c56d09c9c44034f4d34ab9fa71b895bfa0f15ccf59eb94e6c59df8cfadfb",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {}
}
]
docker container inspect testcon
[
{
"Id": "55cf54b814fe1fc06bdb12e6c065f07938b9f2163e87932ffaf38ada8b4f5ccd",
"Created": "2018-02-01T09:20:06.355916288Z",
"Path": "apache2-foreground",
"Args": [],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 22516,
"ExitCode": 0,
"Error": "",
"StartedAt": "2018-02-01T09:20:07.206854744Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:f999137c00b74486f8f88920e468e252ea40f52d4be4a8d90590610f68fbbb3d",
"ResolvConfPath": "/var/lib/docker/containers/55cf54b814fe1fc06bdb12e6c065f07938b9f2163e87932ffaf38ada8b4f5ccd/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/55cf54b814fe1fc06bdb12e6c065f07938b9f2163e87932ffaf38ada8b4f5ccd/hostname",
"HostsPath": "/var/lib/docker/containers/55cf54b814fe1fc06bdb12e6c065f07938b9f2163e87932ffaf38ada8b4f5ccd/hosts",
"LogPath": "",
"Name": "/testcon",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "system_u:object_r:container_file_t:s0:c203,c924",
"ProcessLabel": "system_u:system_r:container_t:s0:c203,c924",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "journald",
"Config": {}
},
"NetworkMode": "testnet",
"PortBindings": {
"80/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "80"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "shareable",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/2438f1afa6c949e6b54128fbf2e133abb416fb972b32e2db9d37cb4bd85a7c80-init/diff:/var/lib/docker/overlay2/f54fde484502043cc40558cc0bfa6c9006cab8399dbf0a52efa99a66fcc7bc93/diff:/var/lib/docker/overlay2/62fdbcc8cf041a65b4562c7336aaabddacbd961091d9cb3b455868074a1a9eef/diff:/var/lib/docker/overlay2/1acba50e14ae026691090052830a51f5788d92e6fd263ebee0f088ccb33aa2fd/diff:/var/lib/docker/overlay2/2ed2a830ab02e818b74fc88efa6aac5d91e9103cf4ff86c5612034f5456781f9/diff:/var/lib/docker/overlay2/085558dfda79fd2d5ad11dae7f547ea476fc7ee8067978bd537f14f6ef8ee94f/diff:/var/lib/docker/overlay2/18df8065adf8e0bc240940de2a8bd9008945ef1ce9fab7e05c3eea3e43fd2e52/diff:/var/lib/docker/overlay2/583dc112b648d11e584813112039aec9fea36a5ffdb3066937963b98caa77adb/diff:/var/lib/docker/overlay2/50497ea070347fbae46dfb4d620797caedf03771603294acefc8176019d5a07a/diff:/var/lib/docker/overlay2/6dc43a19e7e5edb541bf751dadffe1d762dd5a791c05c494bfba1802e5d4f1c4/diff:/var/lib/docker/overlay2/d66e0e9c039d2ed1f3b964a58048a6dda5f1e7538a76f93782d540bdb995506f/diff:/var/lib/docker/overlay2/740940fcda53a4db9bef64e5b5e87d0639b1c597ee8c46198e57b533f50753a4/diff:/var/lib/docker/overlay2/c66a7d7f82d8a8dd5a0e8d4f190fb040678f071c275d0fc49bf4a7211b0c3c79/diff:/var/lib/docker/overlay2/bd6209a9bf10ea1bbfe5297cd42b19c9a380a4556c5d37e693d418671a21b8ab/diff:/var/lib/docker/overlay2/059b29aba91fcbf5423a3b19ed906fca552ba91b91824c02283d172649302774/diff:/var/lib/docker/overlay2/185379cd7330e5251a35e4099dc1a59b07f13ebda9c859a87fb4bc1072301c5c/diff:/var/lib/docker/overlay2/b9d4a7fe2a9316deb227bab139b9502c0048854dc200e3958afd8e4c60152395/diff:/var/lib/docker/overlay2/312574bcb4321862d75f28efaac53de5f0f7692437e2e9729b6759e7fdab935b/diff:/var/lib/docker/overlay2/e205ec2de285ee51f7b10878c05e984b249e3cb5534cb0ce30003070408930e5/diff:/var/lib/docker/overlay2/259f17afc40e3f6a61e7c250873f88222ead4e4bebca9cef2f13f6ef80665e7a/diff:/var/lib/docker/overlay2/6f701a7b38db49b19ba684c9511c652209880459e3abba19c9a41d7c72127fe7/diff:/var/lib/docker/overlay2/fea80ba5152cdefea65b40d098c952fd826062d2bc31f4c9fb412b2f8f7beea5/diff:/var/lib/docker/overlay2/1bb127973b83ac759fdc59e1470bf185428fd62415b25b8a91f9fa6337fe5e61/diff:/var/lib/docker/overlay2/8d39eec28302ac191fb40fe720debd167979d2af921bb5244cd7f1c256a8dd17/diff:/var/lib/docker/overlay2/721f3d6952a4ab75fb8119f5e626205b489c2c229571b184ab53174e136d88d6/diff",
"MergedDir": "/var/lib/docker/overlay2/2438f1afa6c949e6b54128fbf2e133abb416fb972b32e2db9d37cb4bd85a7c80/merged",
"UpperDir": "/var/lib/docker/overlay2/2438f1afa6c949e6b54128fbf2e133abb416fb972b32e2db9d37cb4bd85a7c80/diff",
"WorkDir": "/var/lib/docker/overlay2/2438f1afa6c949e6b54128fbf2e133abb416fb972b32e2db9d37cb4bd85a7c80/work"
},
"Name": "overlay2"
},
"Mounts": [],
"Config": {
"Hostname": "55cf54b814fe",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"80/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"PHP_INI_DIR=/usr/local/etc/php",
"PHP_EXTRA_BUILD_DEPS=apache2-dev",
"PHP_EXTRA_CONFIGURE_ARGS=--with-apxs2",
"GPG_KEYS=0B96609E270F565C13292B24C13C70B87267B52D 0BD78B5F97500D450838F95DFE857D9A90D90EC1 F38252826ACD957EF380D39F2F7956BC5DA04B5D",
"PHP_VERSION=5.5.30"
],
"Cmd": [
"apache2-foreground"
],
"Image": "stackatoapps/phpinfo",
"Volumes": null,
"WorkingDir": "/var/www/html",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "aa51b9481d1da93e80b497c9093d1999fc5a6631ba785eef9fc7192e4e609329",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/aa51b9481d1d",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"testnet": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"55cf54b814fe"
],
"NetworkID": "c952572231efe71ea025df259b0a4f7b1c2d4a03e5cf15cf1e62e839379078a3",
"EndpointID": "c5b9c56d09c9c44034f4d34ab9fa71b895bfa0f15ccf59eb94e6c59df8cfadfb",
"Gateway": "172.18.0.1",
"IPAddress": "172.18.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:12:00:02",
"DriverOpts": null
}
}
}
}
]
Output of docker version:
Client:
Version: 18.01.0-ce
API version: 1.35
Go version: go1.9.2
Git commit: 03596f5
Built: Wed Jan 10 20:10:02 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.01.0-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.2
Git commit: 03596f5
Built: Wed Jan 10 20:13:41 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 28
Running: 4
Paused: 0
Stopped: 24
Images: 2164
Server Version: 18.01.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
selinux
Kernel Version: 4.14.14-300.fc27.x86_64
Operating System: Fedora 27 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 6.753GiB
Name: yajolap.yajodomain
ID: KUBN:F7JL:URX6:HO55:R3L2:SCUU:IWVY:EZ2O:F53G:WHTO:3G4D:R4YU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Physical, fedora 27.
I'm on x86_84, @GordonTheTurtle 😉
I had similar issues for the past couple of days. I tried setting --net=host and everything worked again. We then set it back to --internal and everything worked as before we had the issue. I have not been able to reproduce the error but if comes again I would check the ARP table (while failing) and try sudo arp -d ContainerIpAddress as described in #36069
Hope it helps. I would like to see your findings.
➤ sudo arp -d 172.22.0.4
No ARP entry for 172.22.0.4
After that, restarting or recreating containers still does not open any ports... 😕
I tried adding this to the docker-compose.yaml:
version: "2.1"
services:
someservice:
#[... other stuff]
networks:
default:
host:
aliases: []
networks:
default:
internal: true
host:
external: trueBut it says:
ERROR: for someservice network-scoped alias is supported only for containers in user defined networks
😭
Hmm. Then we have different issues. Jenkins is building every container right, but there is one container that after every build I need to apply sudo arp -d ContainerIpAddress for it to start working correctly.
As far as I'm aware, this is by design, and --internal networks should not be reachable, even with -p being used (which I think is what moby/libnetwork#1151 was about).
Then is this a bug? 🤔
I was hoping to use --internal to isolate development and staging environments, but it would need then a whitelist system to somehow allow only the required accesses, such as exposing ports or contacting only specific controlled endpoints. Is there support or plan for any of that?
@yajo To isolate multiple stages on a single swarm cluster, you need to place a simple port forwarding service on each stack. This way you can always use the relative naming per stage (develop, integration, etc.) and the stages are fully isolated to each other. Somehow the feature to publish to a named network instead of the default ingress network is missing.
In case anybody stumbles with this issue, I just created https://hub.docker.com/r/tecnativa/whitelist/ for such purpose.
@yajo I recommend to use haproxy for that. haproxy supports zero-copy and tcp-splicing, so the perfect fit for your requirement. Thats how we do it in production at scale to isolate stacks from each other.
If you prefer a dead simple approach to isolate stacks, that does not need the mount of the most sensitive docker deamon on the most exposed container, then I recommend "swarm-router": https://github.com/flavioaiello/swarm-router - Haproxy offers unbeaten performance at very low cpu.
Is there a 1st party way to work around this yet? Or is there no official way to have a network with outgoing traffic blocked without disabling exposing ports?
I want to do this too. Hope there will be a solution/workaround soon...
Same for me... During this time, will try the proxy workaround, thx for the tip
Hit this as well... strange, as having an internal network shouldn't really mean no communication to the host when directly requesting to expose ports...