Docker Containerizer IP-per-container tasks are not auto-allocated a Service Port
ozdanborne opened this issue · comments
I'm launching two near-identical applications, one includes the "ipAddress" field as shown below, the other does not. They are both specifiying VHOSTS.
The one that does not specify the ipAddress field is assigned a servicePort.
The one that does specify the ipAddress field is not assigned a servicePort.
Is there any reason for this difference in behavior? If an app is using IP-per-container and specifies a VHOST, I believe it should still receive a serviceport to map to on the load balancer.
{
"id": "calico-2",
"ipAddress": {
"discovery": {
"ports": [{ "number": 80, "name": "http", "protocol": "tcp" }]
}
},
"container": {
"type": "DOCKER",
"docker": {
"network": "BRIDGE",
"image": "nginx",
"parameters": [{"key": "net", "value": "databases"}]
}
},
"instances": 1,
"cpus": 0.1,
"mem": 65,
"labels":{
"HAPROXY_GROUP":"external",
"HAPROXY_0_VHOST": "my.marathon.app"
}
}
Whoops, setting network to HOST seems to have worked. Sorry for the issue flash.
OK Confirmed that this behavior is definitely happening, regardless of what type of Network is specified as the network type.
Here's some more info to illustrate the problem:
This app does not specify an ipAddress field:
# APP 4
{
"id": "calico-4",
"container": {
"type": "DOCKER",
"docker": {
"network": "HOST",
"image": "nginx",
"parameters": [{"key": "net", "value": "databases"}]
}
},
"instances": 1,
"cpus": 0.1,
"mem": 65,
"labels":{
"HAPROXY_GROUP":"external",
"HAPROXY_0_VHOST": "my.marathon.app"
}
}
And the response includes a service port of 10001 (I've trimmed the response to only show important information):
{'labels': {'HAPROXY_0_VHOST': 'my.marathon.app', 'HAPROXY_GROUP': 'external'}, 'id': '/calico-4', 'container': {'docker': {'image': 'nginx', 'network': 'HOST', 'parameters': [{'value': 'databases', 'key': 'net'}], 'privileged': False}, 'type': 'DOCKER', 'volumes': []}, 'tasks': [{'host': '172.24.197.103', 'ipAddresses': [{'protocol': 'IPv4', 'ipAddress': '192.168.0.64'}], 'appId': '/calico-4', 'ports': [31998]}], 'portDefinitions': [{'labels': {}, 'protocol': 'tcp', 'port': 10001}], 'ports': [10001], 'ipAddress': None}
This app does specify the ipAddress field:
{
"id": "calico-3",
"ipAddress": {
"discovery": {
"ports": [{ "number": 80, "name": "http", "protocol": "tcp" }]
}
},
"container": {
"type": "DOCKER",
"docker": {
"network": "HOST",
"image": "nginx",
"parameters": [{"key": "net", "value": "databases"}]
}
},
"instances": 1,
"cpus": 0.1,
"mem": 65,
"labels":{
"HAPROXY_GROUP":"external",
"HAPROXY_0_VHOST": "my2.marathon.app"
}
}
But now we no longer are allocated a servicePort to map to:
'{"labels": {"HAPROXY_0_VHOST": "my2.marathon.app", "HAPROXY_GROUP": "external"}, "id": "/calico-3", "container": {"docker": {"image": "nginx", "network": "HOST", "parameters": [{"value": "databases", "key": "net"}]}, "type": "DOCKER"}, "tasks": [{"ipAddresses": [{"protocol": "IPv4", "ipAddress": "192.168.0.1"}], "appId": "/calico-3", "ports": []}], "deployments": [], "portDefinitions": [], "cmd": null, "ports": [], "ipAddress": {"labels": {}, "groups": [], "discovery": {"ports": [{"protocol": "tcp", "number": 80, "name": "http"}]}}}'
Marathon-lb operates on the assumption that all services (even those which are just using vhosts) are mapped to a service port. Since marathon is responsible for allocating the service ports, this bug makes marathon-lb incompatible with ip-per-container tasks. Furthermore, marathon-lb is unable to compensate for this bug without the chance of service port collisions happening.
I implemented a marathon-lb fix (mesosphere/marathon-lb#144) which has now been merged into master - this will allow the LB to utilize IP per container.
However, without the fix to this issue in Marathon, the change in the LB is pretty redundant fix - the LB can't do anything without the service port.
It is possible to implement a workaround in the LB to auto-assign service ports when they aren't returned by Marathon, but this isn't an ideal workaround because it's only useful when using vhost since the marathon-lb assigned service port will be arbitrarily chosen by the LB and so you can't rely on accessing a specific LB port for your service.
Trying to understand the ask here. In the case of IP-per-CT it sounds like you want Marathon to allocate a hostPort from resource offers and link that host port to a containerAddress (containerIP:containerPort)? If MarathonLB can already determine which containerIP:containerPort tuples as assigned to a task and use such address for LB (implied by the marathon-lb fix above) then why does Marathon need to provide the hostPort-to-containerAddress linkage?
[EDIT]
If I'm grok'ing the issue here and this really is a requirement, then we could probably configure the Docker container in BRIDGE mode pretty easily since Docker already understands how to link host ports and container ports. Any container type operating in HOST mode probably needs assistance from an executor/containerizer to proxy network connections.
With IP per task/container, we don't want a host port - the IP and port of the container is directly exposed and so there is no requirement for a host port.
The problem is that in the IP per container app data, Marathon does not return a Service Port - so the LB has to assign it's own service port - which isn't particularly sound - Marathon should really be the one responsible for Service port management, otherwise there are potential clashes between ports assigned by Marathon and Marathon-LB
@jdef We're not really talking about host or container ports here. Let me rephrase the issue with some context:
Currently, all Marathon applications that register with marathon-lb are assigned a service port. Despite the fact that servicePorts are a marathon-lb concept, Marathon is the one that allocates and keeps track of available servicePorts.
When launching a ip-per-container application, it is currently impossible to receive a servicePort from Marathon.
There are two scenarios that should trigger the allocation of a service port.
- Ip-per-container applications need a way to request a specific servicePort, just as non-ip-per-container applications Can. Traditional apps can specify this in their portMappings section. But this field cannot be specified if a container is being launched as ip-per-container. So I'd imagine something like a field in the discoveryPorts that would allow the user to request it.
- Ip-per-container applications which specify a VHOST should automatically receive a servicePort. This behavior currently exists for traditional tasks, and is necessary because the haproxy.cfg that marathon-lb generates internally ties all entries (vhost or not) to a base servicePort. Basically, marathon-lb can't make a vhost entry without a service port.
The first point is an enhancement request, but not one that this issue is requesting, as it isn't a useful use-case for ip-per-container anyways.
Instead, this issue was opened to fix the second point, which is a bug.
I am trying to deploy the marathon-lb on DCOS cluster to scale some of the applications (internal). I am using overlay network for all the apps (Calico). Based on https://hub.docker.com/r/mesosphere/marathon-lb/ , confirmed that it's possible. Then found this thread which has useful information. Still, I am not able to complete the deployment. I used the JSON config which was downloaded as part of "dcos package install --options=options.json marathon-lb", and edited that to use 'calico-nw', instead of host. I get the following error in logs
Signature ok
subject=/CN=*
Getting Private key
sysctl: cannot stat /proc/sys/net/ipv4/tcp_tw_reuse: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv4/tcp_max_syn_backlog: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv4/tcp_max_tw_buckets: No such file or directory
sysctl: cannot stat /proc/sys/net/ipv4/tcp_max_orphans: No such file or directory
Is this the correct way? Is there a doc/guide doing this. Thanks.
@PranjalMishra you should hop on the Calico or DC/OS slack or start a new issue so we can troubleshoot why your system is not working. Those sysctl logs aren't something I've ever seen, so maybe your options.json has some incorrect settings.
Additionally, see the above mentioned issue for a small status update on Calico/IP-per-ct + Marathon-LB. Basically, even if you resolve those sysctl issues, Marathon-LB isn't compatible with Marathon's IP-per-ct at the moment.
@djosborne used the writeup https://www.projectcalico.org/calico-dcos-demo-security-speed-and-no-more-port-forwarding/ (Step 5)
It worked for me.
Thank you for you time.
The portMappings field does correctly trigger Marathon to assign a servicePort to Docker Containerizer tasks.
However, I've found that this is still not happening with Unified / CNI apps. I've filed a follow up issue to track the issue at #4548
@djosborne @jdef , We're running Nuage SDN and it looks like when Marthon IP-Per-Task is invoked, containers are not getting servicePort assigned, and as a result no Marathon-LB load balancing possible.
Environment:
marathon-1.3.0-1.0.505
mesos-1.0.1-2.0.93
docker-1.11.2
Marathon-LB (latest - 1.4.2) = haproxy-1.6.3
@djosborne docker containerizer. I am using marathon-1.3.0-1 + Nuage SDN with IP-Per-Task. Marathon LB does not get servicePort passed on, hence no load balancing.
@djosborne I just upgraded my Marathon to 1.3.5. Re-run Marathon-LB. No dice. Here is the JSON file of the services I want to put behind LB
File: zk.kafka.json
{
"id": "/kafka3",
"groups": [{
"id": "/kafka3/zk",
"apps": [{
"id": "zk1",
"cpus": 0.5,
"mem": 128,
"instances": 1,
"maxLaunchDelaySeconds": 10,
"labels": {
"HAPROXY_GROUP": "internal"
},
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "mesoscloud/zookeeper:latest",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 2181,
"hostPort": 22181,
"protocol": "tcp",
"servicePort": 10000
}, {
"containerPort": 2888,
"hostPort": 22888,
"protocol": "tcp",
"servicePort": 10001
}, {
"containerPort": 3888,
"hostPort": 33888,
"protocol": "tcp",
"servicePort": 10002
}
],
"privileged": false,
"parameters": [{
"key": "net",
"value": "developernet"
}, {
"key": "hostname",
"value": "zk1.zk.kafka3.marathon.mesos"
}, {
"key": "dns",
"value": "192.168.120.161"
}, {
"key": "dns",
"value": "192.168.120.162"
}, {
"key": "dns",
"value": "192.168.120.163"
}, {
"key": "dns-search",
"value": "localnet.net"
}, {
"key": "dns-search",
"value": "developer.local"
}],
"forcePullImage": false
}
},
"constraints": [
["hostname", "UNIQUE"]
],
"portDefinitions": [],
"ipAddress": {
"networkName": "developernet"
},
"env": {
"MYID": "1",
"ZOOKEEPER_INIT_LIMIT": "30",
"ZOOKEEPER_CLIENT_PORT": "2181",
"ZOOKEEPER_SYNC_LIMIT": "30",
"ZOOKEEPER_LOG4J_ROOT_LOGLEVEL": "INFO",
"SERVERS": "zk1.zk.kafka3.marathon.mesos,zk2.zk.kafka3.marathon.mesos,zk3.zk.kafka3.marathon.mesos"
}
}, {
"id": "zk2",
"dependencies": ["/kafka3/zk/zk1"],
"cpus": 0.5,
"mem": 128,
"instances": 1,
"maxLaunchDelaySeconds": 10,
"labels": {
"HAPROXY_GROUP": "internal"
},
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "mesoscloud/zookeeper:latest",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 2181,
"hostPort": 22181,
"protocol": "tcp",
"servicePort": 10000
}, {
"containerPort": 2888,
"hostPort": 22888,
"protocol": "tcp",
"servicePort": 10001
}, {
"containerPort": 3888,
"hostPort": 33888,
"protocol": "tcp",
"servicePort": 10002
}
],
"privileged": false,
"parameters": [{
"key": "net",
"value": "developernet"
}, {
"key": "hostname",
"value": "zk2.zk.kafka3.marathon.mesos"
}, {
"key": "dns",
"value": "192.168.120.161"
}, {
"key": "dns",
"value": "192.168.120.162"
}, {
"key": "dns",
"value": "192.168.120.163"
}, {
"key": "dns-search",
"value": "localnet.net"
}, {
"key": "dns-search",
"value": "developer.local"
}],
"forcePullImage": false
}
},
"constraints": [
["hostname", "UNIQUE"]
],
"portDefinitions": [],
"ipAddress": {
"networkName": "developernet"
},
"env": {
"MYID": "2",
"ZOOKEEPER_INIT_LIMIT": "30",
"ZOOKEEPER_CLIENT_PORT": "2181",
"ZOOKEEPER_SYNC_LIMIT": "30",
"ZOOKEEPER_LOG4J_ROOT_LOGLEVEL": "INFO",
"SERVERS": "zk1.zk.kafka3.marathon.mesos,zk2.zk.kafka3.marathon.mesos,zk3.zk.kafka3.marathon.mesos"
}
}, {
"id": "zk3",
"dependencies": ["/kafka3/zk/zk2"],
"cpus": 0.5,
"mem": 128,
"instances": 1,
"maxLaunchDelaySeconds": 10,
"labels": {
"HAPROXY_GROUP": "internal"
},
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "mesoscloud/zookeeper:latest",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 2181,
"hostPort": 22181,
"protocol": "tcp",
"servicePort": 10000
}, {
"containerPort": 2888,
"hostPort": 22888,
"protocol": "tcp",
"servicePort": 10001
}, {
"containerPort": 3888,
"hostPort": 33888,
"protocol": "tcp",
"servicePort": 10002
}
],
"privileged": false,
"parameters": [{
"key": "net",
"value": "developernet"
}, {
"key": "hostname",
"value": "zk3"
}, {
"key": "dns",
"value": "192.168.120.161"
}, {
"key": "dns",
"value": "192.168.120.162"
}, {
"key": "dns",
"value": "192.168.120.163"
}, {
"key": "dns-search",
"value": "localnet.net"
}, {
"key": "dns-search",
"value": "developer.local"
}],
"forcePullImage": false
}
},
"constraints": [
["hostname", "UNIQUE"]
],
"portDefinitions": [],
"ipAddress": {
"networkName": "developernet"
},
"env": {
"MYID": "3",
"ZOOKEEPER_INIT_LIMIT": "30",
"ZOOKEEPER_CLIENT_PORT": "2181",
"ZOOKEEPER_SYNC_LIMIT": "30",
"ZOOKEEPER_LOG4J_ROOT_LOGLEVEL": "INFO",
"SERVERS": "zk1.zk.kafka3.marathon.mesos,zk2.zk.kafka3.marathon.mesos,zk3.zk.kafka3.marathon.mesos"
}
}
]
}, {
"id": "/kafka3/kafkabrokers",
"dependencies": ["/kafka3/zk"],
"apps": [{
"id": "broker1",
"cpus": 0.5,
"mem": 512,
"instances": 1,
"maxLaunchDelaySeconds": 10,
"labels": {
"HAPROXY_GROUP": "internal"
},
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "akamalov/kafka2:0.10.0",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 9092,
"hostPort": 9092,
"protocol": "tcp",
"servicePort": 11000
}
],
"privileged": false,
"parameters": [{
"key": "net",
"value": "developernet"
}, {
"key": "hostname",
"value": "broker1.kafkabrokers.kafka3.marathon.mesos"
}, {
"key": "dns",
"value": "192.168.120.161"
}, {
"key": "dns",
"value": "192.168.120.162"
}, {
"key": "dns",
"value": "192.168.120.163"
}, {
"key": "dns-search",
"value": "localnet.net"
}, {
"key": "dns-search",
"value": "developer.local"
}],
"forcePullImage": false
}
},
"constraints": [
["hostname", "UNIQUE"]
],
"portDefinitions": [],
"ipAddress": {
"networkName": "developernet"
},
"env": {
"BROKER_ID": "1",
"ZOOKEEPER_SERVERS": "zk1.zk.kafka3.marathon.mesos:2181,zk2.zk.kafka3.marathon.mesos:2181,zk3.zk.kafka3.marathon.mesos:2181",
"ADVERTISED_LISTENERS": "PLAINTEXT://broker1.kafkabrokers.kafka3.marathon.mesos:9092",
"ADVERTISED_HOST_NAME": "broker1.kafkabrokers.kafka3.marathon.mesos",
"ADVERTISED_PORT": "9092",
"LOG4J_ROOT_LOGLEVEL": "INFO"
}
}, {
"id": "broker2",
"cpus": 0.5,
"mem": 512,
"instances": 1,
"maxLaunchDelaySeconds": 10,
"labels": {
"HAPROXY_GROUP": "internal"
},
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "akamalov/kafka2:0.10.0",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 9092,
"hostPort": 9092,
"protocol": "tcp",
"servicePort": 11000
}
],
"privileged": false,
"parameters": [{
"key": "net",
"value": "developernet"
}, {
"key": "hostname",
"value": "broker2.kafkabrokers.kafka3.marathon.mesos"
}, {
"key": "dns",
"value": "192.168.120.161"
}, {
"key": "dns",
"value": "192.168.120.162"
}, {
"key": "dns",
"value": "192.168.120.163"
}, {
"key": "dns-search",
"value": "localnet.net"
}, {
"key": "dns-search",
"value": "developer.local"
}],
"forcePullImage": false
}
},
"constraints": [
["hostname", "UNIQUE"]
],
"portDefinitions": [],
"ipAddress": {
"networkName": "developernet"
},
"env": {
"BROKER_ID": "2",
"ZOOKEEPER_SERVERS": "zk1.zk.kafka3.marathon.mesos:2181,zk2.zk.kafka3.marathon.mesos:2181,zk3.zk.kafka3.marathon.mesos:2181",
"ADVERTISED_LISTENERS": "PLAINTEXT://broker2.kafkabrokers.kafka3.marathon.mesos:9092",
"ADVERTISED_HOST_NAME": "broker2.kafkabrokers.kafka3.marathon.mesos",
"ADVERTISED_PORT": "9092",
"LOG4J_ROOT_LOGLEVEL": "INFO"
}
}, {
"id": "broker3",
"cpus": 0.5,
"mem": 512,
"instances": 1,
"maxLaunchDelaySeconds": 10,
"labels": {
"HAPROXY_GROUP": "internal"
},
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "akamalov/kafka2:0.10.0",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 9092,
"hostPort": 9092,
"protocol": "tcp",
"servicePort": 11000
}
],
"privileged": false,
"parameters": [{
"key": "net",
"value": "developernet"
}, {
"key": "hostname",
"value": "broker3.kafkabrokers.kafka3.marathon.mesos"
}, {
"key": "dns",
"value": "192.168.120.161"
}, {
"key": "dns",
"value": "192.168.120.162"
}, {
"key": "dns",
"value": "192.168.120.163"
}, {
"key": "dns-search",
"value": "localnet.net"
}, {
"key": "dns-search",
"value": "developer.local"
}],
"forcePullImage": false
}
},
"constraints": [
["hostname", "UNIQUE"]
],
"portDefinitions": [],
"ipAddress": {
"networkName": "developernet"
},
"env": {
"BROKER_ID": "3",
"ZOOKEEPER_SERVERS": "zk1.zk.kafka3.marathon.mesos:2181,zk2.zk.kafka3.marathon.mesos:2181,zk3.zk.kafka3.marathon.mesos:2181",
"ADVERTISED_LISTENERS": "PLAINTEXT://broker3.kafkabrokers.kafka3.marathon.mesos:9092",
"ADVERTISED_HOST_NAME": "broker3.kafkabrokers.kafka3.marathon.mesos",
"ADVERTISED_PORT": "9092",
"LOG4J_ROOT_LOGLEVEL": "INFO"
}
}
]
}]
}
Here is the JSON file of the Marathon LB I am deploying:
File: marathon-lb.json
{
"id": "marathon-lb-internal",
"instances": 5,
"cpus": 0.5,
"mem": 64.0,
"maintainer": "support@mesosphere.io",
"container": {
"type": "DOCKER",
"docker": {
"image": "mesosphere/marathon-lb:latest",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 9090,
"hostPort": 9090,
"protocol": "tcp"
}, {
"containerPort": 9091,
"hostPort": 9191,
"protocol": "tcp"
}],
"privileged": true,
"parameters": [{
"key": "net",
"value": "developernet"
}, {
"key": "add-host",
"value": "mslave1:192.168.120.161"
}, {
"key": "add-host",
"value": "mslave2:192.168.120.162"
}, {
"key": "add-host",
"value": "mslave3:192.168.120.163"
}, {
"key": "add-host",
"value": "mslave4:192.168.120.164"
}, {
"key": "add-host",
"value": "mslave5:192.168.120.165"
}]
}
},
"constraints": [
["hostname", "UNIQUE"]
],
"healthChecks": [{
"path": "/_haproxy_health_check",
"portIndex": 0,
"protocol": "HTTP",
"gracePeriodSeconds": 300,
"intervalSeconds": 60,
"timeoutSeconds": 20,
"maxConsecutiveFailures": 3,
"ignoreHttp1xx": false
}],
"args": [
"sse",
"-m", "http://master.mesos:8080",
"--health-check",
"--dont-bind-http-https",
"--group", "internal"
],
"env": {
"HAPROXY_SSL_CERT": "/etc/ssl/localnet/localnet.com.pem"
},
"portDefinitions": [],
"ipAddress": {
"networkName": "developernet"
}
}
Marathon-LB is not getting servicePorts passed along:
@djosborne would you be able to take a look at my reply above, please? If yes and confirmed, can we re-open the ticket, please?
@akamalov Mine is working on Marathon 1.3.2. But your issue is different than the original issue here.
You've explicitly specified a ServicePort in your application definitions. The original issue is for cases where servicePort is not specified, and it is expected one be allocated to the application by Marathon.
For your case, I suggest you try the following:
- View the app's "configuration" tab in Marathon, and see if the Container section includes PortMappings with a ServicePort
- I question why you run Marathon-LB in bridged mode. It is usually meant to be run with host networking (unless you know what you are doing).
- If you do see ServicePorts in Marathon, your issue may be with Marathon-LB itself, so you may want to open an issue there
Thanks for replying @djosborne
- I question why you run Marathon-LB in bridged mode. It is usually meant to be run with host networking (unless you know what you are doing).
One of the reasons I went with BRIDGE mode on Marathon-LB is to assign an IP address of Nuage to Marathon-LB containers , which completely divorces it from the host. Containers to be load-balanced are also getting their IP off SDN framework.
I guess I have to open up a ticket with Marathon-LB.
Thanks again.