Switch to using hyphens as a separator in hostnames
kung-foo opened this issue · comments
The hostname format used by fig creates names that are not strictly valid.
Current pseudo-code:
name = '_'.join([project, container, instance_num])This generates names like cluster_hadoop_1. Underscores are not valid (though in practice most components are tolerant of this).
Valid names should match [a-zA-Z0-9\-]+.
I came across this error when trying to test out some hadoop/hdfs containers and hadoop bailed with an exception saying that hdfs://flume_hadoop_1/ was not valid URI (even though it was in /etc/hosts and ping flume_hadoop_1 worked just fine).
See: http://en.wikipedia.org/wiki/Hostname#Restrictions_on_valid_host_names
Changing this to dashes is easy, but it would break existing configurations that depend on hard coded container names.
also see: moby/moby#5418
This is causing pain for me as well, in pretty much the exact same scenario (the hostname being parsed by Java's standard library URI parser). Is this a fig issue, or should it be fixed in docker?
Frustratingly, underscores are invalid in hostnames, and dashes are invalid in shell variable names. So if you can't name a Docker link without either breaking one of those or smushing everything together (e.g. hadoop1). Not ideal.
However, looks like moby/moby#6270 might fix this by sanitising environment variable names, in which case Fig can switch to dashes and everything will hopefully work:
web:
links:
- hadoop$ fig run web cat /etc/hosts
...
172.17.0.42 hadoop-1
...
$ fig run web env
...
HADOOP_1_PORT=tcp://172.17.0.42:5432
HADOOP_1_PORT_5432_TCP=tcp://172.17.0.42:5432
HADOOP_1_PORT_5432_TCP_ADDR=172.17.0.42
...#349 is also relevant.
Technically speaking, I don't believe that hyphens in shell variable names are strictly forbidden. You just can't access them in the normal way:
$ env foo-bar=baz bash
$ printenv foo-bar
baz
👍 switching to hyphens
What about writing both hyphenated and underscored to the hosts file for now?
@zeeraw that would be out of fig's scope - a job for docker itself.
We can do this now that moby/moby#5418 has been fixed, right?
Ugh java's URI is so annoying. The single-value constructor does not throw an exception for underscores in the hostname.
// Error: java.net.URISyntaxException: Illegal character in hostname at index 10: http://foo_bar:9200/path?query=1
URI u1 = new URI("http", null, "foo_bar", 9200, "/path", "query=1", null);
// works fine:
URI u2 = new URI("http://foo_bar:9200/path?query=1");
Will this also be used for the container names? If so, I think moby/moby#8961 should also be taken into account.
We could add aliases for dashes when we implement #2312, keep both underscores and dashes for a release or two, then drop support for underscores in hostnames.
Same here, cannot run spark cluster with docker-compose. Showstopper
for docker-compose, falling back to "manual docker".
@susu Why is this a showstopper? The problem of underscores in hostnames is easily worked around, either with link aliases or by changing service names.
Nope, because spark somehow extracts the underscored hostname of docker
container (projectname_container_1.default_network), and try to use it
(which is an invalid hostname in URI). Anyway, falling back to version 1
"solved" the issue.
On Jul 13, 2016 6:52 PM, "Aanand Prasad" notifications@github.com wrote:
@susu https://github.com/susu Why is this a showstopper? The problem of
underscores in hostnames is easily worked around, either with link aliases
or by changing service names.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#229 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AA5B7o9ixPlutkGcoBwiBnp2N5EcqBszks5qVRfQgaJpZM4B9pLe
.
spark somehow extracts the underscored hostname of docker container (projectname_container_1.default_network), and try to use it
However it's doing that, that's the wrong thing to do. Is this an image you're using from the Hub, or did you build it yourself?
Switching back to version 1 is not a future-proof solution.
@susu The workaround I used for this was to create a network so I could control the name:
docker network create spark
Use that as an external network in my (version 2) docker-compose.yml file:
networks:
spark:
external: true
Connect each service to that network:
services:
spark-master:
image: spark
command: org.apache.spark.deploy.master.Master --host spark-master
networks:
- spark
And finally override the container name and hostname in the spark worker service:
spark-worker:
image: spark
container_name: "spark-worker"
hostname: "spark-worker"
command: org.apache.spark.deploy.worker.Worker --host spark-worker spark://spark-master:7077
networks:
- spark
P.S. I determined that the spark worker is using reverse DNS lookup on its IP address to get the DNS name to register with the master.
Thanks @gordontyler ! I didn't know I can override container and hostname! Already migrated to version 2 :)
@aanand I know version 1 is not future-proof, however at the moment I'm only creating a PoC with spark, so clean, production-grade solution is not a must-have :) But anyway, using @gordontyler 's solution, I can go forward with version 2! Thank you guys!
FYI, that issue has also been raised in Docker Forum as Underscore in domain names.
In particular, I explain there where & why Spark (from version 1.6.x) doesn't allow underscores in hostnames.
// We identify hosts on which the block is cached with this prefix. Because this prefix contains
// underscores, which are not legal characters in hostnames, there should be no potential for
// confusion. See RFC 952 and RFC 1123 for information about the format of hostnames.
@gordontyler Great workaround!
You can even have a simpler code by just specifying your external network as the default one:
networks:
default:
external:
name: spark
@gordontyler solution is a great workaround. But in my case, I need to use docker-compose scale function, and that is a problem again because of the fixed container name.
I was just experimenting with this today again and I was able to get a working Spark cluster with a scalable worker service (i.e. not using container and host name overrides) and I was able to successfully deploy an app to a Spark cluster containing 2 worker instances created by docker-compose scale spark-worker=2.
The only difference that I can see is that I'm using the bridge driver for the network that the cluster containers are attached to instead of overlay.
Duh... Another big difference in my latest tests is that I was using Spark 2.0.0. It may be more lenient with regards to underscores in DNS name or somehow avoids the problem.
A new release of the Python library requests has been released today which enforces hostname validation which leads it to refuse to works with hyphen anymore. I think compose use it internally, is there a chance that compose might break when updating requests? I don't think compose is making direct HTTP calls to the containers it manage but be safe than sorry.
Container discovery with consul isn't working because hostnames with underscores are invalid for DNS discovery.
Would be nice to have (at least) an option for switching from underscore to dashes for the container names.
@aanand As you seem to be an expert on this topic, could you link/explain why services created using docker stack deploy are named "project_service" instead of "project-service" ?
Isn't it possible to define/change this?
As a workaround, I'm considering adding host alias to service definition, but haven't tested it yet
Services within the stack should already be aliased to their service name (the name that appears in the docker-compose.yml), so you shouldn't need any other aliases. You can ignore the names with underscores.
Services within the stack should already be aliased to their service name (the name that appears in the docker-compose.yml), so you shouldn't need any other aliases. You can ignore the names with underscores.
This is not how things are supposed to work. The names are supposed to be the DNS names. They aren't some field that should be mangled.
That is how things are supposed to work for an "isolated environment". I should be able to launch a stack from a Compose file that has service names like web and db, and not worry about them conflicting with services from another project. Each service in that stack should be able to reference the other services with this short name (web or db).
When we have a server side stack, or namespace concept we can adjust how this works, but until then it is necessary to mangle.
@dnephin We should have gated these features on namespaces. What is the impetus to do namespaces now that we have support backwards compatible name mangling? We end up having to support two things instead of one.
That is how things are supposed to work for an "isolated environment". I should be able to launch a stack from a Compose file that has service names like web and db, and not worry about them conflicting with services from another project. Each service in that stack should be able to reference the other services with this short name (web or db).
@dnephin I still have to test a few things before properly answering...but how is "changing from _ to -" in project name (ie: mystack_myservice to mystack-myservice, hence achieving compatibility) against your statement?
I would like to confirm containers can talk to each other using its "service name" (eg: myservice), but at this point hostnames are set to mystack_myservice.
@dnephin Still wondering what's the issue not to move away from underscores "_".
What is the status of this feature? I still see my containers created with underscores, which causes unexpected problems, as described in detail here
Same here, and still no fix possible with scale feature.
This is causing massive headaches for me as well, any updates on this? Temporary workaround is to use the internal IP addresses instead of hostname but it's becoming frustrating for our development team.
+1
One of the bigger problems with the underscore in hostnames, which is generated by docker stack is:
It doesn't work with the default configuration of Spring Boot.
Spring Boot uses Apache Tomcat as default, which itself cannot serve HTTP-Requests, when it is addressed by its fully qualified internal service name (<stack-name>_<service-name>). Only a switch to jetty, which seems to be more relaxed, fixed my problems.
A little suggestion: Why not use dots instead of underscore and so use `subdomains' to map services to stacks?
Django 2.1 also rejects http header like "Host: myapp_django" if the service name is myapp_django. It's hard to avoid.
Someone suggested to create different network for each stack, so you can access a service without stack name, like django. But I have services in different stacks communicating with each other, and I can't put them all in one stack.
I'm seeing the same problem. While Jetty is OK with inbound requests with a host name with underscores, it uses the Java URI library for outbound proxy requests and breaks if the hostname has an underscore.
Right now, I'm able to work around the problem with explicit aliases in say, external links, but it would definitely be a plus to be able to switch the separator to hyphens with an option in the environment or in the compose file.
Just came across this problem with spring cloud config and docker stack deploy on docker 18.09, the error is the following:
2018-11-25 12:37:21.566 INFO 1 --- [nio-8090-exec-2] o.apache.coyote.http11.Http11Processor : The host [tskur-ci_tskur-svc-config:8090] is not valid
Note: further occurrences of request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: The character [_] is never valid in a domain name.
at org.apache.tomcat.util.http.parser.HttpParser$DomainParseState.next(HttpParser.java:926) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]
Is anyone able to point me to a workaround?
Just got bit by this today as well. Kibana and logstash absolutely will not allow connecting to host names with an underscore in them which means you are forced to use the host IP or workarounds such as creating a service name with a dash in it to force docker to deploy a valid service name.
An option to make docker use a dash character instead of underscores in service names would be very useful to avoid these types of problems.
@blackknight36 I would recommend using network aliases, which let you assign valid hostnames while still allowing you to use whatever service name you want.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Not stale at all, I'd say this is still unresolved? Consul DNS also doesn't like underscores in hostnames and complains... Is it possible to make this more flexible? Like letting the user change the seperator?
This issue has been automatically marked as not stale anymore due to the recent activity.
just ran into a problem with logstash related to this.
can somebody at docker HQ at least make a decision?
shouldn't docker-compose comply with the relevant RFCs?
I am having this problem as well, related to Hazelcast. I can work around it using fixed container names without underscores but it precludes the use of the docker-compose scaling mechanism since the system assigned name is mystack_myservice_n.
389 directory server's admin console, 389-console, is not able to handle the underscores in container DNS names, but adding a network alias hostname with no underscores resolved that. I wasn't even able to complete the directory server configuration until adding the hostname aliases with no underscores.
I had to also use "StrictHostCheck = false" in the initial-setup.inf file and "/usr/sbin/setup-ds-admin.pl --silent --file=initial-setup.inf General.StrictHostCheck=false" when installing the directory server binaries, admin server binaries and the initial directory configuration. This was because Docker DNS is a can of worms regarding reverse DNS entries.
Is this being done any time soon? This makes docker-compose unusable with many languages, which is really unfortunate.
How is this still an issue after 6 years? This is a basic DNS violation.
RFC 1034 and 1035 ->
<subdomain> ::= <label> | <subdomain> "." <label>
<label> ::= <letter> [ [ <ldh-str> ] <let-dig> ]
<ldh-str> ::= <let-dig-hyp> | <let-dig-hyp> <ldh-str>
<let-dig-hyp> ::= <let-dig> | "-"
<let-dig> ::= <letter> | <digit>
<letter> ::= any one of the 52 alphabetic characters A through Z in
upper case and a through z in lower case
<digit> ::= any one of the ten digits 0 through 9
This makes compose absolutely useless for more "strict" or rather correct tools.
There is also no reasonable way to overwrite this. Als if i spawn more than 1 container of a service, the alias and container names no longer work.
Reading through this thread, there don't appear to be any real arguments against the proposed change. (Perhaps I missed them?)
Are there any philosophical or practical reasons that a PR to address this would not be merged, or is this simply a matter of nobody putting in the work yet?
+1
This is a problem with Tomcat (and, by extension, with many Spring Boot applications) too: https://stackoverflow.com/q/53504857/1374678
Just got bit by this myself. Connecting two containers from different services should have been a breeze, instead it has been a day-long nightmare. Please fix this. It has been open for almost seven years.
I discovered that we can put dots in service aliases! That means that I can define names how I want them: in FQDN format and no underscores. I have scripting that standardizes our docker-compose.yml files and automatically defines services under the RFC6762 .private domain as <service-name>.<stack-name>.<network-name>.private, e.g. mariadb.db.swarm.private. This also makes it easier to filter swarm service name queries on upstream nameservers when the service isn't up yet, and many recursive nameservers won't recurse for that domain. Docker can keep its wonky underscores and unqualified names because it doesn't matter to me anymore. 😃
Just came across this problem with spring cloud config and docker stack deploy on docker 18.09, the error is the following:
2018-11-25 12:37:21.566 INFO 1 --- [nio-8090-exec-2] o.apache.coyote.http11.Http11Processor : The host [tskur-ci_tskur-svc-config:8090] is not valid Note: further occurrences of request parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: The character [_] is never valid in a domain name. at org.apache.tomcat.util.http.parser.HttpParser$DomainParseState.next(HttpParser.java:926) ~[tomcat-embed-core-9.0.12.jar!/:9.0.12]Is anyone able to point me to a workaround?
Rename your containers to change the underscores to hyphens.
IMHO - the documentation should indicate that container names that include characters that are not valid for hostnames might break, dependent on the underlying server URL parser (e.g. Spring, tomcat)
eventually closing this issue, by Compose V2 hostname will be set using hyphens, unless --compatibility flag is set
Whoa this is a big change to spring on users after it wasn't enabled in the last RC (it's the default even for existing projects). Is there a way to make this setting global e.g. via a config file, rather than updating all my scripts?
unknown flag: --compatibility
Use this env variable instead: COMPOSE_COMPATIBILITY=true
→ docker compose --help
Options:
--compatibility Run compose in backward compatibility mode@ndeloof something is messed with the command line.
This command works:
docker compose build --pull --force-rm
This doesn't:
docker compose build --pull --force-rm --compatibility
unknown flag: --compatibility
This doesn't either:
docker compose --pull --force-rm --compatibilty build
unknown flag: --pull
Version on MacOS:
~ % docker --version
Docker version 20.10.8, build 3967b7d
Edit:
Command is ment to be like this I guess:
docker compose --compatibility build --pull --force-rm
It takes some getting used to :-S
"compatibility" applies to the top-level "compose" verb:
docker compose --compatibilty (other flags and commands) ...
MinIO, which is a S3-compatible storage solution, doesn't support underscores in given hostnames anymore. Which means all my stacks that are deployed in my docker swarm with docker stack deploy -c docker-compose.yml stackname are not usable anymore.
As far as I know compose verison 3 uses underscores like this: stackname_containername by default. Is there any way to change the underscore myself in some setting? Or do i have to use a newer compose version?
@Issykul This is how I solved the problem in Docker compose file:
hostname: user-001-login.login
networks:
login:
aliases:
- user-001-login.login
- user-001-login
Docker's default DNS service naming is a little silly because it does not take advantage of standard DNS hierarchy conventions, which could still be useful.
I like to ignore Docker's automatic service names and define my own instead, just as @aki-k suggested. The only difference is that I like to make mine fully qualified. Doing it this way allows me to use filter the top-level domain and avoid hammering my upstream nameservers if a service isn't up yet. I could filter all unqualified names, but I don't like that. I like to name things [service].[stack].[network].private, for example:
networks:
swarm:
aliases:
- prometheus.monitor.swarm.private