When DCOS can't authenticate, we get connection errors, instead of a better error message
travisbhartwell opened this issue · comments
I have a DC/OS 1.8.3 cluster that I have installed the RMF on. I have seen this error before, so I duplicated it like this:
- Configured the DCOS url for my DC/OS cluster
- Authenticated against the DC/OS cluster by:
dcos auth login
- Installed RMF by:
dcos package repo add RMF https://github.com/basho-labs/riak-mesos-dcos-repo/archive/develop.zip
- Backed up my
~/.dcos/dcos.toml
file and then removed thedcos_acs_token
line - Attempted to create a cluster:
$ dcos riak-ts --debug cluster create
Insecure SSL Mode: False
Verbose Mode: True
Debug Mode: True
JSON Mode: False
Couldn't find config file
Attempting to create DCOSClient
cannot concatenate 'str' and 'NoneType' objects
Defaulting to configuration based URLs
INFO:dcos.http:Sending HTTP ['get'] to ['http://marathon.mesos:8080/ping']: {'Accept': 'application/json'}
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): marathon.mesos
ERROR:dcos.http:HTTP Connection Error
Traceback (most recent call last):
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/dcos/http.py", line 96, in _request
**kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='marathon.mesos', port=8080): Max retries exceeded with url: /ping (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x109dc8a50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
DEBUG:dcos.util:duration: dcos.http._request: 0.02s
URL [http://marathon.mesos:8080/ping] is unreachable: HTTPConnectionPool(host='marathon.mesos', port=8080): Max retries exceeded with url: /ping (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x109dc8a50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
Traceback (most recent call last):
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/bin/dcos-riak-ts", line 11, in <module>
load_entry_point('dcos-riak-ts==0.0.1', 'console_scripts', 'dcos-riak-ts')()
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args[1:], **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/riak_mesos/commands/cmd_cluster.py", line 178, in create
data='')
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/riak_mesos/cli.py", line 396, in api_request
exit_on_failure, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/riak_mesos/cli.py", line 409, in framework_request
raise e
riak_mesos.cli.CliError: 'Unable to to find marathon URL'
I'm not sure how to properly handle this; I assume by the output that accessing the DC/OS cluster failed because of authentication errors and so then it tried to fall back to using configuration values for Marathon and Mesos Master and those were not set. Perhaps we need a better error message when authentication fails to clue the user into what is going on. It really was only intuition that this was the error when I ran into it previously.
The error is actually slightly different than I showed above.
I fixed the bug on riak_mesos/cli.py#L81 by wrapping ssl_verify
in str()
.
If I don't have dcos_acs_token
in my ~/.dcos/dcos.toml
at all, it just automatically prompts me to login, like when you do dcos auth login
. If the token is wrong, (say, by removing one of the characters of the token in the config file), you get this output, which is slightly different:
to to find framework URL'
travisbhartwell@bashombp:~/.dcos$ dcos riak-ts --debug cluster create
Insecure SSL Mode: False
Verbose Mode: True
Debug Mode: True
JSON Mode: False
Couldn't find config file
Attempting to create DCOSClient
DCOS core.ssl_verify value = None
INFO:dcos.http:Sending HTTP ['get'] to [u'http://158.85.251.23/service/riak-ts/healthcheck']: {'Accept': 'application/json'}
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 158.85.251.23
DEBUG:requests.packages.urllib3.connectionpool:"GET /service/riak-ts/healthcheck HTTP/1.1" 401 None
INFO:dcos.http:Received HTTP response [401]: {'Transfer-Encoding': 'chunked', 'Server': 'openresty/1.9.15.1', 'Connection': 'keep-alive', 'Date': 'Tue, 13 Sep 2016 22:25:02 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'WWW-Authenticate': 'oauthjwt'}
DEBUG:dcos.util:duration: dcos.http._request: 0.23s
INFO:dcos.http:Sending HTTP ['get'] to [u'http://158.85.251.23/service/riak-ts/healthcheck']: {'Accept': 'application/json'}
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 158.85.251.23
DEBUG:requests.packages.urllib3.connectionpool:"GET /service/riak-ts/healthcheck HTTP/1.1" 401 None
INFO:dcos.http:Received HTTP response [401]: {'Transfer-Encoding': 'chunked', 'Server': 'openresty/1.9.15.1', 'Connection': 'keep-alive', 'Date': 'Tue, 13 Sep 2016 22:25:03 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'WWW-Authenticate': 'oauthjwt'}
DEBUG:dcos.util:duration: dcos.http._request: 0.21s
Your core.dcos_acs_token is invalid. Please run: `dcos auth login`
Traceback (most recent call last):
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/bin/dcos-riak-ts", line 11, in <module>
load_entry_point('dcos-riak-ts==0.0.1', 'console_scripts', 'dcos-riak-ts')()
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 696, in main
rv = self.invoke(ctx)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args[1:], **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/click/core.py", line 534, in invoke
return callback(*args, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/riak_mesos/commands/cmd_cluster.py", line 178, in create
data='')
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/riak_mesos/cli.py", line 396, in api_request
exit_on_failure, **kwargs)
File "/Users/travisbhartwell/.dcos/subcommands/riak-ts/env/lib/python2.7/site-packages/riak_mesos/cli.py", line 409, in framework_request
raise e
riak_mesos.cli.CliError: 'Unable to to find framework URL'
If the user has not passed --debug
to that command-line, you will miss the actual error:
Your core.dcos_acs_token is invalid. Please run: `dcos auth login`
If --debug
is omitted, it only shows the traceback about being unable to find the framework URL.