Error Kubernetes Connection - Requesting "https://10.254.0.1:443" failed: "HTTP 401: Unauthorized"
Tedezed opened this issue · comments
My instalation of ElasticKube
I'm trying to add elastickube to my cluster Kubernetes in HA. Structure of the nodes image.
curl -s https://elastickube.com | bash -s -- -u http://10.0.0.39:8080
(10.0.0.39:8080 is a virtual IP for Kubernetes Masters in HA)
_____ _ _ _ _ __ _
| ____| | __ _ ___| |_(_) ___| |/ / _| |__ ___
| _| | |/ _` / __| __| |/ __| ' / | | | '_ \ / _ \
| |___| | (_| \__ \ |_| | (__| . \ |_| | |_) | __/
|_____|_|\__,_|___/\__|_|\___|_|\_\__,_|_.__/ \___| by ElasticBox
Checking kubectl is available [ ✓ ]
Verifying Kubernetes cluster [ ✓ ]
Setting up elastickube-server svc [ ✓ ]
Setting up elastickube-mongo svc [ ✓ ]
Setting up elastickube-mongo [ ✓ ]
Setting up elastickube-server [ ✓ ]
WARNING: LoadBalancer Ingress not detected, please ensure the address is accessible from outside the cluster. Check http://kubernetes.io/docs/user-guide/ingress/ for more information.
Waiting for LB to be ready [ ✓ ]
ElasticKube has been deployed!
Please complete the installation here: http://10.254.51.59
I install Heapster for solve error Heapster Connection Not Found 404. Install /elastickube/tree/master/build/kubegrunt/heapster
for file in $(ls | grep "\.yaml") ; do \
kubectl create -f $file
done
Error:
State
NAMESPACE NAME READY STATUS RESTARTS AGE NODE
kube-system elastickube-mongo-5d3nn 1/1 Running 0 6m artio
kube-system elastickube-server-0pz5j 4/4 Running 0 6m artio
kube-system heapster-v1.0.2-glbes 2/2 Running 0 7m artio
kube-system kube-dns-v9-k2ir3 4/4 Running 0 16m artio
kube-system kubernetes-dashboard-v1.0.1-625z4 1/1 Running 0 16m artio
kube-system monitoring-influxdb-grafana-v3-ao44e 2/2 Running 0 7m esus
NAME LABELS STATUS AGE
artio kubernetes.io/hostname=artio,role=loadbalancer Ready 22d
esus kubernetes.io/hostname=esus Ready 22d
Kubernetes master is running at http://10.0.0.39:8080
Heapster is running at http://10.0.0.39:8080/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at http://10.0.0.39:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at http://10.0.0.39:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
Grafana is running at http://10.0.0.39:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at http://10.0.0.39:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb
Hi,
Can you add the logs of elastickube-server (the diagnostics and the api containers)?
Can you give us also the environment variables available in the elastickube-server?
It seems that there is not access from the elastickube-server pod to the kubernetes cluster. Usually, there is a secret kube-api that gives read/write access to the kube-apiserver. It is needed to access the kubernetes api to function so it seems that secret is missing or the API cannot be reached. The logs should provide more information.
Hi, thx for your answer, the information:
Logs
kubectl logs elastickube-server-0pz5j elastickube-diagnostics --namespace=kube-system
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-90d3804f0231704c15ccc5861245e8ce.woff (0.0.0.0) 2.58ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-cc85ce37b4256966e6f3a3559239c5c0.ttf (0.0.0.0) 1.96ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-90d3804f0231704c15ccc5861245e8ce.woff (0.0.0.0) 1.44ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-cc85ce37b4256966e6f3a3559239c5c0.ttf (0.0.0.0) 9.05ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-90d3804f0231704c15ccc5861245e8ce.woff (0.0.0.0) 3.79ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-cc85ce37b4256966e6f3a3559239c5c0.ttf (0.0.0.0) 1.80ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-90d3804f0231704c15ccc5861245e8ce.woff (0.0.0.0) 3.93ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-cc85ce37b4256966e6f3a3559239c5c0.ttf (0.0.0.0) 1.60ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-90d3804f0231704c15ccc5861245e8ce.woff (0.0.0.0) 3.46ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-cc85ce37b4256966e6f3a3559239c5c0.ttf (0.0.0.0) 2.06ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-90d3804f0231704c15ccc5861245e8ce.woff (0.0.0.0) 4.08ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-cc85ce37b4256966e6f3a3559239c5c0.ttf (0.0.0.0) 1.54ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-90d3804f0231704c15ccc5861245e8ce.woff (0.0.0.0) 2.86ms
WARNING:tornado.access:404 GET /assets/fonts/Roboto-Thin-cc85ce37b4256966e6f3a3559239c5c0.ttf (0.0.0.0) 1.25ms
kubectl logs elastickube-server-0pz5j elastickube-api --namespace=kube-system
Initializing
MongoDB shell version: 3.2.6
connecting to: 10.254.101.14:27017/admin
bye
INFO:root:Reading token from '/var/run/secrets/kubernetes.io/serviceaccount/token'.
DEBUG:root:Building available metrics
INFO:root:Initializing database...
DEBUG:root:Initial Settings document created, 57514c513313d60010e4f509
INFO:root:Initializing SyncNamespaces
INFO:root:start_sync SyncNamespaces
INFO:root:Initializing SyncMetrics
INFO:root:start_sync SyncMetrics
INFO:root:Initializing watcher...
INFO:root:Watching from timestamp: 2016-06-03 09:22:26+00:00
DEBUG:root:Tailable cursor recreated.
INFO:root:Initializing MainWebSocketHandler
INFO:root:Initializing LogsActions
INFO:root:Initializing InstancesActions
INFO:root:Initializing NamespacesActions
INFO:root:Initializing SettingsActions
INFO:root:Initializing UsersActions
INFO:root:Initializing InviteActions
INFO:root:Closing MainWebSocketHandler
WARNING:root:Disconnected from kubeclient in SyncNamespaces
INFO:root:Initializing MainWebSocketHandler
INFO:root:Initializing LogsActions
INFO:root:Initializing InstancesActions
INFO:root:Initializing NamespacesActions
INFO:root:Initializing SettingsActions
INFO:root:Initializing UsersActions
INFO:root:Initializing InviteActions
INFO:root:Closing MainWebSocketHandler
Environment variables
printenv
HEAPSTER_SERVICE_PORT=80
MONITORING_INFLUXDB_PORT_8083_TCP_PROTO=tcp
HOSTNAME=elastickube-server-0pz5j
GPG_KEY=C01E1CAD5EA2C4F0B8E35745546C367C218ADD4FF
KUBE_DNS_PORT_53_UDP_ADDR=10.254.0.10
ELASTICKUBE_SERVER_PORT_80_TCP_ADDR=10.254.51.59
KUBE_DNS_PORT_53_UDP_PROTO=udp
KUBERNETES_PORT_443_TCP_PORT=443
ELASTICKUBE_SERVER_PORT_80_TCP_PROTO=tcp
MONITORING_INFLUXDB_SERVICE_PORT_HTTP=8083
KUBERNETES_PORT=tcp://10.254.0.1:443
KUBE_DNS_SERVICE_PORT=53
MONITORING_GRAFANA_PORT=tcp://10.254.42.145:80
KUBERNETES_DASHBOARD_PORT_80_TCP_ADDR=10.254.8.75
KUBERNETES_SERVICE_PORT=443
HEAPSTER_SERVICE_HOST=10.254.138.182
ELASTICKUBE_PATH=/opt/elastickube
KUBERNETES_SERVICE_HOST=10.254.0.1
ELASTICKUBE_MONGO_SERVICE_PORT=27017
ELASTICKUBE_MONGO_PORT=tcp://10.254.101.14:27017
MONITORING_INFLUXDB_PORT_8083_TCP_ADDR=10.254.150.183
KUBERNETES_DASHBOARD_PORT=tcp://10.254.8.75:80
HEAPSTER_PORT_80_TCP_PORT=80
KUBE_DNS_SERVICE_PORT_DNS_TCP=53
KUBE_DNS_PORT_53_TCP_PORT=53
MONITORING_GRAFANA_PORT_80_TCP_PORT=80
ELASTICKUBE_MONGO_PORT_27017_TCP_ADDR=10.254.101.14
MONITORING_GRAFANA_PORT_80_TCP_PROTO=tcp
KUBE_DNS_PORT_53_TCP_PROTO=tcp
KUBERNETES_DASHBOARD_PORT_80_TCP_PORT=80
HEAPSTER_PORT=tcp://10.254.138.182:80
MONITORING_INFLUXDB_PORT_8083_TCP_PORT=8083
KUBERNETES_DASHBOARD_SERVICE_HOST=10.254.8.75
PYTHON_VERSION=2.7.11
MONITORING_INFLUXDB_SERVICE_PORT=8083
HEAPSTER_PORT_80_TCP_PROTO=tcp
MONITORING_GRAFANA_SERVICE_HOST=10.254.42.145
ELASTICKUBE_SERVER_PORT_80_TCP_PORT=80
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MONITORING_INFLUXDB_PORT_8086_TCP_ADDR=10.254.150.183
ELASTICKUBE_SERVER_SERVICE_PORT=80
KUBERNETES_DASHBOARD_PORT_80_TCP_PROTO=tcp
HEAPSTER_PORT_80_TCP_ADDR=10.254.138.182
MONITORING_INFLUXDB_SERVICE_HOST=10.254.150.183
MONITORING_INFLUXDB_PORT_8086_TCP=tcp://10.254.150.183:8086
PWD=/var/log
KUBE_DNS_SERVICE_PORT_DNS=53
LANG=C.UTF-8
KUBE_DNS_PORT_53_UDP_PORT=53
MONITORING_INFLUXDB_PORT_8083_TCP=tcp://10.254.150.183:8083
KUBE_API_TOKEN_PATH=/var/run/secrets/kubernetes.io/serviceaccount/token
HEAPSTER_PORT_80_TCP=tcp://10.254.138.182:80
MONITORING_GRAFANA_SERVICE_PORT=80
KUBE_DNS_PORT=udp://10.254.0.10:53
PYTHON_PIP_VERSION=8.1.2
ELASTICKUBE_MONGO_PORT_27017_TCP=tcp://10.254.101.14:27017
MONITORING_INFLUXDB_PORT_8086_TCP_PORT=8086
MONITORING_INFLUXDB_SERVICE_PORT_API=8086
ELASTICKUBE_SERVER_PORT=tcp://10.254.51.59:80
KUBE_DNS_PORT_53_UDP=udp://10.254.0.10:53
KUBERNETES_DASHBOARD_PORT_80_TCP=tcp://10.254.8.75:80
SHLVL=1
HOME=/root
ELASTICKUBE_MONGO_PORT_27017_TCP_PORT=27017
KUBERNETES_DASHBOARD_SERVICE_PORT=80
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
ELASTICKUBE_SERVER_SERVICE_HOST=10.254.51.59
PYTHONPATH=/opt/elastickube
MONITORING_INFLUXDB_PORT_8086_TCP_PROTO=tcp
ELASTICKUBE_MONGO_SERVICE_HOST=10.254.101.14
KUBE_DNS_PORT_53_TCP_ADDR=10.254.0.10
MONITORING_GRAFANA_PORT_80_TCP_ADDR=10.254.42.145
ELASTICKUBE_MONGO_PORT_27017_TCP_PROTO=tcp
ELASTICKUBE_SERVER_PORT_80_TCP=tcp://10.254.51.59:80
KUBE_DNS_PORT_53_TCP=tcp://10.254.0.10:53
KUBERNETES_PORT_443_TCP_ADDR=10.254.0.1
MONITORING_GRAFANA_PORT_80_TCP=tcp://10.254.42.145:80
MONITORING_INFLUXDB_PORT=tcp://10.254.150.183:8083
KUBE_DNS_SERVICE_HOST=10.254.0.10
KUBERNETES_PORT_443_TCP=tcp://10.254.0.1:443
_=/usr/bin/printenv
OLDPWD=/
Test
root@elastickube-server-0pz5j:/var/log# curl -s 10.0.0.39:8080
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/healthz",
"/healthz/ping",
"/logs/",
"/metrics",
"/resetMetrics",
"/swaggerapi/",
"/version"
]
}
kubectl exec busybox -- nslookup kubernetes
Server: 10.254.0.10
Address 1: 10.254.0.10
Name: kubernetes
Address 1: 10.254.0.1
kubectl get services --all-namespaces=true
NAMESPACE NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
default glusterfs-cluster 10.254.192.43 <none> 1/TCP <none> 28d
default kubernetes 10.254.0.1 <none> 443/TCP <none> 38d
default mysql-kmanager 10.254.15.73 <none> 3306/TCP node=mysql-kmanager 15d
kube-system elastickube-mongo 10.254.101.14 <none> 27017/TCP name=elastickube-mongo 2h
kube-system elastickube-server 10.254.51.59 80/TCP name=elastickube-server 2h
kube-system heapster 10.254.138.182 <none> 80/TCP k8s-app=heapster 2h
kube-system kube-dns 10.254.0.10 <none> 53/UDP,53/TCP k8s-app=kube-dns 29d
kube-system kubernetes-dashboard 10.254.8.75 nodes 80/TCP k8s-app=kubernetes-dashboard 24d
kube-system monitoring-grafana 10.254.42.145 <none> 80/TCP k8s-app=influxGrafana 2h
kube-system monitoring-influxdb 10.254.150.183 <none> 8083/TCP,8086/TCP k8s-app=influxGrafana 2h
Also try to:
apiVersion: v1
kind: ReplicationController
metadata:
name: elastickube-mongo
namespace: kube-system
labels:
name: elastickube-mongo
spec:
replicas: 1
selector:
name: elastickube-mongo
template:
metadata:
labels:
name: elastickube-mongo
spec:
containers:
- image: mongo
name: elastickube-mongo
args:
- --replSet=elastickube
ports:
- name: mongo
containerPort: 27017
hostPort: 27017
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/mongodb
volumes:
- name: mongo-persistent-storage
hostPath:
path: /data/mongodb
----
apiVersion: v1
kind: Service
metadata:
name: elastickube-mongo
namespace: kube-system
labels:
name: elastickube-mongo
spec:
ports:
- port: 27017
targetPort: 27017
selector:
name: elastickube-mongo
----
apiVersion: v1
kind: ReplicationController
metadata:
name: elastickube-server
namespace: kube-system
labels:
name: elastickube-server
spec:
replicas: 1
selector:
name: elastickube-server
template:
metadata:
labels:
name: elastickube-server
spec:
containers:
- name: elastickube-api
image: elasticbox/elastickube-api:latest
resources:
limits:
cpu: 100m
memory: 300Mi
volumeMounts:
- name: elastickube-run
mountPath: /var/run
env:
- name: KUBERNETES_SERVICE_HOST
value: http://10.0.0.39:8080
- name: elastickube-charts
image: elasticbox/elastickube-charts:latest
resources:
limits:
cpu: 100m
memory: 300Mi
volumeMounts:
- name: elastickube-charts
mountPath: /var/elastickube/charts
- name: elastickube-nginx
image: elasticbox/elastickube-nginx:latest
resources:
limits:
cpu: 100m
memory: 300Mi
volumeMounts:
- name: elastickube-run
mountPath: /var/run
ports:
- containerPort: 80
hostPort: 80
name: http
protocol: TCP
- name: elastickube-diagnostics
image: elasticbox/elastickube-diagnostics:latest
resources:
limits:
cpu: 10m
memory: 32Mi
volumeMounts:
- name: elastickube-run
mountPath: /var/run
volumes:
- name: elastickube-charts
hostPath:
path: /var/elastickube/charts
- name: elastickube-run
hostPath:
path: /var/run/elastickube
----
apiVersion: v1
kind: Service
metadata:
name: elastickube-server
namespace: kube-system
labels:
name: elastickube-server
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 80
selector:
name: elastickube-server
Hey @Tedezed:
Can you send us more logs of "diagnostics"? The relevant lines are likely before the ones you sent.
Checking the information you have sent this is what I can guess:
- The endpoint is correct. The Heapster check is Ok and that one depends on the
KUBERNETES_SERVICE_HOST - There seems to be a problem with authentication on the kubernetes API. The error we are getting is
401 Unauthorized. This seems like the authentication is not working. - The token should be there by the environment
KUBE_API_TOKEN_PATH=/var/run/secrets/kubernetes.io/serviceaccount/token
Could you go into the API and the Diagnostics containers and issue the following commands:
TOKEN=`cat $KUBE_API_TOKEN_PATH`
curl -v -k https://$KUBERNETES_SERVICE_HOST
curl -v -k -H "Authorization: Bearer $TOKEN" https://$KUBERNETES_SERVICE_HOST
curl -v -k -H "Authorization: Bearer $TOKEN" https://$KUBERNETES_SERVICE_HOST/api/
The idea is to check if we have visibility on the Kubernetes API. The secret given in the pod should be correct so the last calls should return a 200 exit code.
We are using this method to access the Kubernetes API: http://kubernetes.io/docs/user-guide/accessing-the-cluster/#without-kubectl-proxy
Thx, @davisein
I think I found the problem, I have not the Kubernetes API in HTTPS only in HTTP.
But the strange thing is that I get to work before without that.
It's strange indeed. I expected in that case to have a 404 error instead of 401. Maybe something else is listening on the https port.
Thanks for letting us know the result.