Unable to deploy grafana in AWS EKS cluster
jrphub opened this issue · comments
Hey,
I was following https://aws.amazon.com/blogs/opensource/deploy-openfaas-aws-eks/ and
I am facing with Grafana dashboard. I am using AWS EKS cluster, connecting from local ubuntu 8.04 LTS with aws configured.
$ kubectl -n openfaas run --image=stefanprodan/faas-grafana:4.6.3 --port=3000 grafana
pod/grafana created
Then when I run
kubectl expose deploy/grafana --type ClusterIP -n openfaas --port 3000
I am getting output as below
Error from server (NotFound): deployments.extensions "grafana" not found
Then I checked
kubectl get all -n openfaas
NAME READY STATUS RESTARTS AGE
pod/alertmanager-76988d48df-mf9c8 2/2 Running 0 24h
pod/basic-auth-plugin-5df9cb99f6-5nsxc 1/1 Running 0 24h
pod/faas-idler-b65944665-rjtjg 1/1 Running 2 24h
pod/gateway-58cb9bf6d7-j499b 2/2 Running 0 24h
pod/gateway-58cb9bf6d7-q7r4d 2/2 Running 0 24h
pod/grafana 1/1 Running 0 118s
pod/nats-7c4d874896-dklbq 1/1 Running 0 24h
pod/prometheus-d46884b56-9x7kh 2/2 Running 0 24h
pod/queue-worker-6b89d664f-f8jhf 1/1 Running 0 24h
pod/queue-worker-6b89d664f-zjlkb 1/1 Running 1 24h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager ClusterIP 10.100.222.55 9093/TCP 24h
service/basic-auth-plugin ClusterIP 10.100.17.159 8080/TCP 24h
service/gateway ClusterIP 10.100.160.36 8080/TCP 24h
service/gateway-external LoadBalancer 10.100.77.196 a39927ace11d247bb821ab129fba69c7-1861578323.us-east-2.elb.amazonaws.com 8080:31484/TCP 24h
service/nats ClusterIP 10.100.22.77 4222/TCP 24h
service/prometheus ClusterIP 10.100.66.231 9090/TCP 24h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/alertmanager 1/1 1 1 24h
deployment.apps/basic-auth-plugin 1/1 1 1 24h
deployment.apps/faas-idler 1/1 1 1 24h
deployment.apps/gateway 2/2 2 2 24h
deployment.apps/nats 1/1 1 1 24h
deployment.apps/prometheus 1/1 1 1 24h
deployment.apps/queue-worker 2/2 2 2 24h
NAME DESIRED CURRENT READY AGE
replicaset.apps/alertmanager-76988d48df 1 1 1 24h
replicaset.apps/basic-auth-plugin-5df9cb99f6 1 1 1 24h
replicaset.apps/faas-idler-b65944665 1 1 1 24h
replicaset.apps/gateway-58cb9bf6d7 2 2 2 24h
replicaset.apps/nats-7c4d874896 1 1 1 24h
replicaset.apps/prometheus-d46884b56 1 1 1 24h
replicaset.apps/queue-worker-6b89d664f 2 2 2 24h
I am not seeing any service or deployment for grafana
Then I checked your readme file and run
kubectl -n openfaas expose deployment grafana --type=NodePort --name=grafana
The same error
Error from server (NotFound): deployments.extensions "grafana" not found
Please help.
I used below command, now I able to see grafana dashboard
kubectl expose pod/grafana --type ClusterIP -n openfaas --port 3000
service/grafana exposed
$ kubectl port-forward -n openfaas svc/grafana 3000:3000
but now, the issue is , even if I invoke some functions, it doesn't show any metrics on dashboard.
It's all blank like below
I'm having the same problem trying to expose the deployment after starting up a pod with this image. It only starts a pod and there is no deployment... Is something missing from the readme? Or is it that this needs a deployment template? How does not everyone using this face the same problem?
EDIT (message on slack):
running kubectl version I see
client 1.18.3
server 1.17.3
Hi Both,
This looks a lot like the breaking change in kubectl 1.18 to me (you indicated 1.18 kubectl in your messages on slack)
kubectl run no longer creates a deploymeny, only a pod...
As to the fix for this... well...
export DEPLOYMENT=grafana
(cat<<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: $DEPLOYMENT
labels:
app: grafana
spec:
replicas: 1
selector:
matchLabels:
run: grafana
template:
metadata:
creationTimestamp: null
labels:
run: grafana
spec:
containers:
- image: stefanprodan/faas-grafana:4.6.3
name: grafana
ports:
- containerPort: 3000
EOF
) | kubectl apply -f -
should create the grafana deployment
You'll need expose pod/ instead of deploy/ since Kubernetes broke the CLI in 1.18
https://medium.com/p/kubernetes-1-18-broke-kubectl-run-heres-what-to-do-about-it-2a88e5fb389a
Tnx for the replies @Waterdrips @alexellis.
Exposing the pod is what I've been doing now as a work around.
@alexellis thank you for your article on this issue, it explains it really well.
@jrphub @stefanprodan guess this can be closed since it's not a problem with this repo... but I would suggest to update the readme.
Feel free to send us a PR to the OpenFaaS workshop repo if required.
Replied with result here
openfaas/workshop#177
Grafana worked after exposing pod grafana
kubectl -n openfaas expose pod grafana --type=NodePort --name=grafana
Thank you verymuch :) :)