Router crashes on launch
sfxworks opened this issue · comments
C:\Users\shaba>kubectl logs -f mc-router-5f6c9bf466-hn2jt
time="2020-07-05T23:06:39Z" level=info msg="Listening for Minecraft client connections" listenAddress=":25565"
time="2020-07-05T23:06:39Z" level=info msg="Serving API requests" binding=":8080"
time="2020-07-05T23:06:39Z" level=info msg="Monitoring kubernetes for minecraft services"
I0705 23:07:09.086513 1 trace.go:81] Trace[1298498081]: "Reflector pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94 ListAndWatch" (started: 2020-07-05 23:06:39.083161704 +0000 UTC m=+0.011299522) (total time: 30.002979398s):
Trace[1298498081]: [30.002979398s] [30.002979398s] END
log: exiting because of error: log: cannot create log: open /tmp/mc-router.mc-router-5f6c9bf466-hn2jt.unknownuser.log.INFO.20200705-230709.1: no such file or directory
kubectl version
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:39:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Same deployment spec aside from node role assignment.
Minor version difference between cluster where it works:
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
I'm not sure that log comes from mc-router itself, but rather from kubernetes. You said the node role assignment differs between the two? Can you provide that info? I wasn't sure how the kubernetes version difference was helpful.
...meanwhile I’ll see if there’s any updates for the Kubernetes client library and see if something there might avoid this issue.
Sure!
Cluster Red (not working)
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
creationTimestamp: "2020-07-05T22:27:24Z"
generation: 3
labels:
run: mc-router
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
run: mc-router
strategy:
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
run: mc-router
spec:
containers:
- args:
- --api-binding
- :8080
- --in-kube-cluster
image: itzg/mc-router:latest
imagePullPolicy: Always
name: mc-router
ports:
- containerPort: 25565
name: proxy
protocol: TCP
- containerPort: 8080
name: web
protocol: TCP
resources:
limits:
cpu: 500m
memory: 250Mi
requests:
cpu: 500m
memory: 250Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
dnsPolicy: ClusterFirst
nodeSelector:
node-role.kubernetes.io/infra: ""
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: mcsh-operator
serviceAccountName: mcsh-operator
terminationGracePeriodSeconds: 30
Cluster Blue (working)
kubectl get deployment -o yaml mc-router -n default
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
creationTimestamp: "2020-05-01T11:17:16Z"
generation: 5
labels:
run: mc-router
manager: kube-controller-manager
operation: Update
time: "2020-07-04T21:18:19Z"
name: mc-router
namespace: default
resourceVersion: "28083908"
selfLink: /apis/apps/v1/namespaces/default/deployments/mc-router
uid: 66449fdc-6c55-4134-83eb-8db659d83971
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
run: mc-router
strategy:
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
run: mc-router
spec:
containers:
- args:
- --api-binding
- :8080
- --in-kube-cluster
image: itzg/mc-router:latest
imagePullPolicy: Always
name: mc-router
ports:
- containerPort: 25565
name: proxy
protocol: TCP
- containerPort: 8080
name: web
protocol: TCP
resources:
limits:
cpu: 250m
memory: 100Mi
requests:
cpu: 100m
memory: 50Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: mc-router
serviceAccountName: mc-router
terminationGracePeriodSeconds: 30
The original for red and blue is here https://github.com/mcserverhosting-net/cluster-red/blob/master/route/mc-router.yaml
I've tried modifying the deployment for red. Tweaking it to get more logs with terminationMessagePolicy, giving it more resources, allowing it to use cluster-admin etc. The final difference is above.
I will state that Cluster red is running dual-stack with a different networking plugin, but all is resolvable using ipv4 or ipv6. Other than that, the only difference between clusters is their versions.
The names of the two serviceAccount attributes differ, but have you bound the same roles and rules to both in each cluster. Here is the binding and rules that I use:
https://github.com/itzg/mc-router/blob/master/docs/k8s-example-auto.yaml#L6-L27
FWIW I'm running on kubernetes v1.18.4-1+6f17be3f1fd54a
Yes. Sorry, to clarify I assigned it cluster-admin via another service account mc-operator that had its cluster role binding set to use cluster-admin
C:\Users\shaba>kubectl get clusterrolebinding -o yaml mcsh-operator
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2020-07-05T22:16:29Z"
name: mcsh-operator
resourceVersion: "1464902"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/mcsh-operator
uid: f64cb550-8cc9-478a-8cca-2752344e290d
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: mcsh-operator
namespace: default
I bumped the version of the kubernetes client library to the newest and built a new image itzg/mc-router:1.10.2. Can you see if that happens to fix the issue?
Ah, thanks for this!
time="2020-07-06T01:43:19Z" level=info msg="Listening for Minecraft client connections" listenAddress=":25565"
time="2020-07-06T01:43:19Z" level=info msg="Serving API requests" binding=":8080"
time="2020-07-06T01:43:19Z" level=info msg="Monitoring kubernetes for minecraft services"
I0706 01:43:49.168153 1 trace.go:116] Trace[1298498081]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125 (started: 2020-07-06 01:43:19.166955598 +0000 UTC m=+0.016972030) (total time: 30.001068629s):
Trace[1298498081]: [30.001068629s] [30.001068629s] END
E0706 01:43:49.168232 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0706 01:44:20.724306 1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125 (started: 2020-07-06 01:43:50.720976317 +0000 UTC m=+31.570992882) (total time: 30.002963771s):
Trace[1427131847]: [30.002963771s] [30.002963771s] END
E0706 01:44:20.727575 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0706 01:44:53.032678 1 trace.go:116] Trace[911902081]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125 (started: 2020-07-06 01:44:23.030096676 +0000 UTC m=+63.880113261) (total time: 30.002507822s):
Trace[911902081]: [30.002507822s] [30.002507822s] END
E0706 01:44:53.032739 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0706 01:45:28.432405 1 trace.go:116] Trace[140954425]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125 (started: 2020-07-06 01:44:58.430869391 +0000 UTC m=+99.280886082) (total time: 30.001468839s):
Trace[140954425]: [30.001468839s] [30.001468839s] END
E0706 01:45:28.432445 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0706 01:46:05.838132 1 trace.go:116] Trace[208240456]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125 (started: 2020-07-06 01:45:35.834776563 +0000 UTC m=+136.684793107) (total time: 30.003221127s):
Trace[208240456]: [30.003221127s] [30.003221127s] END
E0706 01:46:05.838211 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0706 01:46:52.491687 1 trace.go:116] Trace
Appears to be some internal networking issues with this node. The updated version helped with making this known. I will continue to troubleshoot this from here as it would be out of scope for this operator.
Thanks again and sorry for the trouble!
mcserverhosting-net $ kubectl logs -f mc-router-5458f677fd-lgdpd
time="2020-07-06T02:28:51Z" level=info msg="Listening for Minecraft client connections" listenAddress=":25565"
time="2020-07-06T02:28:51Z" level=info msg="Serving API requests" binding=":8080"
time="2020-07-06T02:28:51Z" level=info msg="Monitoring kubernetes for minecraft services"
time="2020-07-06T02:28:51Z" level=info msg="Creating route" backend="10.97.8.236:25565" serverAddress=ahr9194.play.mcserverhosting.red
time="2020-07-06T02:28:51Z" level=info msg="Creating route" backend="10.108.45.29:25565" serverAddress=buffhilter69.mcserverhosting.red
cilium/cilium#6320 Was the issue if you're curious 👍
Aha! It gave me an excuse to refresh the library and Go version :)