Cockroachdb installation with TLS enabled through certificate manager is failing
jarlva opened this issue · comments
Hi, please help with 2 issues:
- A similar issue to #147 Using my own cert-manager, the nodes fail to start due to:
MountVolume.SetUp failed for volume "certs-secret" : secret "cockroachdb-node" not found
I show these secrets.
cockroach-node-vf64d Opaque
cockroach-root-client-846kz Opaque
crdb-migrator Opaque
sh.helm.release.v1.cockroach.v1 helm.sh/release.v
- config - note the
useCertManagerV1CRDs: true. Without it I get another error:UPGRADE FAILED: unable to recognize "": no matches for kind "Certificate" in version "cert-manager.io/v1alpha2"
Update: changed my Chart.yml to have the name and alias match (name was cockroachdb and alias was cockroach) and everything came up. However, it creates self-generated certificates. Even though the values file says otherwise.
What am I doing wrong?
values.yaml:
tls:
enabled: true
certs:
selfSigner:
enabled: false
certManager: true
helm:
apiVersion: v1
name: crdb
description: CockroachDB Cluster
appVersion: "1.0"
version: 0.1
dependencies:
- name: cockroachdb
version: 10.0.8
alias: cockroachdb
repository: https://charts.cockroachdb.com
NAME: cockroachdb
LAST DEPLOYED: Tue May 2 07:28:45 2023
NAMESPACE: test-jh
STATUS: pending-install
REVISION: 1
HOOKS:
---
# Source: crdb/charts/cockroach/templates/tests/client.yaml
kind: Pod
apiVersion: v1
metadata:
name: cockroachdb-test
namespace: "test-jh"
annotations:
helm.sh/hook: test-success
spec:
restartPolicy: Never
volumes:
- name: client-certs
projected:
sources:
- secret:
name: cockroachdb-root
items:
- key: ca.crt
path: ca.crt
mode: 0400
- key: tls.crt
path: client.root.crt
mode: 0400
- key: tls.key
path: client.root.key
mode: 0400
containers:
- name: client-test
image: "cockroachdb/cockroach:v22.2.8"
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: client-certs
mountPath: /cockroach-certs
command:
- /cockroach/cockroach
- sql
- --certs-dir
- /cockroach-certs
- --host
- cockroachdb-public.test-jh
- --port
- "26257"
- -e
- SHOW DATABASES;
---
# Source: crdb/charts/cockroach/templates/job.init.yaml
kind: Job
apiVersion: batch/v1
metadata:
name: cockroachdb-init
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
app.kubernetes.io/component: init
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation
spec:
template:
metadata:
labels:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: init
spec:
securityContext:
runAsGroup: 1000
runAsUser: 1000
fsGroup: 1000
runAsNonRoot: true
restartPolicy: OnFailure
terminationGracePeriodSeconds: 0
serviceAccountName: cockroachdb
initContainers:
- name: copy-certs
image: "busybox"
imagePullPolicy: "IfNotPresent"
command:
- /bin/sh
- -c
- "cp -f /certs/* /cockroach-certs/; chmod 0400 /cockroach-certs/*.key"
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: client-certs
mountPath: /cockroach-certs/
- name: certs-secret
mountPath: /certs/
containers:
- name: cluster-init
image: "cockroachdb/cockroach:v22.2.8"
imagePullPolicy: "IfNotPresent"
# Run the command in an `while true` loop because this Job is bound
# to come up before the CockroachDB Pods (due to the time needed to
# get PersistentVolumes attached to Nodes), and sleeping 5 seconds
# between attempts is much better than letting the Pod fail when
# the init command does and waiting out Kubernetes' non-configurable
# exponential back-off for Pod restarts.
# Command completes either when cluster initialization succeeds,
# or when cluster has been initialized already.
command:
- /bin/bash
- -c
- >-
initCluster() {
while true; do
local output=$(
set -x;
/cockroach/cockroach init \
--certs-dir=/cockroach-certs/ \
--host=cockroachdb-0.cockroachdb:26257
2>&1);
local exitCode="$?";
echo $output;
if [[ "$exitCode" == "0" || "$output" == *"cluster has already been initialized"* ]]
then break;
fi
sleep 5;
done
}
initCluster;
env:
volumeMounts:
- name: client-certs
mountPath: /cockroach-certs/
volumes:
- name: client-certs
emptyDir: {}
- name: certs-secret
projected:
sources:
- secret:
name: cockroachdb-root
items:
- key: ca.crt
path: ca.crt
mode: 0400
- key: tls.crt
path: client.root.crt
mode: 0400
- key: tls.key
path: client.root.key
mode: 0400
MANIFEST:
---
# Source: crdb/charts/cockroach/templates/poddisruptionbudget.yaml
kind: PodDisruptionBudget
apiVersion: policy/v1
metadata:
name: cockroachdb-budget
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
spec:
selector:
matchLabels:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: cockroachdb
maxUnavailable: 1
---
# Source: crdb/charts/cockroach/templates/serviceaccount.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
name: cockroachdb
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
---
# Source: crdb/charts/migrator/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: crdb-migrator
labels:
app.kubernetes.io/name: crdb-migrator
helm.sh/chart: migrator-0.1.92
app.kubernetes.io/instance: cockroachdb
app.kubernetes.io/version: "1.0"
app.kubernetes.io/managed-by: Helm
app: migrator
---
# Source: crdb/charts/migrator/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: crdb-migrator
labels:
app.kubernetes.io/name: crdb-migrator
helm.sh/chart: migrator-0.1.92
app.kubernetes.io/instance: cockroachdb
app.kubernetes.io/version: "1.0"
app.kubernetes.io/managed-by: Helm
app: migrator
type: Opaque
data:
MIGRATOR_MIGRATION_PATH: "L3Zhci9saWIvbWlncmF0aW9ucw=="
MIGRATOR_MIGRATION_URL: "Y29ja3JvYWNoOi8vbWlncmF0aW9uc0Bjb2Nrcm9hY2gtMC5jb2Nrcm9hY2guY29ja3JvYWNoLnN2Yy5jbHVzdGVyLmxvY2FsOjI2MjU3L2thcHBhP3NzbG1vZGU9ZGlzYWJsZQ=="
---
# Source: crdb/charts/cockroach/templates/role.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cockroachdb
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
# Source: crdb/charts/cockroach/templates/rolebinding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cockroachdb
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cockroachdb
subjects:
- kind: ServiceAccount
name: cockroachdb
namespace: "test-jh"
---
# Source: crdb/charts/cockroach/templates/service.discovery.yaml
# This service only exists to create DNS entries for each pod in
# the StatefulSet such that they can resolve each other's IP addresses.
# It does not create a load-balanced ClusterIP and should not be used directly
# by clients in most circumstances.
kind: Service
apiVersion: v1
metadata:
name: cockroachdb
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
app.kubernetes.io/component: cockroachdb
annotations:
# Use this annotation in addition to the actual field below because the
# annotation will stop being respected soon, but the field is broken in
# some versions of Kubernetes:
# https://github.com/kubernetes/kubernetes/issues/58662
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
# Enable automatic monitoring of all instances when Prometheus is running
# in the cluster.
prometheus.io/scrape: "true"
prometheus.io/path: _status/vars
prometheus.io/port: "8080"
spec:
clusterIP: None
# We want all Pods in the StatefulSet to have their addresses published for
# the sake of the other CockroachDB Pods even before they're ready, since they
# have to be able to talk to each other in order to become ready.
publishNotReadyAddresses: true
ports:
# The main port, served by gRPC, serves Postgres-flavor SQL, inter-node
# traffic and the CLI.
- name: "grpc"
port: 26257
targetPort: grpc
# The secondary port serves the UI as well as health and debug endpoints.
- name: "http"
port: 8080
targetPort: http
selector:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: cockroachdb
---
# Source: crdb/charts/cockroach/templates/service.public.yaml
# This Service is meant to be used by clients of the database.
# It exposes a ClusterIP that will automatically load balance connections
# to the different database Pods.
kind: Service
apiVersion: v1
metadata:
name: cockroachdb-public
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
app.kubernetes.io/component: cockroachdb
annotations:
service.alpha.kubernetes.io/app-protocols: '{"http":"HTTPS"}'
spec:
type: "ClusterIP"
ports:
# The main port, served by gRPC, serves Postgres-flavor SQL, inter-node
# traffic and the CLI.
- name: "grpc"
port: 26257
targetPort: grpc
# The secondary port serves the UI as well as health and debug endpoints.
- name: "http"
port: 8080
targetPort: http
selector:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: cockroachdb
---
# Source: crdb/charts/cockroach/templates/statefulset.yaml
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: cockroachdb
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
app.kubernetes.io/component: cockroachdb
spec:
serviceName: cockroachdb
replicas: 3
updateStrategy:
type: RollingUpdate
podManagementPolicy: "Parallel"
selector:
matchLabels:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: cockroachdb
template:
metadata:
labels:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: cockroachdb
spec:
serviceAccountName: cockroachdb
initContainers:
- name: copy-certs
image: "busybox"
imagePullPolicy: "IfNotPresent"
command:
- /bin/sh
- -c
- "cp -f /certs/* /cockroach-certs/; chmod 0400 /cockroach-certs/*.key"
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: certs
mountPath: /cockroach-certs/
- name: certs-secret
mountPath: /certs/
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: service
operator: In
values:
- gke-shared
- general
- key: cloud.google.com/gke-spot
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: cockroachdb
topologySpreadConstraints:
- labelSelector:
matchLabels:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: cockroachdb
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
# No pre-stop hook is required, a SIGTERM plus some time is all that's
# needed for graceful shutdown of a node.
terminationGracePeriodSeconds: 60
containers:
- name: db
image: "cockroachdb/cockroach:v22.2.8"
imagePullPolicy: "IfNotPresent"
args:
- shell
- -ecx
# The use of qualified `hostname -f` is crucial:
# Other nodes aren't able to look up the unqualified hostname.
#
# `--join` CLI flag is hardcoded to exactly 3 Pods, because:
# 1. Having `--join` value depending on `statefulset.replicas`
# will trigger undesired restart of existing Pods when
# StatefulSet is scaled up/down. We want to scale without
# restarting existing Pods.
# 2. At least one Pod in `--join` is enough to successfully
# join CockroachDB cluster and gossip with all other existing
# Pods, even if there are 3 or more Pods.
# 3. It's harmless for `--join` to have 3 Pods even for 1-Pod
# clusters, while it gives us opportunity to scale up even if
# some Pods of existing cluster are down (for whatever reason).
# See details explained here:
# https://github.com/helm/charts/pull/18993#issuecomment-558795102
- >-
exec /cockroach/cockroach
start --join=${STATEFULSET_NAME}-0.${STATEFULSET_FQDN}:26257,${STATEFULSET_NAME}-1.${STATEFULSET_FQDN}:26257,${STATEFULSET_NAME}-2.${STATEFULSET_FQDN}:26257
--advertise-host=$(hostname).${STATEFULSET_FQDN}
--certs-dir=/cockroach/cockroach-certs/
--http-port=8080
--port=26257
--cache=25%
--max-sql-memory=25%
--logtostderr=INFO
env:
- name: STATEFULSET_NAME
value: cockroachdb
- name: STATEFULSET_FQDN
value: cockroachdb.test-jh.svc.cluster.local
- name: COCKROACH_CHANNEL
value: kubernetes-helm
ports:
- name: grpc
containerPort: 26257
protocol: TCP
- name: http
containerPort: 8080
protocol: TCP
volumeMounts:
- name: datadir
mountPath: /cockroach/cockroach-data/
- name: certs
mountPath: /cockroach/cockroach-certs/
livenessProbe:
httpGet:
path: /health
port: http
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 5
readinessProbe:
httpGet:
path: /health?ready=1
port: http
scheme: HTTPS
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 2
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
- name: certs
emptyDir: {}
- name: certs-secret
projected:
sources:
- secret:
name: cockroachdb-node
items:
- key: ca.crt
path: ca.crt
mode: 256
- key: tls.crt
path: node.crt
mode: 256
- key: tls.key
path: node.key
mode: 256
securityContext:
fsGroup: 1000
runAsGroup: 1000
runAsUser: 1000
runAsNonRoot: true
volumeClaimTemplates:
- metadata:
name: datadir
labels:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "standard-wait"
resources:
requests:
storage: "10Gi"
---
# Source: crdb/charts/migrator/templates/job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: crdb-migrator-2023-05-02-07-28-45
labels:
app.kubernetes.io/name: crdb-migrator
helm.sh/chart: migrator-0.1.92
app.kubernetes.io/instance: cockroachdb
app.kubernetes.io/version: "1.0"
app.kubernetes.io/managed-by: Helm
app: migrator
spec:
ttlSecondsAfterFinished: 600
backoffLimit: 6
template:
metadata:
annotations:
checksum/secret: fc67c3315cad90bfa31bef8199fc2a5ecca138230979ecd34e9e303f958d55fb
labels:
app.kubernetes.io/name: crdb-migrator
app.kubernetes.io/instance: cockroachdb
app: migrator
spec:
containers:
- env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: POD_SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
- name: MIGRATOR_MIGRATION_PATH
valueFrom:
secretKeyRef:
key: MIGRATOR_MIGRATION_PATH
name: crdb-migrator
- name: MIGRATOR_MIGRATION_URL
valueFrom:
secretKeyRef:
key: MIGRATOR_MIGRATION_URL
name: crdb-migrator
image: us-docker.pkg.dev/wk-artifacts/containers/crdb-migrator:latest
name: crdb-migrator
dnsConfig:
options:
- name: ndots
value: "2"
nodeSelector:
service: gke-shared
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: service
operator: In
values:
- gke-shared
- key: cloud.google.com/gke-spot
operator: DoesNotExist
restartPolicy: Never
terminationGracePeriodSeconds: 7
serviceAccountName: crdb-migrator
---
# Source: crdb/charts/cockroach/templates/certificate.client.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cockroachdb-root-client
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
spec:
duration: 672h
renewBefore: 48h
usages:
- digital signature
- key encipherment
- client auth
privateKey:
algorithm: RSA
size: 2048
commonName: root
subject:
organizations:
- Cockroach
secretName: cockroachdb-root
issuerRef:
name: cockroachdb
kind: Issuer
group: cert-manager.io
---
# Source: crdb/charts/cockroach/templates/certificate.node.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cockroachdb-node
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
spec:
duration: 8760h
renewBefore: 168h
usages:
- digital signature
- key encipherment
- server auth
- client auth
privateKey:
algorithm: RSA
size: 2048
commonName: node
subject:
organizations:
- Cockroach
dnsNames:
- "localhost"
- "127.0.0.1"
- "cockroachdb-public"
- "cockroachdb-public.test-jh"
- "cockroachdb-public.test-jh.svc.cluster.local"
- "*.cockroachdb"
- "*.cockroachdb.test-jh"
- "*.cockroachdb.test-jh.svc.cluster.local"
secretName: cockroachdb-node
issuerRef:
name: cockroachdb
kind: Issuer
group: cert-manager.io
---
# Source: crdb/charts/cockroach/templates/serviceMonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: cockroachdb
namespace: "test-jh"
labels:
helm.sh/chart: cockroach-10.0.8
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/managed-by: "Helm"
spec:
selector:
matchLabels:
app.kubernetes.io/name: cockroach
app.kubernetes.io/instance: "cockroachdb"
app.kubernetes.io/component: cockroachdb
namespaceSelector:
any: true
endpoints:
- port: "http"
path: /_status/vars
interval: 15s