RKE custom_certs does not work.

[jrowinski3d]

RKE version:
v1.4.10

Docker version: (docker version,docker info preferred)

Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:44 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:44 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 nvidia:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

5.15.0-84-generic

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Bare-metal, 3x hosts running Ubuntu 22.04

cluster.yml file:

default after rke config

Steps to Reproduce:

rke cert generate-csr
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./cluster_certs/kube-service-account-token-key.pem -out ./cluster_certs/kube-service-account-token.pem
rke up --custom-certs

Results:

DEBU[0001] [certificates] reading file kube-admin-csr.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-admin-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-apiserver-csr.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-apiserver-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-apiserver-proxy-client-csr.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-apiserver-proxy-client-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-controller-manager-csr.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-controller-manager-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-etcd-10-2-78-209-csr.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-etcd-10-2-78-209-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-node-csr.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-node-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-proxy-csr.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-proxy-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-scheduler-csr.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-scheduler-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-service-account-token-key.pem from directory [./cluster_certs] 
DEBU[0001] [certificates] reading file kube-service-account-token.pem from directory [./cluster_certs] 
DEBU[0001] Certificate file [./cluster_certs/kube-service-account-token.pem] content is greater than 0 
FATA[0001] Failed to validates certificates from dir [./cluster_certs]: Failed to find master CA certificate 

[Sebastiaan van Steenis]

The error is valid, it requires kube-ca.pem in the directory as described on https://rke.docs.rancher.com/installation/certs/.

But while I was verifying this, it should be better worded as its unclear. Currently it is just this part that refers to it but the file listing in the end is "incomplete".

You can use them to sign the certificates by a real CA. After the certificates are signed, those certificates can be used by RKE as custom certificates.

And we should make the logging better as well.

[jrowinski3d]

Hi @superseb ,

Thank you for your reply. Indeed this file is missing in our default cluster_certs directory. I noticed that when doing any operations on my development cluster with just a rke up, the certificates constantly change. I was hoping to use the rke cert generate-csr to pin our certificates. Should the private key not be generated during this operation?

The documentation is a bit unclear to me when just trying to do a self-signed certificate.

Do I need to generate my private key first into the default location, run the generate-csr and then sign it against the key? Any help would be appreciated in trying to figure this out to run our own self signed certificate.

[Sebastiaan van Steenis]

@jrowinski3d Sorry for the delay.

Here are the notes for getting this working, I will be adding this to the docs in a cleaner form but it should help for now:

# On Ubuntu Linux
# Prerequisites
apt install -y jq
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
sudo install -o root -g root -m 0755 cfssl_1.6.4_linux_amd64 /usr/local/bin/cfssl

# Create cluster.yml && generate CSR
vi cluster.yml
rke cert generate-csr
cd cluster_certs

# Create CA
# Make sure you specify `kube-ca` for CN question.
openssl genrsa -out kube-ca-key.pem 4096
openssl req -x509 -new -nodes -key kube-ca-key.pem -sha256 -days 1024 -out kube-ca.pem

# Make sure you specify `kube-apiserver-requestheader-ca` for CN question.
openssl genrsa -out kube-apiserver-requestheader-ca-key.pem 4096
openssl req -x509 -new -nodes -key kube-apiserver-requestheader-ca-key.pem -sha256 -days 1024 -out kube-apiserver-requestheader-ca.pem

cfssl sign -ca kube-ca.pem -ca-key kube-ca-key.pem -csr kube-apiserver-csr.pem -loglevel=5 | jq .cert -r > kube-apiserver.pem
cfssl sign -ca kube-ca.pem -ca-key kube-ca-key.pem -csr kube-controller-manager-csr.pem -loglevel=5 | jq .cert -r > kube-controller-manager.pem
cfssl sign -ca kube-ca.pem -ca-key kube-ca-key.pem -csr kube-scheduler-csr.pem -loglevel=5 | jq .cert -r > kube-scheduler.pem
cfssl sign -ca kube-ca.pem -ca-key kube-ca-key.pem -csr kube-proxy-csr.pem -loglevel=5 | jq .cert -r > kube-proxy.pem
cfssl sign -ca kube-ca.pem -ca-key kube-ca-key.pem -csr kube-node-csr.pem -loglevel=5 | jq .cert -r > kube-node.pem
cfssl sign -ca kube-ca.pem -ca-key kube-ca-key.pem -csr kube-admin-csr.pem -loglevel=5 | jq .cert -r > kube-admin.pem
cfssl sign -ca kube-ca.pem -ca-key kube-ca-key.pem -csr kube-etcd-x-x-x-x-csr.pem -loglevel=5 | jq .cert -r > kube-etcd-x-x-x-x.pem
cfssl sign -ca kube-apiserver-requestheader-ca.pem -ca-key kube-apiserver-requestheader-ca-key.pem -csr kube-apiserver-proxy-client-csr.pem -loglevel=5 | jq .cert -r > kube-apiserver-proxy-client.pem

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout kube-service-account-token-key.pem -out kube-service-account-token.pem

# Run rke up
rke up --custom-certs --cert-dir=cluster_certs/

[github-actions]

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.