gardener / gardener

How to categorize this issue?

/area os
/kind bug

What happened:

I can set additional CA certificates using the .spec.caBundle property in the cloudprofile. This should put additional CA certificates on my worker nodes for communicating with services using our own CA for signing.

When I put our Root-CA in that property, the download-cloud-config.sh script still fails with the following error message:

Checking wether we need to preload a new hyperkube image...
Preloading hypercube image (<REDACTED>) because last downloaded image () is outdated
Error response from daemon: Get "<REDACTED>": x509: certificate signed by unknown authority
hyperkube image preload failed

When I check the execute-cloud-config.tpl.sh script, I see that the script tries to pull the hyperkube image before applying the cloud config (including the needed CA certificates).

At this point the hyperkube image is being pulled:

gardener/pkg/component/extensions/operatingsystemconfig/executor/templates/scripts/execute-cloud-config.tpl.sh

Lines 103 to 107 in cc65556

    
           echo "Preloading hyperkube image ({{ .hyperkubeImage }}) because last downloaded image ($LAST_DOWNLOADED_HYPERKUBE_IMAGE) is outdated" 
        
           if ! {{ .pathDockerBinary }} pull "{{ .hyperkubeImage }}" ; then 
        
             echo "hyperkube image preload failed" 
        
             exit 1 
        
           fi

But only at this later point, the cloud-config is applied, which includes the CA certificate(s) I have set with .spec.caBundle in the cloudprofile:

gardener/pkg/component/extensions/operatingsystemconfig/executor/templates/scripts/execute-cloud-config.tpl.sh

Line 252 in cc65556

if {{ .reloadConfigCommand }}; then

What you expected to happen:

The CA certificates which I set using .spec.caBundle should be placed on the worker nodes before the hyperkube image is pulled.
Otherwise the only chance to get the worker nodes bootstrapped is to use a custom VM image, which already includes my CA certificates.

How to reproduce it (as minimally and precisely as possible):

use a custom image registry with a custom CA used for signing the registries certificate
set the CA certificate in the .spec.caBundle property within the cloudprofile
create a shoot-cluster

Environment:

Gardener version:
v1.84.0
Kubernetes version (use kubectl version):
runtime-cluster: v1.24.9+vmware.1
garden-cluster: v1.27.8
Cloud provider or hardware configuration:
vSphere provider

Hi @nitrocb.
First of all - we are in the process of deprecating and removing the cloud-config-downloader as part of #8023.
Secondly, the .spec.caBundle field is not supposed to cover such use case - hence, I wouldn't consider this a bug. It's purpose is to add CA bundles after the typical bootstrapping/provisioning process of the systemd units has been completed.
Thirdly, I think the proper way to achieve what you need is to write an extension that serves a webhook which implements EnsureAdditionalProvisionFiles which adds the root CA bundle before the images are pulled. You could take provider-local as an example for how to implement such a webhook.
Hence, I would propose to close this issue.

Hi @rfranzke,

thank you very much for the clarification!
I will have a look at how to create an appropriate Extension for ensureAdditionalProvisionFiles then.

You may close this issue appropriately.

Best regards

Thanks!
/close

@rfranzke: Closing this issue.

In response to this:

Thanks!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

	echo "Preloading hyperkube image ({{ .hyperkubeImage }}) because last downloaded image ($LAST_DOWNLOADED_HYPERKUBE_IMAGE) is outdated"
	if ! {{ .pathDockerBinary }} pull "{{ .hyperkubeImage }}" ; then
	echo "hyperkube image preload failed"
	exit 1
	fi

worker node cloud-config pulls hyperkube image before CA certificates have been configured