Offline validation using signatures

[raffis]

Is your feature request related to a problem? Please describe.
9 out of 10 people have problems sealing secrets correctly. Correct namespace, correct cluster and so on.
That statement is obviously an overreaction. However that said I'd like to validate SealedSecrets before they are applied to clusters.

Now I am aware that there is the --validate flag but that requires the ability to talk to the correct cluster. That is fine for human-to-cluster but not for validations on ci pipelines.
Speaking best practices a ci pipeline should never be able to talk to a cluster directly which kinda defeats the point of using that way of validation.

Describe the solution you'd like
I did not exactly think that through but wouldn't we be able to add signatures to the generated SealedSecrets? If the ci or whatever knows the public key the SealedSecret could be validated if its encrypted correctly for the intended cluster and scope while not having the risk of any exposures nor the ability to decrypt it.

[M. Essam]

I would also love to see this happening, however there are a few issues with this approach that I can see:

1. Encryption keys get rotated and there's usually more than 1 at a given time in the cluster as the controller doesn't enforce rotation of all sealedsecrets when keys are rotated, so you'd have to have some part of your ci commu icate with the cluster to get the signatures themselves either way.
2. The signature would need to be added to the value of the sealedsecret encrypted data in a way that is probably incompatible with the current schema, to account for people sealing data in --raw.

Point (1) could be circumvented by having the controller generate a unique UUID for itself on the first startup (to identify itself), store it in some configmap/secret, and use that instead of a signature, (i.e to say this specific controller sealed this specific field).

Point (2) however, would require changes that would most likely not be backward compatible, for example it could possibly change the secret value to be signature;base64EncodedSecret, the signature would be mostly useless to the controller (other than maybe fast failing decryption and not retrying).

This could be done behind a feature flag (that's disabled by default) to ensure proper backward compatibility.

The above only ensures that the cluster is correct, however it cannot ensure that the scope/namespace/name are correct (e.g seal a secret for default/some-secret but paste it in default/some-other-secret) and it will appear as valid in the CI, this could be further added in the sealedsecret format as a base64 encoded json in the same format as above base64EncodedJson;base64EncodedSecret with the json containing:

• Scope of seal
• Namespace of seal (if scope is namespace)
• Secret name
• Controller signature

Not sure if this is the best idea (both implementation and security-wise) but this should resolve the issue and allow CI to verify sealed secrets without decrypting them.

[M. Essam]

@alemorcuq sorry for mentioning you here but wanted to get your input on my approach above, if it seems good I can go ahead with implementing it

[raffis]

@mohamed-essam thanks for your input. I do like the implementation proposal. Security wise it sounds good to me.
Feature flag is likely a must here.

[M. Essam]

Just to put it in more of actions to be done:

• kubeseal
  • Add feature flag for adding validation data to encrypted secrets (e.g --enable-offline-validation) (optional: allow enabling this feature flag with env var as well)
    • Get controller UUID (if UUID is on a standalone API that isn't called already)
    • Prefix the generated secret when that flag is enabled with base64 encoded JSON of scope, namespace (optional), secret name, and controller UUID
  • Add --validate=offline to validate only using the data included in secrets, skip secrets that don't contain validation data but output warning
• controller
  • Support unsealing secrets containing validation data (i.e if secret contains ;, split and use second part only)
  • Generate UUID for cluster if doesn't exist and save it in configmap
  • Add API for UUID/return UUID as part of an already existing API if it makes sense

[Alejandro Moreno]

While offline validation is an interesting topic to discuss, I have a few concerns with this approach:

Firstly, the UUID being valid doesn't necessarily guarantee that the SealedSecret can be decrypted by the controller. The UUID could have been tampered with, or the public key might have been removed from the cluster. In such scenarios, a "validated" SealedSecret could actually turn out to be invalid. Conversely, an "invalid" SealedSecret might unexpectedly be valid. This is no different to manually annotating the SealedSecret with metadata on the target cluster - it might offer some assistance, but it doesn't provide any solid guarantees.

Secondly, I believe this introduces too much complexity for a validation process that essentially boils down to a "could be valid/invalid" scenario, both for users and the underlying codebase.

We are open to explore new ways of validating a SealedSecret, but in my opinion this is not the way forward. I appreciate the effort you have put on this proposal, @mohamed-essam, and I thank you for it, but we have decided not to implement it at this time.

In the meantime, @raffis, have you considered using an Ingress to expose the /v1/verify endpoint of the Sealed Secrets controller? This could provide a way for you to validate your SealedSecrets without granting your CI full access to the cluster.

Regards,
Alejandro

[raffis]

Too bad. I see the point though.

In the meantime, @raffis, have you considered using an Ingress to expose the /v1/verify endpoint of the Sealed Secrets

Its not really a nice solution. A gitops pipeline should not be aware of where things go. Its a pull mechanism not a push one. Hence these are just workarounds but not solutions. Also gitops pipelines should be able to run without any network connection to the receiving cluster, ingress or not doesn't matter.
(I'm aware with sealedsecrets its still kinda aware where it should go but with a signature validation it could at least be fully decoupled from clusters)