Deployment with customDeployKeySecret results in crashing cloud connector pod.

[Michael-John Sakellaropoulos]

Describe the bug
Deploying Pixie using the Helm chart or via YAML while specifying a customDeployKeySecret results in the following error in the cloud connector pod:

time="2023-11-21T00:25:22Z" level=fatal msg="Failed to register vizier deployment. If deploying via Helm or Manifest, please check your deployment key. Otherwise, ensure you have deployed with an authorized account." error="rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR"

Specifying deployKey directly via helm values, vizier crd or CLI works as expected and deployment is successful.

To Reproduce

1. Create pl namespace first
2. Create secret px-deploy-key:

apiVersion: v1
data:
  deploy-key: _base64_encoded_deploy_key_
kind: Secret
metadata:
  name: px-deploy-key
  namespace: pl
type: Opaque

3. If using Helm, specify customDeployKeySecret: px-deploy-key in values.yaml, if using YAML, modify 06_vizier.yaml as follows:

apiVersion: px.dev/v1alpha1
kind: Vizier
metadata:
  name: pixie
  namespace: pl
spec:
  version: 0.14.7
  customDeployKeySecret: px-deploy-key
  cloudAddr: withpixie.ai:443
  disableAutoUpdate: false
  useEtcdOperator: false
  clusterName: cedille-cedille-cluster
  pemMemoryLimit: 3Gi
  dataAccess: Full
  dataCollectorParams:

Expected behavior: Possibility to use customDeployKeySecret to avoid putting the deploy key as cleartext in our repo.

Logs
pixie_logs_20231120193837.zip

App information (please complete the following information):

• Pixie version: Latest from CLI or helm chart version 0.1.4
• K8s cluster version: 1.27.3
• Node Kernel version: 6.1.51-talos (Talos Linux)
• Browser version: Not applicable

Additional context
Our repo is open source, all the pixie related configs are here: https://github.com/ClubCedille/Plateforme-Cedille/tree/master/system/pixie

[Dom Delnano]

That looks correct from what I can tell. Can you inspect the PL_DEPLOY_KEY environment variable from the cloud connector container (either via $ strings /proc/$pid/environ or from a container shell)? Looking at your pl__vizier-cloud-connector-64c99d97f4-44nbt__describe.json file it appears that the PL_DEPLOY_KEY environment variable is getting its value from the correct secret name.

  "name": "PL_DEPLOY_KEY",
  "valueFrom": {
    "secretKeyRef": {
      "name": "px-deploy-key",
      "key": "deploy-key",
      "optional": true
    }

[Michael-John Sakellaropoulos]

Thanks for your answer. I was able to confirm that the env var value was correct.
I started over from the beginning and regenerated (again) a deployment key, and now it's working
Not sure what exactly happened, I did re-deploy and regenerate the key many times before finally giving up and filing an issue. Murphy's law.
Maybe some unprintable character snuck into my base64 command

Either way, it seems to be a non issue - sorry for the trouble

[Dom Delnano]

No worries and glad it is working now!