App Framework: ValidateAppFrameworkSpec fails for maxConcurrentAppDownloads

Question

App Framework: ValidateAppFrameworkSpec fails for maxConcurrentAppDownloads

PaintedShepherd opened this issue 5 months ago · comments

PaintedShepherd commented 5 months ago

Please select the type of request

Bug

Tell us more

Describe the request

We have installed an splunk operator followed by an enterprise deployment.
Unfortunately since we enabled the app repos, to download apps from our s3-bucket, the operator is throwing the following error:

We have an OIDC provider to use an iam-role with a policy to access the s3-bucket.
We also added a remote access key, because the operator complained, that there is no secretRef set.

Error Log of Splunk Operator

2024-01-09T13:23:36.742227343Z INFO Starting workers {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "worker count": 15} 2024-01-09T13:23:36.742393296Z INFO start {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "MonitoringConsole": {"name":"mc","namespace":"splunk"}, "namespace": "splunk", "name": "mc", "reconcileID": "523eeb55-ac23-4c8f-b518-802a890aeccf", "monitoringconsole": "splunk/mc", "CR version": "1036624"} 2024-01-09T13:23:36.742429568Z INFO ValidateAppFrameworkSpec configCheck {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "MonitoringConsole": {"name":"mc","namespace":"splunk"}, "namespace": "splunk", "name": "mc", "reconcileID": "523eeb55-ac23-4c8f-b518-802a890aeccf", "scope": true} 2024-01-09T13:23:36.742445208Z INFO ValidateAppFrameworkSpec Invalid value of maxConcurrentAppDownloads {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "MonitoringConsole": {"name":"mc","namespace":"splunk"}, "namespace": "splunk", "name": "mc", "reconcileID": "523eeb55-ac23-4c8f-b518-802a890aeccf", "configured value": 0, "Setting it to default value": 5} 2024-01-09T13:23:36.742503399Z INFO Observed a panic in reconciler: assignment to entry in nil map {"controller": "monitoringconsole", "controllerGroup": "enterprise.splunk.com", "controllerKind": "MonitoringConsole", "MonitoringConsole": {"name":"mc","namespace":"splunk"}, "namespace": "splunk", "name": "mc", "reconcileID": "523eeb55-ac23-4c8f-b518-802a890aeccf"} panic: assignment to entry in nil map [recovered] panic: assignment to entry in nil map

goroutine 786 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1e5
panic({0x19b6fe0?, 0x2046e50?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/splunk/splunk-operator/pkg/splunk/enterprise.validateSplunkAppSources(0xc000aba230, 0x1, {0x184b609, 0x11})
/workspace/pkg/splunk/enterprise/configuration.go:1474 +0x305
github.com/splunk/splunk-operator/pkg/splunk/enterprise.ValidateAppFrameworkSpec({0x2065858, 0xc000017ef0}, 0xc000aba230, 0xc000aba2f8, 0x1, {0x184b609, 0x11})
/workspace/pkg/splunk/enterprise/configuration.go:1575 +0x72f
github.com/splunk/splunk-operator/pkg/splunk/enterprise.validateMonitoringConsoleSpec({0x2065858, 0xc000017ef0}, {0x7f27ac05c688, 0xc0009958c0}, 0xc000ab9b00)
/workspace/pkg/splunk/enterprise/monitoringconsole.go:212 +0x137
github.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplyMonitoringConsole({0x2065858, 0xc000017ef0}, {0x7f27ac05c688, 0xc0009958c0}, 0xc000ab9b00)
/workspace/pkg/splunk/enterprise/monitoringconsole.go:58 +0x1ad
github.com/splunk/splunk-operator/controllers.glob..func6({0x2065858, 0xc000017ef0}, {0x206d598?, 0xc0009958c0}, 0x5?)
/workspace/controllers/monitoringconsole_controller.go:114 +0x52
github.com/splunk/splunk-operator/controllers.(*MonitoringConsoleReconciler).Reconcile(0xc000a1a738, {0x2065858, 0xc000017ef0}, {{{0xc000aa2cc8, 0x6}, {0xc000aa2cc6, 0x2}}})
/workspace/controllers/monitoringconsole_controller.go:104 +0x55b

Expected behavior

No more error and apps will be installed from our private and encrypted s3-bucket.

Splunk setup on K8S

Details of the Splunk setup on the K8s cluster.

This is our values.yaml of the splunk operator and the enterprise deployment. Note that it is converted to typescript/json, because we are deploying with AWS CDK.

Splunk Operator Values Override

        image: {
          repository: 'REPONAME/splunk/splunk:' + (props.splunkImageVersion ?? '9.1.1'),
        },
        kubeRbacProxy: {
          image: {
            repository: 'REPONAME/kubebuilder/kube-rbac-proxy',
            tag: props.rbacImageVersion ?? 'v0.13.1',
          },
        },
        splunkOperator: {
          image: {
            repository: 'REPONAME/splunk/splunk-operator:' + (props.splunkOperatorVersion ?? '2.4.0'),
          },
          annotations:{
            "eks.amazonaws.com/role-arn": serviceAccount.role.roleArn
          },
          podAnnotations: {
            "eks.amazonaws.com/role-arn": serviceAccount.role.roleArn
          }
        },

Splunk Enterprise Values Override

    image: {
      repository: 'REPONAME/splunk/splunk:'+ (props.splunkImageVersion ?? '9.1.1'),
    },
    'splunk-operator': {
      enabled: false,
    },
    clusterManager: {
      enabled: true,
      serviceAccount: serviceAccount.serviceAccountName,
      appRepo:
      {
        appsRepoPollIntervalSeconds: 900,
        defaults:
        {
          volumeName: "volume_app_repo_zlom-sandbox_idx",
          scope: "cluster"
        },
        appSources:
          [
            {
              name: "platform-apps",
              location: "platform-apps/"
            },
            {
              name: "consumer-apps",
              location: "consumer-apps/"
            }
          ],
        volumes: [
          {
            name: "volume_app_repo_zlom-sandbox_idx",
            storageType: "s3",
            provider: "aws",
            path: "our-s3-bucket/idx/",
            endpoint: "https://s3-eu-central-1.amazonaws.com",
            region: "eu-central-1",
            secretRef: "s3-secret"
          }
        ],
      },
      etcVolumeStorageConfig: {
        storageClassName: 'ebs-csi-default',
      },
      varVolumeStorageConfig: {
        storageClassName: 'ebs-csi-default',
      },
      licenseUrl: 'LICENSE_SERVER',
      defaults: {
        splunk: {
          idxc: {
            replication_factor: 2,
            search_factor: 2,
          },
        },
      },
      resources: {
        requests: {
          memory: '4Gi',
          cpu: '2',
        },
        limits: {
          memory: '4Gi',
          cpu: '2',
        },
      },
    },
    indexerCluster: {
      enabled: true,
      serviceAccount: serviceAccount.serviceAccountName,
      replicaCount: 1,
      licenseUrl: 'LICENSE_SERVER',
      etcVolumeStorageConfig: {
        ephemeralStorage: false,
        storageCapacity: '10Gi',
        storageClassName: 'ebs-csi-default',
      },
      varVolumeStorageConfig: {
        ephemeralStorage: false,
        storageCapacity: '100Gi',
        storageClassName: 'ebs-csi-default',
      },
      resources: {
        requests: {
          memory: '12Gi',
          cpu: '6',
        },
        limits: {
          memory: '12Gi',
          cpu: '6',
        },
      },
    },
    searchHeadCluster: {
      enabled: true,
      serviceAccount: serviceAccount.serviceAccountName,
      appRepo:
      {
        appsRepoPollIntervalSeconds: 900,
        defaults:
        {
          volumeName: "volume_app_repo_zlom-sandbox_sh",
          scope: "cluster"
        },
        appSources:
          [{
            name: "platform-apps",
            location: "platform-apps/"
          },
          {
            name: "consumer-apps",
            location: "consumer-apps/"
          }],
        volumes:
          [{
            name: "volume_app_repo_zlom-sandbox_sh",
            storageType: "s3",
            provider: "aws",
            path: "our-s3-bucket/sh/",
            endpoint: "https://s3-eu-central-1.amazonaws.com",
            region: "eu-central-1",
            secretRef: "s3-secret"
          }],
      },
      resources: {
        requests: {
          memory: '12Gi',
          cpu: '6',
        },
        limits: {
          memory: '12Gi',
          cpu: '6',
        },
      },
    },
    monitoringConsole: {
      enabled: true,
      appRepo:
      {
        appsRepoPollIntervalSeconds: 900,
        defaults:
        {
          volumeName: "volume_app_repo_zlom-sandbox_mc",
          scope: "cluster"
        },
        appSources:
          [{
            name: "platform-apps",
            location: "platform-apps/"
          },
          {
            name: "consumer-apps",
            location: "consumer-apps/"
          }],
        volumes:
          [{
            name: "volume_app_repo_zlom-sandbox_mc",
            storageType: "s3",
            provider: "aws",
            path: "our-s3-bucket/mc/",
            endpoint: "https://s3-eu-central-1.amazonaws.com",
            region: "eu-central-1",
            secretRef: "s3-secret"
          }],
      },        
    },
    standalone: {
      enabled: true,
      name: "hf",
      serviceAccount: serviceAccount.serviceAccountName,
      replicaCount: 2,
      appRepo:
      {
        appsRepoPollIntervalSeconds: 900,
        defaults:
        {
          volumeName: "volume_app_repo_zlom-sandbox_hf",
          scope: "cluster"
        },
        appSources:
          [{
            name: "platform-apps",
            location: "platform-apps/"
          },
          {
            name: "consumer-apps",
            location: "consumer-apps/"
          }],
        volumes:
          [{
            name: "volume_app_repo_zlom-sandbox_hf",
            storageType: "s3",
            provider: "aws",
            path: "our-s3-bucket/hf/",
            endpoint: "https://s3-eu-central-1.amazonaws.com",
            region: "eu-central-1",
            secretRef: "s3-secret"
          }],
      },
      licenseUrl: 'LICENSE_SERVER',
      etcVolumeStorageConfig: {
        storageCapacity: '10Gi',
        storageClassName: 'ebs-csi-default',
      },
      varVolumeStorageConfig: {
        storageCapacity: '100Gi',
        storageClassName: 'ebs-csi-default',
      },
      resources: {
        requests: {
          memory: '8Gi',
          cpu: '4',
        },
        limits: {
          memory: '8Gi',
          cpu: '4',
        },
      },
    },

Reproduction/Testing steps

install the operator and the splunk enterprise deployment,
activate the app repo in SH, IDX, STNDL and CM
create and add a service account to the sh, idx, stndl and cm via enterprise deployment values.yml
add annotation and podAnnotation like we did in the values of the splunk operator
add apps to the s3 bucket
watch the operator logs of the manager container

K8s environment

Useful information about the K8S environment being used. Eg. version of K8s, kind of K8s cluster etc..
Version 1.26
Operator Version 2.4.0
Splunk Version: 9.1.1

Additional context(optional)

If you need anything else, let me know.

PaintedShepherd · Answer 1 · Tue Jan 09 2024 22:27:28 GMT+0800 (China Standard Time)

I found the issue myself. It was the wrong scope of the app repos.
The MC and he SNTLN app repos has to be "local" instead of "cluster".
Hope this helps someone.