Giters

splunk / splunk-connect-for-snmp

Splunk connect for SNMP

Home Page:https://splunk.github.io/splunk-connect-for-snmp/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

SNMPv3 credentials not being able to be mounted

minhnhupham opened this issue · comments

commented

Hi,
There seems to be issues setting up SC4SNMP for polling against an SNMPv3 endpoint which requires authentication credentials.
We've been getting the following errors in our poller logs:

[2022-09-30 06:14:24,967: ERROR/ForkPoolWorker-2] Task splunk_connect_for_snmp.snmp.tasks.walk[27cdbb0e-cde7-48c4-bb42-16680d295e72] raised unexpected: Exception('invalid username from secret sc4snmp-hsm-snmp-creds')
Traceback (most recent call last):
  File "/app/.venv/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/celery/app/autoretry.py", line 34, in run
    return task._orig_run(*args, **kwargs)
  File "/app/.venv/lib/python3.10/site-packages/splunk_connect_for_snmp/snmp/tasks.py", line 78, in walk
    retry, result = self.do_work(ir, walk=True, profiles=profile)
  File "/app/.venv/lib/python3.10/site-packages/splunk_connect_for_snmp/snmp/manager.py", line 282, in do_work
    authData = GetAuth(logger, ir, self.snmpEngine)
  File "/app/.venv/lib/python3.10/site-packages/splunk_connect_for_snmp/snmp/auth.py", line 167, in GetAuth
    return getAuthV3(logger, ir, snmpEngine)
  File "/app/.venv/lib/python3.10/site-packages/splunk_connect_for_snmp/snmp/auth.py", line 147, in getAuthV3
    raise Exception(f"invalid username from secret {ir.secret}")
Exception: invalid username from secret sc4snmp-hsm-snmp-creds
[2022-09-30 06:14:24,968: DEBUG/ForkPoolWorker-2] postrun signal task_id=27cdbb0e-cde7-48c4-bb42-16680d295e72

We've verified the secrets credentials manually using snmpget and they appear to be correct.

After some debugging we noticed that the helm chart uses Projected Volumes for the snmpv3-secret volume mounts.
https://github.com/splunk/splunk-connect-for-snmp/blob/main/charts/splunk-connect-for-snmp/templates/worker/poller/deployment.yaml#L98-L137

This could potentially be the problem as when we checked the deployment manifest for the pollers we see the secret being null:

      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: config.yaml
            path: config.yaml
          name: splunk-connect-for-snmp-config
        name: config
      - name: snmpv3-secrets
        projected:
          defaultMode: 420
          sources: null
      - emptyDir: {}
        name: pysnmp-cache-volume
      - emptyDir: {}
        name: tmp

This appears to basically stops SNMPv3 secrets from working. Can someone confirm our observations above?

As a side note we are using GKE v1.21, which at least from the documentation pages, does not appear to have Projected Volumes in the Secrets spec.
Reference:

If this is in fact the issue, can this template be written in a way that doesn't require Projected Volumes?

commented

Just a follow up to this.
On further investigation we think we've identified the root cause of the issue.
We are currently using ArgoCD to deploy all applications base on helm charts into our k8s cluster.
Unfortunately ArgoCD does not currently have support for the helm3 lookup function that is being used in the chart.
Reference:

And considering it's been a 2yr old issue on the ArgoCD code repo, it doesn't look like it'll be fix any time soon.
ArgoCD uses helm template under the hood to generate/prerender object manifests rather then using helm cli.

Can we change this to using standard k8s secrets without using the helm lookup function? as this effects several of the current chart templates.
Ref:

This could also possibly effect several other systems similar to ArgoCD, such as FLUX/spinnaker, which pre-renders the charts templates into k8s object manifests before applying them into the k8s cluster rather then using the helm cli/binary.

commented

Hi, thanks for sharing all the info. So I understand in order to make SC4SNMP compatible with your system it can't use both projected volumes and lookup functions? From the documentation links, you posted it's clear that GKE v1.21 does not support projected volumes, but I want to make sure as both things refer to the same place.

For sure it's possible to introduce the change so that none of those functions is used, I'll do some experiments with that and let you know when having any working prototype.

commented

Thanks @omrozowicz-splunk.
After further testing, although not officially documented in the k8s 1.21 spec, projected volumes appears to work, the main culprit appears to be the lookup function call, since that would stop the range loop that uses projected volumes from even working in the first place.
It's probably best to remove the use of both for better backwards compatibility.

commented

Hello,
The change we discussed was merged to version 1.8.3-beta.2. Feel free to test it. It will be included in the next main version (probably in a few days). Here is how to run the beta version.