Nothing in the Splunk index
alithhero opened this issue · comments
Can somebody help me with this problem? I see nothing significant in the em_logs index that can explain me why I see nothing in the netops index ... Here is my value.yaml file :
All XXXXX are properly completed in my configuration.
splunk:
enabled: true
protocol: http
host: XXXXXX
token: XXXXXXXX
insecureSSL: "true"
port: "8088"
eventIndex: "myindex_netops"
metricsIndex: "myindex_netmetrics"
image:
pullPolicy: "Always"
traps:
communities:
2c:
- public
- homelab
usernameSecrets:
- XXXXXX
# - sc4snmp-hlab-sha-des
#loadBalancerIP: The IP address in the metallb pool
loadBalancerIP: 10.128.149.27
worker:
# There are 3 types of workers
trap:
# replicaCount: number of trap-worker pods which consumes trap tasks
replicaCount: 2
#autoscaling: use it instead of replicaCount in order to make pods scalable by itself
#autoscaling:
# enabled: true
# minReplicas: 2
# maxReplicas: 10
# targetCPUUtilizationPercentage: 80
poller:
# replicaCount: number of poller-worker pods which consumes polling tasks
replicaCount: 2
#autoscaling: use it instead of replicaCount in order to make pods scalable by itself
#autoscaling:
# enabled: true
# minReplicas: 2
# maxReplicas: 10
# targetCPUUtilizationPercentage: 80
sender:
# replicaCount: number of sender-worker pods which consumes sending tasks
replicaCount: 1
# autoscaling: use it instead of replicaCount in order to make pods scalable by itself
#autoscaling:
# enabled: true
# minReplicas: 2
# maxReplicas: 10
# targetCPUUtilizationPercentage: 80
# udpConnectionTimeout: timeout in seconds for SNMP operations
#udpConnectionTimeout: 5
logLevel: "INFO"
scheduler:
logLevel: "INFO"
profiles: |
generic_switch:
frequency: 300
varBinds:
- ['SNMPv2-MIB', 'sysDescr']
- ['SNMPv2-MIB', 'sysName', 0]
- ['TCP-MIB', 'tcpActiveOpens']
- ['TCP-MIB', 'tcpAttemptFails']
- ['IF-MIB']
groups: |
ciusss_group_commutateur:
- address: 10.112.104.5
- address: 10.130.44.1
- address: 10.130.32.9
- address: 10.130.32.8
- address: 10.130.32.1
- address: 10.130.32.6
- address: 10.130.32.7
- address: 10.130.40.9
- address: 10.130.40.1
- address: 10.130.40.6
- address: 10.130.40.10
- address: 10.130.40.8
- address: 10.130.40.7
- address: 10.132.92.1
- address: 10.132.92.5
- address: 10.132.92.6
- address: 10.48.180.1
- address: 10.133.0.136
- address: 10.133.0.134
- address: 10.133.0.1
- address: 10.133.0.139
- address: 10.133.0.131
- address: 10.133.0.132
- address: 10.133.0.143
- address: 10.133.0.144
- address: 10.133.0.145
- address: 10.133.0.133
- address: 10.133.0.140
- address: 10.133.0.148
- address: 10.133.0.146
- address: 10.133.0.135
- address: 10.133.0.138
- address: 10.133.4.1
- address: 10.133.8.1
- address: 10.133.16.1
- address: 10.133.24.1
- address: 10.133.20.1
- address: 10.112.88.1
- address: 10.49.168.5
- address: 10.49.168.6
- address: 10.112.136.1
- address: 10.130.192.5
- address: 10.154.69.5
- address: 10.132.136.5
- address: 10.132.136.6
- address: 10.130.164.6
- address: 10.132.144.1
- address: 10.133.64.15
- address: 10.133.64.5
- address: 10.133.64.1
- address: 10.133.64.6
- address: 10.133.64.8
- address: 10.133.64.13
- address: 10.132.156.5
- address: 10.132.184.5
- address: 10.130.176.5
- address: 10.130.132.1
- address: 10.132.140.6
- address: 10.133.140.5
- address: 10.131.60.8
- address: 10.131.60.1
- address: 10.131.60.9
- address: 10.130.208.1
- address: 10.130.180.7
- address: 10.130.180.6
- address: 10.133.76.1
- address: 10.133.76.11
- address: 10.133.76.12
- address: 10.133.88.1
- address: 10.133.84.1
- address: 10.132.128.8
- address: 10.132.128.1
- address: 10.132.128.7
- address: 10.132.128.9
- address: 10.133.96.1
- address: 10.133.128.5
- address: 10.130.168.1
- address: 10.133.100.1
- address: 10.130.160.5
- address: 10.131.4.6
- address: 10.130.204.5
- address: 10.130.240.5
- address: 10.133.92.1
- address: 10.131.52.5
- address: 10.133.104.1
- address: 10.130.128.5
- address: 10.133.116.1
- address: 10.131.20.5
- address: 10.140.64.97
- address: 10.140.64.102
- address: 10.140.64.106
- address: 10.140.64.123
- address: 10.140.64.114
- address: 10.140.64.98
- address: 10.140.64.99
- address: 10.140.68.34
- address: 10.140.68.33
- address: 10.140.68.35
- address: 10.140.68.36
- address: 10.49.100.1
- address: 10.134.128.1
- address: 10.134.128.6
- address: 10.53.166.150
- address: 10.53.166.66
- address: 10.53.166.68
- address: 10.53.166.70
- address: 10.53.166.72
- address: 10.53.166.74
- address: 10.53.166.76
- address: 10.53.166.78
- address: 10.53.166.56
- address: 10.53.166.58
- address: 10.53.166.62
- address: 10.53.166.64
- address: 10.53.166.54
- address: 10.53.166.52
- address: 10.53.166.46
- address: 10.53.166.48
- address: 10.53.166.50
- address: 10.53.166.80
- address: 10.53.166.104
- address: 10.53.166.106
- address: 10.53.166.90
- address: 10.53.166.92
- address: 10.53.166.94
- address: 10.53.166.96
- address: 10.53.166.98
- address: 10.53.166.100
- address: 10.53.166.102
- address: 10.53.166.82
- address: 10.53.166.83
- address: 10.53.166.84
- address: 10.53.166.86
- address: 10.53.166.81
- address: 10.53.166.114
- address: 10.53.166.118
- address: 10.53.166.120
- address: 10.53.166.122
- address: 10.53.166.110
- address: 10.53.166.124
- address: 10.53.166.126
- address: 10.53.166.128
- address: 10.53.166.130
- address: 10.53.166.132
- address: 10.53.166.108
- address: 10.53.166.151
- address: 10.53.166.152
- address: 10.53.166.103
- address: 10.53.166.112
- address: 10.53.166.109
- address: 10.53.166.42
- address: 10.53.166.40
- address: 10.53.166.138
- address: 10.53.166.140
- address: 10.53.166.142
- address: 10.53.166.144
- address: 10.53.166.146
- address: 10.53.166.136
- address: 10.53.166.134
- address: 10.134.224.19
- address: 10.134.224.17
- address: 10.134.224.16
- address: 10.134.224.5
- address: 10.134.224.14
- address: 10.134.224.15
- address: 10.134.224.11
- address: 10.134.224.12
- address: 10.134.224.18
- address: 10.134.224.20
- address: 10.134.224.7
- address: 10.134.224.6
- address: 10.134.224.4
- address: 10.134.224.21
- address: 10.134.224.22
- address: 10.134.224.23
- address: 10.133.12.1
- address: 10.140.160.8
- address: 10.140.160.7
- address: 10.140.160.6
- address: 10.140.160.24
- address: 10.140.160.22
- address: 10.140.160.23
- address: 10.140.160.21
- address: 10.140.160.17
- address: 10.140.160.9
- address: 10.140.160.11
- address: 10.140.160.12
- address: 10.140.160.13
- address: 10.140.160.15
- address: 10.140.160.18
- address: 10.140.160.14
- address: 10.140.160.16
- address: 10.140.160.10
- address: 10.140.160.31
- address: 10.140.160.29
- address: 10.140.160.28
- address: 10.140.160.30
- address: 10.140.160.27
- address: 10.140.160.25
- address: 10.140.160.26
- address: 10.140.160.19
- address: 10.140.160.20
- address: 10.140.160.33
- address: 10.140.160.34
- address: 10.51.0.7
- address: 10.51.0.8
- address: 10.51.0.6
- address: 10.51.0.9
- address: 10.51.0.1
- address: 10.51.0.10
- address: 10.51.0.21
- address: 10.51.0.20
- address: 10.133.36.1
- address: 10.133.36.7
- address: 10.133.36.6
- address: 10.139.4.1
- address: 10.139.4.6
- address: 10.139.4.8
- address: 10.139.4.9
- address: 10.139.4.7
- address: 10.133.32.1
- address: 10.133.32.7
- address: 10.133.32.6
- address: 10.136.100.1
- address: 10.141.32.1
- address: 10.141.32.6
- address: 10.141.32.7
- address: 10.141.32.8
- address: 10.141.36.1
- address: 10.139.0.1
- address: 10.139.0.6
- address: 10.139.0.7
- address: 10.139.0.8
- address: 10.51.12.5
- address: 10.51.12.1
- address: 10.51.12.6
- address: 10.51.12.7
- address: 10.141.40.1
- address: 10.141.40.3
- address: 10.141.40.4
- address: 10.141.40.6
- address: 10.51.160.6
- address: 10.51.160.1
- address: 10.51.160.7
- address: 10.51.160.8
- address: 10.51.160.9
- address: 10.51.164.1
- address: 10.143.128.3
- address: 10.143.128.5
- address: 10.143.128.7
- address: 10.143.128.6
- address: 10.143.128.4
- address: 10.143.128.8
- address: 10.143.128.1
- address: 10.51.4.1
- address: 10.48.127.5
- address: 10.51.16.1
- address: 10.48.156.44
- address: 10.53.20.1
- address: 10.112.64.10
- address: 10.112.64.15
- address: 10.112.64.7
- address: 10.112.118.4
- address: 10.112.64.6
- address: 10.112.64.8
- address: 10.112.64.1
- address: 10.112.64.12
- address: 10.112.65.12
- address: 10.112.65.14
- address: 10.112.66.124
- address: 10.112.66.14
- address: 10.112.118.7
- address: 10.112.118.5
- address: 10.112.240.1
- address: 10.112.156.5
- address: 10.112.76.8
- address: 10.112.76.5
- address: 10.112.76.6
- address: 10.112.76.7
- address: 10.112.148.1
- address: 10.112.72.9
- address: 10.112.72.6
- address: 10.112.72.7
- address: 10.112.72.8
- address: 10.112.72.1
- address: 10.112.68.9
- address: 10.112.68.8
- address: 10.112.68.7
- address: 10.112.68.1
- address: 10.112.68.6
- address: 10.112.68.10
- address: 10.112.68.11
- address: 10.112.68.12
- address: 10.112.68.5
- address: 10.112.152.1
- address: 10.112.108.1
- address: 10.113.0.1
- address: 10.113.0.6
- address: 10.113.0.7
- address: 10.112.84.10
- address: 10.112.84.8
- address: 10.112.84.7
- address: 10.112.84.9
- address: 10.112.84.1
- address: 10.112.84.6
- address: 10.112.96.10
- address: 10.112.96.9
- address: 10.112.96.6
- address: 10.112.96.11
- address: 10.112.96.8
- address: 10.112.96.7
- address: 10.112.96.1
- address: 10.112.96.12
- address: 10.112.81.7
- address: 10.112.80.9
- address: 10.112.81.5
- address: 10.112.82.14
- address: 10.112.81.6
- address: 10.112.80.1
- address: 10.112.82.15
- address: 10.112.82.10
- address: 10.112.82.11
- address: 10.112.82.12
- address: 10.112.82.13
- address: 10.112.82.6
- address: 10.112.82.7
- address: 10.112.82.8
- address: 10.112.82.9
- address: 10.112.80.20
- address: 10.112.80.7
- address: 10.112.80.11
- address: 10.112.92.7
- address: 10.112.92.11
- address: 10.112.92.9
- address: 10.112.92.10
- address: 10.112.92.8
- address: 10.112.92.6
- address: 10.112.92.1
- address: 10.112.132.1
- address: 10.112.112.5
- address: 10.112.112.1
- address: 10.112.196.1
- address: 10.112.184.1
- address: 10.112.208.1
- address: 10.112.248.1
- address: 10.112.216.1
- address: 10.112.220.1
- address: 10.112.164.5
- address: 10.112.192.1
- address: 10.112.236.1
- address: 10.112.204.1
- address: 10.112.224.1
- address: 10.112.168.1
- address: 10.112.160.1
- address: 10.112.172.1
- address: 10.112.228.1
- address: 10.112.212.1
- address: 10.112.176.1
- address: 10.112.140.1
- address: 10.112.128.1
- address: 10.112.100.7
- address: 10.112.100.6
- address: 10.112.100.1
- address: 10.130.68.4
- address: 10.130.68.1
- address: 10.137.164.5
- address: 10.137.164.1
- address: 10.137.164.12
- address: 10.130.64.1
- address: 10.130.64.5
- address: 10.130.64.13
- address: 10.137.160.1
- address: 10.137.160.6
- address: 10.139.232.1
- address: 10.139.228.1
- address: 10.139.228.5
- address: 10.139.224.1
- address: 10.139.224.130
- address: 10.48.196.1
- address: 10.50.224.1
- address: 10.20.224.6
- address: 10.50.224.6
- address: 10.51.64.2
- address: 10.51.64.4
- address: 10.51.64.6
- address: 10.51.64.5
- address: 10.51.64.3
- address: 10.51.64.1
- address: 10.51.64.8
- address: 10.51.64.9
- address: 10.51.68.1
- address: 10.51.68.6
- address: 10.130.104.1
- address: 10.131.97.191
- address: 10.131.97.192
- address: 10.131.97.181
- address: 10.131.97.180
- address: 10.131.120.5
poller:
usernameSecrets:
- XXXXXX
# - sc4snmp-hlab-sha-des
inventory: |
address,port,version,community,secret,security_engine,walk_interval,profiles,smart_profiles,delete
ciusss_group_commutateur,,3,,XXXXX,,1800,,,
sim:
# sim must be enabled if you want to use signalFx
enabled: false
# signalfxToken: BCwaJ_Ands4Xh7Nrg
# signalfxRealm: us0
mongodb:
pdb:
create: true
persistence:
storageClass: "microk8s-hostpath"
volumePermissions:
enabled: true
Ok, so the problem is that you have no data in Splunk at all? Both myindex_netops and myindex_netmetrics?
Did it work with smaller number of devices or you've never tried it?
Also, I know you mentioned that logs look fine, but do you have any logs: An error of SNMP isWalk in em_logs?
The only thing that is coming to my mind without knowing any context is that the devices are very big and SC4SNMP cannot walk it before it reaches the timeout. You can configure walk profile to limit the number of data -> https://splunk.github.io/splunk-connect-for-snmp/main/configuration/configuring-profiles/#walk-profile. In such a case you would see many Task splunk_connect_for_snmp.splunk.tasks.walk in logs, but nothing more.
This could be also a problem with snmpv3 secret, but I think If the secret is badly configured, you would see some Error in logs.
Last thing - when you run a helm upgrade command and then microk8s kubectl logs -f job/snmp-splunk-connect-for-snmp-inventory -n sc4snmp, what does it say? You should see many New record or Unchanged record.
First of all thanks for your help.
I just tried with a smaller number of device (7). I got data for 5/7 devices in my personal index, so it left two devices who didn't send data. All those devices are in snmpv3, so I can say that my secret configuration is correct.
I see nothing with An error of SNMP isWalk in the em_logs index ...
When you say "configure walk profile to limit the number of data" you mean, increase the walk interval, be more specific in the varBinds section or configure a small_walk ?
Yes, I can see a Unchanged record for my 7 devices in the inventory logs.
I meant be more specific in the varBinds section or configure a small_walk, if there's no An error of SNMP isWalk that means walk have a hard time to finish itself and that might mean the scope of walk is too big.
Overall, it is good idea to limit walk scope as you usually don't need to poll all the data. Try to create such a profile and add it to inventory, and let me know if anything has changed.
I think it is a part of the the problem, when I only use those two varBinds it works for my 300 appliances :
['SNMPv2-MIB', 'sysName'] ['SNMPv2-MIB', 'sysUpTime',0]
But data returned are not enough for my client so after analysed the MIBS, I would like to pull those varBinds :
['SNMPv2-MIB', 'sysName'] ['SNMPv2-MIB', 'sysUpTime',0] ['SNMPv2-MIB', 'sysLocation'] ['SNMPv2-MIB', 'sysDescr'] ['IF-MIB', 'ifIndex'] ['IF-MIB', 'ifDescr'] ['IF-MIB', 'ifType'] ['IF-MIB', 'ifSpeed'] ['IF-MIB', 'ifAdminStatus'] ['IF-MIB', 'ifInOctets'] ['IF-MIB', 'ifOutOctets']
And again, nothing comes to Splunk Index and nothing significant in the em_log index ...
Could you please check if you have any data in the myindex_netmetrics index, provided that you used varBinds which you mentioned? Some of those varBinds are sent to the Metric index https://splunk.github.io/splunk-connect-for-snmp/main/configuration/snmp-data-format/