Reappearing outputs.conf in /opt/splunk/etc/system/local/
shoof1072 opened this issue · comments
Hello,
While troubleshooting a license usage issue with splunk support they advised that I file an issue here.
Problem:
Empty/blank rows in license usage reports.
Deployment:
Single Splunk License master consuming license usage data from many Standalone Splunk hosts.
All Splunk instances are based on the 8.0.4-debian Splunk docker image.
splunk.role is set using an environment variable per the documentation in the splunk-docker repo instead of setting this in default.yaml
Observed behavior:
Noticed that the content of outputs.conf in system/local was the following:
$ sudo cat /opt/splunk/etc/system/local/outputs.conf
[indexAndForward]
index = false
This seems like it would cause the license master to not index license usage data locally.
I removed the outputs.conf under system/local and restarted splunk.
At UTC midnight license usage was pulled, and indexed.
A few hours later we kicked off a build and the Splunk License Master was redeployed, the outputs.conf reappeared with the same content as observed above, and all days since the reappearance of the outputs.conf in system/local we have missing rows of licensing data.
Hypothesis:
Throwing this out as a strawman.
We're observing a reappearance of the outputs.conf under system/local.
It seems to have two causes:
This task is not protected by a when:
clause :
https://github.com/splunk/splunk-ansible/blob/master/roles/splunk_common/tasks/enable_forwarding.yml#L57-L67
This section of the upgrade task does not include a check for "not splunk_license_master":
https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/main.yml#L114-L120
The result of those two would cause a two line outputs.conf to be created which prevents indexing on any server with splunk.role
set to splunk_license_master
.
Let me know of the reappearance of the outputs.conf follows from that.
Thanks,
default.yaml from our Splunk LM
---
ansible_post_tasks: null
ansible_pre_tasks: null
config:
baked: default.yml
defaults_dir: /tmp/defaults
host:
headers: null
url: null
verify: true
max_delay: 60
max_retries: 3
max_timeout: 1200
hide_password: false
retry_num: 50
splunk:
allow_upgrade: true
app_paths:
default: /opt/splunk/etc/apps
deployment: /opt/splunk/etc/deployment-apps
httpinput: /opt/splunk/etc/apps/splunk_httpinput
idxc: /opt/splunk/etc/master-apps
shc: /opt/splunk/etc/shcluster/apps
enable_service: false
exec: /opt/splunk/bin/splunk
group: splunk
hec_disabled: 1
hec_enableSSL: 1
hec_port: 8088
hec_token: OUR-HEC-TOKEN
home: /opt/splunk
http_enableSSL: 0
http_enableSSL_cert: null
http_enableSSL_privKey: null
http_enableSSL_privKey_password: null
http_port: 8000
opt: /opt
pid: /opt/splunk/var/run/splunk/splunkd.pid
s2s_enable: true
s2s_port: 9997
smartstore: null
svc_port: 8089
user: splunk
conf:
server:
content:
httpServer:
crossOriginSharingPolicy: "*"
web:
content:
settings:
x_frame_options_sameorigin: False
enableSplunkWebSSL: true
authorize:
content:
tokens_auth:
disabled: false
inputs:
directory: /opt/splunk/etc/apps/splunk_httpinput/local
content:
http:
disabled: 0
indexes:
content:
_internal:
frozenTimePeriodInSecs: '31556952'
maxDataSize: '10000'
splunk_home_ownership_enforcement: true
I believe the design/intent was that license master should not be indexing locally, but rather forwarding its own logs to some search peer. We can make the license master not forward, although I don't know if that's the desired behavior for other topologies.
What roles(s) are the license slaves that are connecting to this license master? Are they free-standing standalones, or is it a cluster of sorts?
We set the role for each Splunk LS to splunk_standalone
. We ETL various types of application/product data from customer environments to a Standalone Splunk instance per customer. License usage for all Splunk deployments is, or is intended to then be aggregated on the Splunk LM.
We've implemented a workaround, but it would be a good idea to document the Splunk topology that these ansible scripts are targeting.
Got it - I discussed this with a few folks and our proposed solution going forward is to allow indexAndForward=true
for the license master. This should support the ability for the license master to index locally, as well as forward to a cluster if desired.
Will be out in the next Splunk image (8.1.0)!