Reappearing outputs.conf in /opt/splunk/etc/system/local/

[shoof1072]

Hello,

While troubleshooting a license usage issue with splunk support they advised that I file an issue here.

Problem:

Empty/blank rows in license usage reports.

Deployment:

Single Splunk License master consuming license usage data from many Standalone Splunk hosts.
All Splunk instances are based on the 8.0.4-debian Splunk docker image.
splunk.role is set using an environment variable per the documentation in the splunk-docker repo instead of setting this in default.yaml

Observed behavior:

Noticed that the content of outputs.conf in system/local was the following:
$ sudo cat /opt/splunk/etc/system/local/outputs.conf
[indexAndForward]
index = false

This seems like it would cause the license master to not index license usage data locally.

I removed the outputs.conf under system/local and restarted splunk.
At UTC midnight license usage was pulled, and indexed.

A few hours later we kicked off a build and the Splunk License Master was redeployed, the outputs.conf reappeared with the same content as observed above, and all days since the reappearance of the outputs.conf in system/local we have missing rows of licensing data.

Hypothesis:

Throwing this out as a strawman.
We're observing a reappearance of the outputs.conf under system/local.
It seems to have two causes:
This task is not protected by a when: clause :
https://github.com/splunk/splunk-ansible/blob/master/roles/splunk_common/tasks/enable_forwarding.yml#L57-L67

This section of the upgrade task does not include a check for "not splunk_license_master":
https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/main.yml#L114-L120

The result of those two would cause a two line outputs.conf to be created which prevents indexing on any server with splunk.role set to splunk_license_master.

Let me know of the reappearance of the outputs.conf follows from that.

Thanks,

default.yaml from our Splunk LM

---
ansible_post_tasks: null
ansible_pre_tasks: null
config:
  baked: default.yml
  defaults_dir: /tmp/defaults
  host:
    headers: null
    url: null
    verify: true
  max_delay: 60
  max_retries: 3
  max_timeout: 1200
hide_password: false
retry_num: 50
splunk:
  allow_upgrade: true
  app_paths:
    default: /opt/splunk/etc/apps
    deployment: /opt/splunk/etc/deployment-apps
    httpinput: /opt/splunk/etc/apps/splunk_httpinput
    idxc: /opt/splunk/etc/master-apps
    shc: /opt/splunk/etc/shcluster/apps
  enable_service: false
  exec: /opt/splunk/bin/splunk
  group: splunk
  hec_disabled: 1
  hec_enableSSL: 1
  hec_port: 8088
  hec_token: OUR-HEC-TOKEN
  home: /opt/splunk
  http_enableSSL: 0
  http_enableSSL_cert: null
  http_enableSSL_privKey: null
  http_enableSSL_privKey_password: null
  http_port: 8000
  opt: /opt
  pid: /opt/splunk/var/run/splunk/splunkd.pid
  s2s_enable: true
  s2s_port: 9997
  smartstore: null
  svc_port: 8089
  user: splunk
  conf:
    server:
      content:
        httpServer:
          crossOriginSharingPolicy: "*"
    web:
      content:
        settings:
          x_frame_options_sameorigin: False
          enableSplunkWebSSL: true
    authorize:
      content:
        tokens_auth:
          disabled: false
    inputs:
      directory: /opt/splunk/etc/apps/splunk_httpinput/local
      content:
        http:
          disabled: 0
    indexes:
      content:
        _internal:
          frozenTimePeriodInSecs: '31556952'
          maxDataSize: '10000'
splunk_home_ownership_enforcement: true

[Nelson Wang]

I believe the design/intent was that license master should not be indexing locally, but rather forwarding its own logs to some search peer. We can make the license master not forward, although I don't know if that's the desired behavior for other topologies.

What roles(s) are the license slaves that are connecting to this license master? Are they free-standing standalones, or is it a cluster of sorts?

[shoof1072]

We set the role for each Splunk LS to splunk_standalone. We ETL various types of application/product data from customer environments to a Standalone Splunk instance per customer. License usage for all Splunk deployments is, or is intended to then be aggregated on the Splunk LM.

[shoof1072]

We've implemented a workaround, but it would be a good idea to document the Splunk topology that these ansible scripts are targeting.

[Nelson Wang]

Got it - I discussed this with a few folks and our proposed solution going forward is to allow indexAndForward=true for the license master. This should support the ability for the license master to index locally, as well as forward to a cluster if desired.

[Nelson Wang]

Will be out in the next Splunk image (8.1.0)!