Splunk Gitops

Template to store the complete configuration (apps and system configuration) of a Splunk installation running on Kubernetes in Git.

Having the complete configuration in a single repository is an important prerequisite to enjoy the benefits of GitOps.

To ensure that this template is running with future versions of Splunk, it is solely extending Splunk Docker images according to the docker-splunk configuration. Furthermore for deploying the images it is using the splunk-operator for Kubernetes as recommended by Splunk.

The repository is the result of my discussions about how to use Gitops with Splunk.

Usage

Clone the repository, then you can configure your Splunk installation by editing these files:

The system configuration is stored in its own app as considered best practice for Splunk. This example is using my splunk-root-config app.
External apps can be retrieved at build time from Github Releases or Splunkbase - you can define the dependencies to download directly in the Dockerfile.
Local apps are stored in the apps folder. Copy apps that you want to include in your installation here - without their configuration (will be added in the next step).
The configuration of local and external apps can be overwritten with app folders in the config/apps directory. You can configure local apps and external apps there.
The SPLUNK_APPS_URL variables in the Dockerfile must reference your local and external apps. Configure it accordingly.

Note: Doing so, we nicely separate the different concerns: apps and their configuration and the different origins of an app (local or external).

Build

According to the configuration in the previous step, the build process is generating a docker image that will be deployed using the splunk-operator in Kubernetes.

Local Build

If you're not using external apps from Splunkbase (as in the example Dockerfile), you can build the image locally with this simple command:

docker build -t splunk-gitops .

In case you want to include external apps from Splunkbase (comment out the appropriate sections in the Dockerfile), then you can do the build like this:

export SPLUNKBASE_USERNAME=<YOUR SPLUNK BASE USERNAME>
export SPLUNKBASE_PASSWORD=<YOUR SPLUNK BASE PASSWORD>

docker build --secret id=SPLUNKBASE_USERNAME --secret id=SPLUNKBASE_PASSWORD -t splunk-gitops .

Github Actions

For your CI/CD workflow, you don't want to build the images locally. I am using a Github Action workflow to build the image and upload it to Github Packages.

In case that you're adding Splunkbase apps (like in the example), then you need to set the following Github secrets with your Splunkbase credentials to retrieve the packages:

SPLUNKBASE_USERNAME
SPLUNKBASE_PASSWORD

Test

To check whether the file structure has been generated correctly you can start a bash shell in the image without running Splunk by calling:

docker run --entrypoint /bin/bash -it splunk-gitops

If everything looks fine, you can start Splunk by calling:

docker run -p 8000:8000 -e "SPLUNK_PASSWORD=<password>" \
             -e "SPLUNK_START_ARGS=--accept-license" \
             -it --name so1 splunk-gitops

Deployment

Deployment requires you to install the splunk-operator in your Kubernetes cluster.

Then you'll reference the Docker image generated by the build step in the Kubernetes YAML file. You can then deploy the image with the following command:

kubectl apply -f ./kubernetes-manifest/standalone.yaml

The example deploys a Splunk standalone installation in its own namespace called test.

Note that the configuration folder of the cluster (/opt/splunk/etc/) can be mounted as an ephemeral storage as its complete state is stored the Git repository.

Welcome to Gitops! 😄

TODOs

Add hook after ansible has finished the configuration to do after system configuration tasks, e.g. adding users

marcusschiesser / splunk-gitops