HTTP Error 400: Bad Request on https://127.0.0.1:8089/services/data/inputs/http/http

[Stephen Herd]

We have quite a few docker containers deployed using splunk 8.2.8 spunk image, and they all work, but there's one specific app's splunk sidecar that seems to be breaking, no matter how many times we deploy it, and in any environment.

TASK [splunk_universal_forwarder : Setup global HEC] ***************************
fatal: [localhost]: FAILED! => {
    "cache_control": "no-store, no-cache, must-revalidate, max-age=0",
    "changed": false,
    "connection": "Close",
    "content_length": "168",
    "content_type": "text/xml; charset=UTF-8",
    "date": "Thu, 25 May 2023 17:33:25 GMT",
    "elapsed": 0,
    "expires": "Thu, 26 Oct 1978 00:00:00 GMT",
    "redirected": false,
    "server": "Splunkd",
    "status": 400,
    "url": "https://127.0.0.1:8089/services/data/inputs/http/http",
    "vary": "Cookie, Authorization",
    "x_content_type_options": "nosniff",
    "x_frame_options": "SAMEORIGIN"
}

MSG:

Status code was 400 and not [200]: HTTP Error 400: Bad Request

The only env variables we pass are:

  - env:
    - name: SPLUNK_DEPLOYMENT_SERVER
      value: deploymentserver.splunk.service.consul
    - name: SPLUNK_START_ARGS
      value: --accept-license --answer-yes
    - name: SPLUNK_USER
      value: splunk

And the env var's for the ansible user are (I did sanitize some of the info)

LANG=C.utf8
HOSTNAME=testhost
ANSIBLE_USER=ansible
SPLUNK_DEPLOYMENT_SERVER=deploymentserver.splunk.service.consul
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=xxx.xxx.xxx.xxx
container=oci
SPLUNK_HOME=/opt/splunkforwarder
SCLOUD_URL=https://github.com/splunk/splunk-cloud-sdk-go/releases/download/v1.11.1/scloud_v7.1.0_linux_amd64.tar.gz
CONTAINER_ARTIFACT_DIR=/opt/container_artifact
KUBERNETES_PORT=tcp://xxx.xxx.xxx.xxx:443
PWD=/opt/splunkforwarder
HOME=/home/ansible
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_PORT=443
SPLUNK_DEFAULTS_URL=
SPLUNK_GROUP=splunk
SPLUNK_OPT=/opt
KUBERNETES_PORT_443_TCP=tcp://xxx.xxx.xxx.xxx:443
SPLUNK_ANSIBLE_HOME=/opt/ansible
TERM=xterm
SPLUNK_ROLE=splunk_universal_forwarder
SPLUNK_PASSWORD=password
PYTHON_GPG_KEY_ID=gpg
PYTHON_VERSION=3.7.10
ANSIBLE_GROUP=ansible
SPLUNK_START_ARGS=--accept-license --answer-yes
SHLVL=1
KUBERNETES_SERVICE_PORT=443
SPLUNK_USER=splunk
PATH=/home/ansible/.local/bin:/home/ansible/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_SERVICE_HOST=xxx.xxx.xxx.xxx
_=/usr/bin/env

There's no other configuration set.

I updated this specific host to 8.2.10's image, but got the same problem.

Any thoughts would be appreciated.