newest update broke my installation
LeitLei opened this issue · comments
Describe the issue
Open Webinterface isn't possible. Docker logstash is in a startup loop (Up for a few seconds, than down. And so on.)
To Reproduce
cd /opt/lme/Chapter\ 3\ Files/
sudo ./deploy.sh update
I was updating from 0.4
Linux Server:
> NAME="Ubuntu"
> VERSION="20.04.3 LTS (Focal Fossa)"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu 20.04.3 LTS"
> VERSION_ID="20.04"
> HOME_URL="https://www.ubuntu.com/"
> SUPPORT_URL="https://help.ubuntu.com/"
> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> VERSION_CODENAME=focal
> UBUNTU_CODENAME=focal
Logstash config:
> ID: XXXX
> Name: logstash.conf
> Created at: 2021-12-16 08:31:49.163791196 +0000 utc
> Updated at: 2021-12-16 08:31:49.163791196 +0000 utc
> Data:
> ##########################
> # LME Logstash Config #
> # Version 0.4 - 24/03/21 #
> ##########################
>
> input {
> beats {
> port => 5044
> ssl => true
> ssl_certificate => "/run/secrets/logstash.crt"
> ssl_key => "/run/secrets/logstash.key"
> ssl_certificate_authorities => ["/run/secrets/ca.crt"]
> ssl_verify_mode => "force_peer"
> tags => "beats"
> }
> syslog {
> port => 12514
> tags => "syslog"
> }
> }
>
> filter {
> clone {
> clones => ['cloned_logs']
> add_tag => ["cloned_logs"]
> }
> # If a windows update event that contains an updateTitle field (these sometimes contain KB numbers) then extract the KB
> if [updateTitle][winlog][event_data] {
> grok {
> match => {
> "[winlog][event_data][updateTitle]" => "(?<KB_number>KB[0-9]{5,})"
> }
> }
> }
> }
>
> output {
> if "cloned_logs" not in [tags] {
> if "beats" in [tags] {
> elasticsearch {
> hosts => "https://elasticsearch:9200"
> index => "winlogbeat-%{+dd.MM.YYYY}"
> user => logstash_writer
> password => "XXX"
> ssl => true
> cacert => '/run/secrets/ca.crt'
> pipeline => "winlogbeat"
> }
> }
> if "syslog" in [tags] {
> elasticsearch {
> hosts => "https://elasticsearch:9200"
> index => "winlogbeat-%{+dd.MM.YYYY}"
> user => logstash_writer
> password => "XXX"
> ssl => true
> cacert => '/run/secrets/ca.crt'
> pipeline => "syslog"
> }
> }
> }
> }
Hi @LeitLei, I'm sorry to hear that that you're having issues. Would you be able to confirm if it's only Logstash that's failing to start or if Elasticsearch or Kibana are having issues as well by posting the output of the command:
sudo docker stack ps lme
If it is just logstash that is encountering an error, would you also be able to post the output of the following command, after redacting it for anything sensitive, so we can see what may be going on?
sudo docker service logs lme_logstash --tail 200
I'm also curious based on the discussion in issue #114 if there have been any changes made to the default configuration of Logstash for your instance that we should be aware of, which we may not have been able to test in our environment before deploying the update? It would be useful to know so we could go back and take a look if so.
Hi @adam-ncc,
Thanks for your help.
First off all: I've restored a backup from monday where all is "original". But the error persists.
I don't use any personal configuration. The only thing is, that I'm using my own certifiactes. Installed like your manual.
As i opened this thread, only Logstash was affected.
Now I've restored again my backup from monday and restart the update with sudo ./deploy.sh update.
And now kibana is effected. And only kibana. WTF??
Output from sudo docker stack ps lme:
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
XXX lme_elasticsearch.1 docker.elastic.co/elasticsearch/elasticsearch:7.16.1 Y Running Running 5 minutes ago
XXX lme_kibana.1 docker.elastic.co/kibana/kibana:7.16.1 Y Running Running 1 second ago
XXX \_ lme_kibana.1 docker.elastic.co/kibana/kibana:7.16.1 Y Shutdown Failed 7 seconds ago "task: non-zero exit (1)"
XXX \_ lme_kibana.1 docker.elastic.co/kibana/kibana:7.16.1 Y Shutdown Failed 17 seconds ago "task: non-zero exit (1)"
XXX \_ lme_kibana.1 docker.elastic.co/kibana/kibana:7.16.1 Y Shutdown Failed 27 seconds ago "task: non-zero exit (1)"
XXX \_ lme_kibana.1 docker.elastic.co/kibana/kibana:7.16.1 Y Shutdown Failed 37 seconds ago "task: non-zero exit (1)"
XXX lme_logstash.1 docker.elastic.co/logstash/logstash:7.16.1 Y Running Running 5 minutes ago
Output from sudo docker service logs lme_kibana --tail 200:
lme_kibana.1.XXX@Y | {"type":"log","@timestamp":"2021-12-16T11:14:39+00:00","tags":["fatal","root"],"pid":8,"message":"ValidationError: [config validation of [server].publicBaseUrl]: expected URI with scheme [http|https].\n at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:95:13)\n at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:246:19)\n at MapSubscriber.project (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:254:169)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28) {\n cause: SchemaTypeError: expected URI with scheme [http|https].\n at URIType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:137:14)\n at Object.error (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:79:47)\n at Object.internals.finalize (/usr/share/kibana/node_modules/joi/lib/validator.js:472:36)\n at Object.internals.rules (/usr/share/kibana/node_modules/joi/lib/validator.js:406:34)\n at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:350:22)\n at internals.Base.$_validate (/usr/share/kibana/node_modules/joi/lib/base.js:773:26)\n at Object.validate (/usr/share/kibana/node_modules/joi/lib/types/keys.js:108:45)\n at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:325:26)\n at Object.internals.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:142:28)\n at Object.exports.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:27:30) {\n cause: undefined,\n path: [ 'publicBaseUrl' ]\n }\n}"}
lme_kibana.1.XXX@Y | {"type":"log","@timestamp":"2021-12-16T11:14:49+00:00","tags":["fatal","root"],"pid":8,"message":"ValidationError: [config validation of [server].publicBaseUrl]: expected URI with scheme [http|https].\n at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:95:13)\n at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:246:19)\n at MapSubscriber.project (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:254:169)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28) {\n cause: SchemaTypeError: expected URI with scheme [http|https].\n at URIType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:137:14)\n at Object.error (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:79:47)\n at Object.internals.finalize (/usr/share/kibana/node_modules/joi/lib/validator.js:472:36)\n at Object.internals.rules (/usr/share/kibana/node_modules/joi/lib/validator.js:406:34)\n at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:350:22)\n at internals.Base.$_validate (/usr/share/kibana/node_modules/joi/lib/base.js:773:26)\n at Object.validate (/usr/share/kibana/node_modules/joi/lib/types/keys.js:108:45)\n at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:325:26)\n at Object.internals.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:142:28)\n at Object.exports.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:27:30) {\n cause: undefined,\n path: [ 'publicBaseUrl' ]\n }\n}"}
lme_kibana.1.XXX@Y | {"type":"log","@timestamp":"2021-12-16T11:14:59+00:00","tags":["fatal","root"],"pid":9,"message":"ValidationError: [config validation of [server].publicBaseUrl]: expected URI with scheme [http|https].\n at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:95:13)\n at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:246:19)\n at MapSubscriber.project (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:254:169)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28) {\n cause: SchemaTypeError: expected URI with scheme [http|https].\n at URIType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:137:14)\n at Object.error (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:79:47)\n at Object.internals.finalize (/usr/share/kibana/node_modules/joi/lib/validator.js:472:36)\n at Object.internals.rules (/usr/share/kibana/node_modules/joi/lib/validator.js:406:34)\n at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:350:22)\n at internals.Base.$_validate (/usr/share/kibana/node_modules/joi/lib/base.js:773:26)\n at Object.validate (/usr/share/kibana/node_modules/joi/lib/types/keys.js:108:45)\n at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:325:26)\n at Object.internals.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:142:28)\n at Object.exports.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:27:30) {\n cause: undefined,\n path: [ 'publicBaseUrl' ]\n }\n}"}
lme_kibana.1.XXX@Y | {"type":"log","@timestamp":"2021-12-16T11:15:10+00:00","tags":["fatal","root"],"pid":9,"message":"ValidationError: [config validation of [server].publicBaseUrl]: expected URI with scheme [http|https].\n at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:95:13)\n at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:246:19)\n at MapSubscriber.project (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:254:169)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28) {\n cause: SchemaTypeError: expected URI with scheme [http|https].\n at URIType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:137:14)\n at Object.error (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:79:47)\n at Object.internals.finalize (/usr/share/kibana/node_modules/joi/lib/validator.js:472:36)\n at Object.internals.rules (/usr/share/kibana/node_modules/joi/lib/validator.js:406:34)\n at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:350:22)\n at internals.Base.$_validate (/usr/share/kibana/node_modules/joi/lib/base.js:773:26)\n at Object.validate (/usr/share/kibana/node_modules/joi/lib/types/keys.js:108:45)\n at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:325:26)\n at Object.internals.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:142:28)\n at Object.exports.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:27:30) {\n cause: undefined,\n path: [ 'publicBaseUrl' ]\n }\n}"}
Ah that is odd, nothing had changed in the logstash settings beyond the version bump so I was surprised that it failed the update, but I guess it seems to be working now.
Less good news with regards to Kibana, it seems as if the PublicBaseURL value is missing or invalid for some reason, which is something new for Kibana 7.14 that we've just added in support for in the latest update. Currently this should be taken from the existing value within the docker-compose-stack-live.yml file if it's already set, or if you're doing a fresh update and this isn't already set it should be taken from the lme/.conf file, but it seems to be incorrectly set for you now despite working before.
Would you mind taking a look at these files and letting me know what the following settings are? You will likely want to redact the hostname itself from both values, but just seeing what the data looks like and whether it's a valid URI should be enough to get started:
/opt/lme/lme.conf
hostname=[xxxx]
/opt/lme/Chapter\ 3\ Files/docker-compose-stack-live.yml
SERVER_PUBLICBASEURL: [yyyyyy]
Thanks!
After Update:
hostname=XX.XXXXXXX.XXX
SERVER_PUBLICBASEURL: insertpublicurlhere
Pre Update:
hostname=XX.XXXXXXX.XXX
XX.XXXXXXX.XXX = my correct FQDN
Ah what seems to have happened is you've ended up with the placeholder value in your live docker compose file for some reason, so when you do an update it keeps the placeholder value rather than using the value from the LME config file.
You should be able to fix this by manually opening the docker-compose-stack-live.yml file within Chapter 3 and then editing the SERVER_PUBLICBASEURL value so that it reads https://XX.XXXXXXX.XXX where XX.XXXXXXX.XXX = your FQDN. Once this value is correct, running the deploy script with the update command again should restart the docker containers with the correct value set and hopefully resolve the issue for you:
sudo [nano/vim] /opt/lme/Chapter\ 3\ Files/docker-compose-stack-live.yml
Change SERVER_PUBLICBASEURL so that it looks like the example below, but using your FQDN instead:
SERVER_PUBLICBASEURL: https://elastic-lme.contoso.local
sudo /opt/lme/Chapter\ 3\ Files/deploy.sh update
I'm not sure what might have caused this, it may be related to how you restored the backup, but we'll keep an eye on it and do some more testing to see if anyone else encounters this problem.
Let me know you get on with the above fix, hopefully this resolves things for you.
Thanks for your help @adam-ncc.
Now it works.
I made a complete VM restore with Veeam Backup. Normally Veeam make a good job.
Thanks guys.
I had the exact same issue trying to update a new 0.4 build.
The solution worked a treat.
Update to this, we've been able to re-produce this issue. It's being caused up the update script replacing the docker compose file with the latest version, but the script itself being cached in memory and using the older version during an update. This means that the placeholder value of the SERVER_PUBLICBASEURL parameter will be populated into the live docker compose file and won't be replaced across future updates.
To resolve this we've pushed a hotfix that will reset the relevant parameter to the value from the lme.conf file if it is currently set to the placeholder, but if you run the update script without first ensuring you are up to date with git you will still see this behaviour until the next time the update is run.
This means that if an automatic update has caused you issues with Kibana it should resolve itself in a day or so once the update triggers itself twice, or if you wish to manually update to get the latest version or solve the issue it's best to make sure you're up to date with git before running the update:
sudo git -C /opt/lme/ pull
sudo /opt/lme/Chapter\ 3\ Files/deploy.sh update
Thanks for reporting this and bringing it to our attention.