Upgrade to cluster-logging.5.4.0-138 breaks kibana deployment.

[Andrei Neagoe]

Describe the bug
Following a cluster-logging upgrade, kibana failst to start.

Environment

• 4.9.0-0.okd-2022-02-12-140851
• cluster-logging.5.4.0-138
• elasticsearch-operator.5.4.0-152

Logs

{"type":"log","@timestamp":"2022-04-21T15:22:11Z","tags":["status","plugin:kibana@6.8.1","info"],"pid":117,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2022-04-21T15:22:11Z","tags":["status","plugin:elasticsearch@6.8.1","info"],"pid":117,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2022-04-21T15:22:11Z","tags":["status","plugin:opendistro_security@6.8.1","info"],"pid":117,"state":"yellow","message":"Status changed from uninitialized to yellow - Security copy JWT params disabled","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2022-04-21T15:22:11Z","tags":["status","plugin:opendistro_security@6.8.1","info"],"pid":117,"state":"yellow","message":"Status changed from yellow to yellow - Security multitenancy registered.","prevState":"yellow","prevMsg":"Security copy JWT params disabled"}
{"type":"log","@timestamp":"2022-04-21T15:22:11Z","tags":["status","plugin:opendistro_security@6.8.1","info"],"pid":117,"state":"yellow","message":"Status changed from yellow to yellow - Routes for Security configuration GUI registered.","prevState":"yellow","prevMsg":"Security multitenancy registered."}
{"type":"log","@timestamp":"2022-04-21T15:22:11Z","tags":["status","plugin:console@6.8.1","info"],"pid":117,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2022-04-21T15:22:11Z","tags":["status","plugin:interpreter@6.8.1","info"],"pid":117,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2022-04-21T15:22:11Z","tags":["status","plugin:tile_map@6.8.1","info"],"pid":117,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2022-04-21T15:22:12Z","tags":["status","plugin:timelion@6.8.1","info"],"pid":117,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2022-04-21T15:22:12Z","tags":["status","plugin:elasticsearch@6.8.1","info"],"pid":117,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2022-04-21T15:22:12Z","tags":["status","plugin:opendistro_security@6.8.1","info"],"pid":117,"state":"yellow","message":"Status changed from yellow to yellow - Setting up index template.","prevState":"yellow","prevMsg":"Routes for Security configuration GUI registered."}
{"type":"error","@timestamp":"2022-04-21T15:22:12Z","tags":["error","migration"],"pid":117,"level":"error","error":{"message":"Authorization Exception: Authorization Exception","name":"Error","stack":"Authorization Exception :: {\"path\":\"/_opendistro/_security/tenantinfo\",\"query\":{},\"statusCode\":403,\"response\":\"\"}\n    at respond (/opt/app-root/src/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/opt/app-root/src/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/opt/app-root/src/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n    at IncomingMessage.wrapper (/opt/app-root/src/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"Authorization Exception: Authorization Exception"}
{"type":"log","@timestamp":"2022-04-21T15:22:12Z","tags":["status","plugin:opendistro_security@6.8.1","info"],"pid":117,"state":"yellow","message":"Status changed from yellow to yellow - Tenant indices migration failed","prevState":"yellow","prevMsg":"Setting up index template."}
{"type":"log","@timestamp":"2022-04-21T15:22:12Z","tags":["info","migrations"],"pid":117,"message":"Creating index .kibana_1."}
{"type":"log","@timestamp":"2022-04-21T15:22:12Z","tags":["info","migrations"],"pid":117,"message":"Pointing alias .kibana to .kibana_1."}
{"type":"log","@timestamp":"2022-04-21T15:22:13Z","tags":["info","migrations"],"pid":117,"message":"Finished in 941ms."}
{"type":"log","@timestamp":"2022-04-21T15:22:13Z","tags":["listening","info"],"pid":117,"message":"Server running at http://localhost:5601"}

Expected behavior
Kibana comes up normally

Actual behavior
Kibana fails to start due to failed tenant indices migration.

To Reproduce
Unclear, issue observed after the upgrade.

Additional context
I've tried kibana restart, did not help. I've tried removing all tenant indices, it also did not help. Operator doesn't provide any relevant logging information.

[Andrei Neagoe]

Deleting kibana from the clusterlogging/instance and adding it back fixed it. Still, it's not clear what happened in the first place.