[Flaky Test] Success rate of `shoot-default-serial-test-suite` for azure_multizone is 50%
ialidzhikov opened this issue · comments
/area testing
/kind flake
/priority 3
/platform azure
Which test(s)/suite(s) are flaking:
Shoot operation testing [It] [DEFAULT] [SERIAL] [SHOOT] Testing if Shoot can be hibernated successfully
Reason for failure:
The logs from a failure:
/src/test/framework/applications/guestbooktest.go:78
occurred
retry failed with context deadline exceeded, last error: stateful set "redis-master" is not healthy: not enough ready replicas (0/1)
}
},
s: "stateful set \"redis-master\" is not healthy: not enough ready replicas (0/1)",
err: {
ctxError: {},
<*retry.Error | 0xc001ed69e0>: {
Unexpected error:
STEP: Cleaning up guestbook app resources
time="2021-05-17T05:30:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:30:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:29:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:28:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:27:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:26:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:25:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:22Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:17Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:07Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:24:02Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:47Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:42Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:37Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:32Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:22Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:17Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:07Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:23:02Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:47Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:42Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:37Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:32Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:22Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:17Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:07Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:22:02Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:47Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:42Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:37Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:32Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:22Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:17Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:07Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:21:02Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:47Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:42Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:37Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:32Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:21Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:16Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:11Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:20:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:46Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:41Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:36Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:31Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:26Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:21Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:16Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:11Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:19:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:46Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:41Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:36Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:31Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:26Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:21Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:16Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:11Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:18:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:46Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:41Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:36Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:31Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:26Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:21Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:16Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:11Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:17:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:46Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:41Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:35Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:30Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:25Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:20Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:15Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:10Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:05Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:16:00Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:55Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:50Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:45Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:40Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:35Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:30Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:25Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:20Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:15Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:10Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:05Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:15:00Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:55Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:50Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:45Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:40Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:35Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:30Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:25Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:20Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:15Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:10Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:05Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:14:00Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:55Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:50Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:45Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:40Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:35Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:30Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:25Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:20Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:15Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:10Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:05Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:13:00Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:12:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:11:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:10:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:09:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:08:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:07:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:06:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:47Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:42Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:37Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:32Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:22Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:17Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:07Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:05:02Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:47Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:42Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:37Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:32Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:22Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:17Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:07Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:04:02Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:47Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:42Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:37Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:32Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:22Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:17Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:07Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:03:02Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:47Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:42Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:37Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:32Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:27Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:22Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:17Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:02:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:46Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:41Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:36Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:31Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:26Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:21Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:16Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:11Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:01:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:46Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:41Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:36Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:31Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:26Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:21Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:16Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:11Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T05:00:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:46Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:41Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:36Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:31Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:26Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:21Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:16Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:11Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:59:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:46Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:41Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:36Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:31Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:26Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:21Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:16Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:11Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:06Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:58:01Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:56Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:51Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:45Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:40Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:35Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:30Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:25Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:20Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:15Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:10Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:05Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:57:00Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:55Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:50Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:45Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:40Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:35Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:30Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:25Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:20Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:15Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:10Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:05Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:56:00Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:55Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:50Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:45Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:40Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:35Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:30Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:25Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:20Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:15Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:10Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:05Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:55:00Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:55Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:50Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:45Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:40Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:35Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:30Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:25Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:20Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:15Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:10Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:05Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:54:00Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:55Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:53:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:52:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:51:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:14Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:09Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:50:04Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:59Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:54Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:49Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:44Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:39Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:34Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:29Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:24Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:19Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:49:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:48:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:47:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:46:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:45:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:18Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:13Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:08Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:44:03Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:43:58Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:43:53Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:43:48Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:43:43Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:43:38Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:43:33Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:43:28Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:43:23Z" level=info msg="Waiting for \"redis-master\" to be ready!"
STEP: test guestbook
time="2021-05-17T04:43:22Z" level=info msg="Shoot tm9kd-bsf has been woken up successfully!"
time="2021-05-17T04:42:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (condition type ControlPlaneHealthy is not true yet, had message Deployment \"vpa-admission-controller\" is unhealthy: Deployment does not have minimum availability (MinimumReplicasUnavailable) with reason DeploymentUnhealthy)"
time="2021-05-17T04:42:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:41:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:41:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:40:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:40:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:39:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:39:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:38:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:38:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:37:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:37:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:36:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (shoot generation did not equal observed generation)"
STEP: wake up shoot
time="2021-05-17T04:36:52Z" level=info msg="Shoot tm9kd-bsf was hibernated successfully!"
time="2021-05-17T04:36:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:35:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:35:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:34:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:34:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:33:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:33:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:32:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:32:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:31:52Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (last operation type was create, reconcile or restore but state was not succeeded)"
time="2021-05-17T04:31:22Z" level=info msg="Shoot tm9kd-bsf not yet reconciled successfully (shoot generation did not equal observed generation)"
STEP: Hibernate shoot
time="2021-05-17T04:31:22Z" level=info msg="Rejoice, the guestbook app urls are available now!"
STEP: Guestbook app was deployed successfully!
time="2021-05-17T04:31:17Z" level=info msg="Waiting for deployment 'gardener-e2e-s9xb8/guestbook' to be ready!"
W0517 04:31:17.280054 11155 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0517 04:31:17.264595 11155 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
STEP: Deploy the guestbook application
time="2021-05-17T04:31:17Z" level=info msg="redis-master is now ready!!"
time="2021-05-17T04:31:12Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:31:07Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:31:02Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:30:57Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:30:52Z" level=info msg="Waiting for \"redis-master\" to be ready!"
time="2021-05-17T04:30:51Z" level=info msg="Chart downloaded to /src/test/framework/resources/charts/redis-10.2.1.tgz"
STEP: Downloading chart artifacts
STEP: Applying redis chart
STEP: Deploy guestbook
/src/test/framework/gingko_utils.go:26
[It] [DEFAULT] [SERIAL] [SHOOT] Testing if Shoot can be hibernated successfully
/src/test/framework/gingko_utils.go:51
[BeforeEach] Shoot operation testing
/src/test/framework/gingko_utils.go:26
[DEFAULT] [SERIAL] [SHOOT] Testing if Shoot can be hibernated successfully
Shoot operation testing
Needs to be checked further to find the root cause.
Anything else we need to know:
Environment:
- Gardener version (if relevant): v1.23.0-dev-b48ae95e29853f7bb590c6838b4e2aa70c6c98d2
- Extension version: v1.20.0
- Kubernetes version (use
kubectl version): v1.20.6 - Cloud provider or hardware configuration:
- Others:
Okay, I managed to reproduce the issue:
$ k -n gardener-e2e-jvn7b get po redis-master-0
NAME READY STATUS RESTARTS AGE
redis-master-0 0/1 Pending 0 20m
$ k -n gardener-e2e-jvn7b describe po redis-master-0
# [...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 13m default-scheduler no nodes available to schedule pods
Warning FailedScheduling 13m default-scheduler no nodes available to schedule pods
Warning FailedScheduling 11m default-scheduler 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
Warning FailedScheduling 11m default-scheduler 0/2 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
Warning FailedScheduling 10m default-scheduler 0/2 nodes are available: 2 node(s) had volume node affinity conflict.
Warning FailedScheduling 10m default-scheduler 0/2 nodes are available: 2 node(s) had volume node affinity conflict.
The PV has the following nodeAffinity with zone westeurope-2:
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/region
operator: In
values:
- westeurope
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- westeurope-2On the other sides the 2 Nodes have the westeurope-1 zone topology labels:
$ k get no -o json | jq '.items[].metadata.labels'
{
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "Standard_DS2_v2",
"beta.kubernetes.io/os": "linux",
"failure-domain.beta.kubernetes.io/region": "westeurope",
"failure-domain.beta.kubernetes.io/zone": "westeurope-1",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "shoot--foo--test-cpu-worker-z1-658f4-492md",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "Standard_DS2_v2",
"node.kubernetes.io/role": "node",
"topology.kubernetes.io/region": "westeurope",
"topology.kubernetes.io/zone": "westeurope-1",
"worker.garden.sapcloud.io/group": "cpu-worker",
"worker.gardener.cloud/pool": "cpu-worker",
"worker.gardener.cloud/system-components": "true"
}
{
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "Standard_DS2_v2",
"beta.kubernetes.io/os": "linux",
"failure-domain.beta.kubernetes.io/region": "westeurope",
"failure-domain.beta.kubernetes.io/zone": "westeurope-1",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "shoot--foo--test-cpu-worker-z2-799b4-dsqrf",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "Standard_DS2_v2",
"node.kubernetes.io/role": "node",
"topology.kubernetes.io/region": "westeurope",
"topology.kubernetes.io/zone": "westeurope-1",
"worker.garden.sapcloud.io/group": "cpu-worker",
"worker.gardener.cloud/pool": "cpu-worker",
"worker.gardener.cloud/system-components": "true"
}
I don't see anything wrong for now on machines, machine classes side.
There are 2 machines:
$ k -n shoot--foo--test get machine,machineclasses
NAME STATUS AGE
machine.machine.sapcloud.io/shoot--foo--test-cpu-worker-z1-658f4-492md Running 49m
machine.machine.sapcloud.io/shoot--foo--test-cpu-worker-z2-799b4-dsqrf Running 49m
NAME PROVIDER AGE
machineclass.machine.sapcloud.io/shoot--foo--test-cpu-worker-5f2ab-z1 Azure 115m
machineclass.machine.sapcloud.io/shoot--foo--test-cpu-worker-5f2ab-z2 Azure 115m
$ k -n shoot--foo--test get machineclasses -o json | jq '.items[].providerSpec.properties.zone'
1
2
I see that Machine shoot--foo--test-cpu-worker-z2-799b4-dsqrf is created in zone 1 in the azure backend.
When I scaled MachineDeployment manually shoot--foo--test-cpu-worker-z2, the newly created Machines were in the correct zone 2.
So having in mind the test failure rate, I suspect that there is something wrong somewhere.
The command to run the test against a multizone azure Shoot:
go test -timeout=0 -mod=vendor ./test/suites/shoot \
--v -ginkgo.v -ginkgo.progress -ginkgo.noColor \
--report-file=/tmp/report.json \
-kubecfg=$KUBECONFIG \
-shoot-name=$SHOOT_NAME \
-project-namespace=$PROJECT_NAMESPACE \
-ginkgo.focus="Testing if Shoot can be hibernated successfully"
@ialidzhikov has identified the issue with the above. The MCM provider azure seems to be passing around a reference object that is being shared among different creation calls. @AxiomSamarth is working on this fix, and will hotfix this ASAP.
Thank you @ialidzhikov for your investigation.
/assign @AxiomSamarth
Okay, I found the issue.
This test became a bad test after the rollout of OOT MCM provider. You can see that test results went worse after the rollout of MCM OOT (05/15) .
The issue itself:
Currently MCM provider-azure has 1 single MachinePlugin obj that is singleton.
This MachinePlugin has a field AzureProviderSpec
Currently on machine creation the MachinePlugin is decoding the MachineClass and setting this AzureProviderSpec field.
And this field is being shared by multiple Machine creations -> multiple Machines end up with the same spec on the cloudprovider when they should have different spec.