Giters

apache / druid

Apache Druid: a high performance real-time analytics database.

Home Page:https://druid.apache.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

druid.indexer.task.restoreTasksOnRestart does not work by default for Docker based deployments on Kubernetes

aho135 opened this issue · comments

commented

druid.indexer.task.restoreTasksOnRestart does not work by default for Docker based deployments on Kubernetes

Affected Version

25.0.0 but the issue still exists in latest version

Description

Hi Druid experts. Our team runs Druid on Kubernetes and ingest data from Kafka. We have druid.indexer.task.restoreTasksOnRestart=true and expected ingestion tasks to restore and resume even when the MiddleManager is restarted. This is the current behavior:

  1. MiddleManager is shut down. Because druid.indexer.task.restoreTasksOnRestart=true, restore.json is created
  2. MiddleManager starts up, but with a different IP because we are running on Kubernetes. The task is restored and continues running.
  3. When the peon reports its status to the Overlord, the Overlord will log that the task is not in known task id's and proceeds to shutdown the task. This is because the MiddleManager IP has changed

The solution to fix this problem is to allow druid.host to use the default value of InetAddress.getLocalHost().getCanonicalHostName() and task restoration works after that. But setting druid.host to the default value requires setting DRUID_SET_HOST to 0 through an environment variable. I am wondering what the original reasoning for using IP instead of canonical host name is. And wondering if we should change the default behavior given that using IP breaks task restoration

commented

I am wondering what the original reasoning for using IP instead of canonical host name is. And wondering if we should change the default behavior given that using IP breaks task restoration

See: #9019 and #6896 to know the history.

The DRUID_SET_HOST environment variable maybe is not well documented.