[MM-less] Clean up volume config when inherit podSpec from Overlord

[Sensor]

Just raise an issue here for discussion :

Shall we just clean up the volume setting when spawn a new K8s Peon(job) whose podSpec inherit from Overlord, as we know Overlord and the new K8s Peon are playing different roles, and there should be no correlations between their volume settings, can we add some change as below code:

  protected PodTemplateSpec createTemplateFromSpec(
      K8sTaskId k8sTaskId,
      PodSpec podSpec,
      Map<String, String> annotations,
      Map<String, String> labels
  )
  {
    // clean up the podSpec
    podSpec.setNodeName(null);
    podSpec.setRestartPolicy("Never");
    podSpec.setHostname(k8sTaskId.getK8sJobName());
    podSpec.setTerminationGracePeriodSeconds(taskRunnerConfig.getGraceTerminationPeriodSeconds());
    ++ podSpec.setVolumes(new ArrayList<>()); // clear up volume setting

    PodTemplateSpec podTemplate = new PodTemplateSpec();
    ObjectMeta objectMeta = new ObjectMeta();
    objectMeta.setAnnotations(annotations);
    objectMeta.setLabels(labels);
    podTemplate.setMetadata(objectMeta);
    podTemplate.setSpec(podSpec);
    return podTemplate;
  }

Just let me know if there are indeed some scenarios that this kind of Volume inheritance is necessary

[Sensor]

Hi, @georgew5656 do you think this is reasonable or not? 😄

[George Shiqi Wu]

hmm, i can think of some scenarios where you would want to keep the same volumes, for example in the druid-operator (https://github.com/datainfrahq/druid-operator) a lot of druid configs and certs and things like this are mounted as volumes and we would want some of those things on the peon. at least if we removed volumes we would want to add a config to specify them again.

in general I think that if you need to do more customizations than the overlord podSpec inheriting adapter allows I would prefer using the PodTemplateTaskAdapter to just include the whole custom pod template instead to avoid adding a ton of configs for each thing that can be overwritten in a pod template

[Sensor]

@georgew5656 Thanks for the clarification, previously we met an issue, which is, our K8s cluster will automatically add some volume mount when launching pod, as the Peon podSpec is inherited from Overlord, so when launching the Peon pod, volume mount will be added again and cause duplicate config issue, and Peon fail to start.

We also consider using PodTemplateTaskAdapter, but whose config entry seems not the same as K8sTaskAdapter, which means we need to maintain two different yaml setting file

Now we had found some workaround which is, adding a specific label(which can be config via druid.indexer.runner.labels) to the Peon podSpec before it launch, to tell the k8s cluster not appending volume config to this pod.
It should be working as aspected.

[George Shiqi Wu]

that seems reasonable, if we want to add this feature i would put it behind a new config (clearVolumeMountForPeons) or something

[Sensor]

Thanks, will close this ticket, as the work around is working fine