kubecfg's recursive jsonnet walk should be exposed to jsonnet somehow

[Angus Lees]

Several times I have wanted to convert kubecfg's "hierarchical" k8s resources into a simple flattened list, for use elsewhere in jsonnet (eg: to do an "add a label to all things" post-processing step).

This should be something that is easier to do than writing out a recursive jsonnet function from scratch every time.

[Marko Mikulicic]

what about a "deep map" function that can do things such "add a label to all things" but preserving the original structure?

[Angus Lees]

Oh interesting. Yes, that would work too, and be more useful.

There's a big gotcha here with jsonnet (and kube.libsonnet in particular) - passing an object through a jsonnet function evaluates all the self expressions, so the nice lazy kube.libsonnet stuff can't be used to make further changes on the other side of a deepMap().

Both flatten and deepMap are quite easy to implement in jsonnet fwiw.

[Marko Mikulicic]

passing an object through a jsonnet function evaluates all the self expressions

what do you mean?

local x = {
  a: self.b + 2,
  b:: 2,
};

local f(x) = x { a: super.a * 10, c: self.b + 2 };

f(x) {
  b: 40,
}

-->

{
   "a": 420,
   "c": 42
}

[Marko Mikulicic]

There is indeed a problem with std.mapWithKey though: it does effectively manifest the object (killing all lazy expressions and also hidden fields)

[Marko Mikulicic]

A more practical example of what I'm talking about.

Given:

// mapObjects applies func on every object in a tree.
local mapObjects(obj, func=function(n, o) {}) = {
  [n]+: mapObjects(obj[n], func) + func(n, obj[n])
  for n in std.objectFields(obj)
  if std.isObject(obj[n])
};

// deepMerge applies a patch to every object matching a predicate.
local deepMerge(obj, patch, pred=std.isObject) = obj + mapObjects(obj, function(n, o) if pred(o) then patch else {});

and see how you can apply an override deeply in the structure while preserving super/self:

local tree = {
  universe: {
    life: {
      kind: 'foo',

      everything: self.b * $.params.mult,
      b:: 0,
      x:: 1,
    },
  },

  params:: {
    mult: 1,
  },
};


local isFoo(o) = std.objectHas(o, 'kind') && o.kind == 'foo';

local patch = {
  b: 20 + self.x,
};

deepMerge(tree, patch, isFoo) {
  params+: { mult: 2 },
},

would return:

{
   "universe": {
      "life": {
         "everything": 42,
         "kind": "foo"
      }
   },
}

While, using mapWithKey would "disconnect" the root object:

local patch = {
  b: 20 + self.x,
};

tree + std.mapWithKey(function(n, o) if isFoo(o) then o + patch else o, tree) + {
  params+: { mult: 2 },
}

->

{
   "universe": {
      "life": {
         "everything": 21,
         "kind": "foo"
      }
   }
}

A more detailed example in https://gist.github.com/mkmik/aa7f495541e4c883ad2426615d6e3525

[Angus Lees]

I'm glad to know the late-bound-self semantics do indeed survive in more situations than I thought :)

(I've been caught out by this in the past, and I can't recall exactly when - it was long before std.mapWithKey existed, so it isn't limited to just that function. I obviously over-learned to just avoid functions 😛 )

I think a structure-preserving map is a great idea, in addition to a flatten (they're both useful, and different).

[Marko Mikulicic]

it seems that the recursive jsonnet walk fits as a kubecfg primitive (so we know it uses the very same logic used by kubecfg internally).

On the same note, perhaps the predicate that tells whether a given object is a k8s resource (and of which kind) could be exposed as a kubecfg function (I assume it's used by the aforementioned walk logic).

What about, deepMerge, should it be bundled by kubecfg or delegated to a library like https://github.com/bitnami-labs/kube-libsonnet ?

[Angus Lees]

What about, deepMerge, should it be bundled by kubecfg or delegated to a library like https://github.com/bitnami-labs/kube-libsonnet ?

I don't care particularly. If we add it to kubecfg.libsonnet, then there's an implication that it will be supported going forward.

... 🤔 I think I would like to stick to "obviously correct" function signatures for kubecfg.libsonnet for now - and I think that means (in some pseudo type syntax): (Happy to bikeshed the specific names)

• flatten(tree) -> [objects]
• deepMap(func(obj)->obj, treeish) -> treeish
• isK8sObject(x) -> bool

In particular, deepMerge is useful but can easily be derived from deepMap, so I don't think we should add deepMerge (yet). Oh, and I'm assuming flatten/deepMap do not recurse inside k8s objects - ie: they're explicitly just the kubecfg-encouraged jsonnet structure above/around k8s resources (except v1.List?) and not general-purpose jsonnet library routines. I think that's appropriate for kubecfg.libsonnet, but I could be convinced otherwise...

Note the above deepMap doesn't provide any context to the function. I could imagine an extended version where we also pass some representation of "where" in the tree we are currently (perhaps as an array (stack) of (parent object/array, name string/integer) pairs).

Thoughts?