Map key merging is different from internal x/config library.

[Will Hughes]

Ran into this migrating one of my services to FX.

If we have two configuration files that we want to merge together like:

base.yaml

test:
  testmap:
    testkey: "test"

and

development.yaml

test:
  testmap:
    testkey2: "anothervalue"

In our current internal config library, the resulting merged config will be:

test:
  testmap:
    testkey2: "anothervalue"

the testmap key of the development.yaml configuration takes precedence and removes the testkey map key from the base.yaml.

Using go.uber.org/config the result config is:

test:
  testmap:
    testkey: "test"
    testkey2: "anothervalue"

Which is different than format we've pushed for internally in all go and python services. And will likely cause a large amount of developer confusion as we move people away from the internal config system and towards the fx world. Unless there is a compelling reason to keep the new way it will be advantageous to keep parity with the current internal config merging.

This is a little invoke function I used to test this:

type Config struct {
	Test map[string]interface{} `yaml:"test"`
}

func testConfig(cfg config.Provider, logger *zap.Logger) {
	var cfgData map[string]interface{}
	if err := cfg.Get("test").Populate(&cfgData); err != nil {
		logger.Error(err.Error())
		return
	}
	logger.Sugar().Infof("cfg: %v", cfgData)


	var xcfg Config
	if err := xconfig.Load(&xcfg); err != nil {
		logger.Error(err.Error())
		return
	}
	logger.Sugar().Infof("xcfg: %v", xcfg.Test)
}

[Deleted user]

The internal merge is based on applying yaml unmarshaler repeatedly on the same variable, which doesn't have any documented merge behavior and is not recursive for interface{}, but is for structs. Here is an example:

package main

import (
	"fmt"

	yaml "gopkg.in/yaml.v2"
)

const (
	base = `
test:
  testmap:
    testkey: test
`
	dev = `
test:
  testmap:
    testkey2: anothervalue
`
)

func main() {
	type Config struct {
		Test map[string]interface{} `yaml:"test"`
	}
	type StructConfig struct {
		Test struct {
			Testmap struct {
				Testkey  string
				Testkey2 string
			}
		}
	}

	var c Config
	var s StructConfig
	for _, str := range []string{base, dev} {
		if err := yaml.Unmarshal([]byte(str), &c); err != nil {
			panic(err)
		}
		if err := yaml.Unmarshal([]byte(str), &s); err != nil {
			panic(err)
		}
	}
	fmt.Printf("interface config:\n %+v\n", c)
	fmt.Printf("struct config:\n %+v\n", s)
}

Output:

interface config:
 {Test:map[testmap:map[testkey2:anothervalue]]}
struct config:
 {Test:{Testmap:{Testkey:test Testkey2:anothervalue}}}

Config library is being consistent in that case and behaves in a similar way as json merge patch by merging everything recursively.

[Abhinav Gupta]

I was under the impression that the internal configuration library was always
performing a deep merge--that seems like the correct behavior to have if we're
going to have merging at all.

@alsamylkin, putting aside whether we should do this or not, is it even
technically feasible to replicate the behavior?

[Deleted user]

@abhinav Unfortunately it doesn't do a deep merge and I don't see we can replicate this behavior: merge and decoding are separate steps in config library, but the internal merge depends on the type of variable being decoded. I am happy to jam offline on a solution though.

[Deleted user]

Synced offline and decided to not to try mimic the internal merge and keep it as is.