Map key merging is different from internal x/config library.
willhug opened this issue · comments
Ran into this migrating one of my services to FX.
If we have two configuration files that we want to merge together like:
base.yaml
test:
testmap:
testkey: "test"
and
development.yaml
test:
testmap:
testkey2: "anothervalue"
In our current internal config library, the resulting merged config will be:
test:
testmap:
testkey2: "anothervalue"
the testmap
key of the development.yaml configuration takes precedence and removes the testkey
map key from the base.yaml.
Using go.uber.org/config
the result config is:
test:
testmap:
testkey: "test"
testkey2: "anothervalue"
Which is different than format we've pushed for internally in all go and python services. And will likely cause a large amount of developer confusion as we move people away from the internal config system and towards the fx world. Unless there is a compelling reason to keep the new way it will be advantageous to keep parity with the current internal config merging.
This is a little invoke function I used to test this:
type Config struct {
Test map[string]interface{} `yaml:"test"`
}
func testConfig(cfg config.Provider, logger *zap.Logger) {
var cfgData map[string]interface{}
if err := cfg.Get("test").Populate(&cfgData); err != nil {
logger.Error(err.Error())
return
}
logger.Sugar().Infof("cfg: %v", cfgData)
var xcfg Config
if err := xconfig.Load(&xcfg); err != nil {
logger.Error(err.Error())
return
}
logger.Sugar().Infof("xcfg: %v", xcfg.Test)
}
The internal merge is based on applying yaml unmarshaler repeatedly on the same variable, which doesn't have any documented merge behavior and is not recursive for interface{}, but is for structs. Here is an example:
package main
import (
"fmt"
yaml "gopkg.in/yaml.v2"
)
const (
base = `
test:
testmap:
testkey: test
`
dev = `
test:
testmap:
testkey2: anothervalue
`
)
func main() {
type Config struct {
Test map[string]interface{} `yaml:"test"`
}
type StructConfig struct {
Test struct {
Testmap struct {
Testkey string
Testkey2 string
}
}
}
var c Config
var s StructConfig
for _, str := range []string{base, dev} {
if err := yaml.Unmarshal([]byte(str), &c); err != nil {
panic(err)
}
if err := yaml.Unmarshal([]byte(str), &s); err != nil {
panic(err)
}
}
fmt.Printf("interface config:\n %+v\n", c)
fmt.Printf("struct config:\n %+v\n", s)
}
Output:
interface config:
{Test:map[testmap:map[testkey2:anothervalue]]}
struct config:
{Test:{Testmap:{Testkey:test Testkey2:anothervalue}}}
Config library is being consistent in that case and behaves in a similar way as json merge patch by merging everything recursively.
I was under the impression that the internal configuration library was always
performing a deep merge--that seems like the correct behavior to have if we're
going to have merging at all.
@alsamylkin, putting aside whether we should do this or not, is it even
technically feasible to replicate the behavior?
@abhinav Unfortunately it doesn't do a deep merge and I don't see we can replicate this behavior: merge and decoding are separate steps in config library, but the internal merge depends on the type of variable being decoded. I am happy to jam offline on a solution though.
Synced offline and decided to not to try mimic the internal merge and keep it as is.