Loading a YAML file makes all keys lowercase, but YAML keys are case sensitive.
barsanuphe opened this issue · comments
Since the recent changes, all keys have become lowercase.
If I'm not mistaken (see this reply from YAML spec author), YAML keys are case sensitive.
For instance, calling GetStringMapStringSlice() while loading a YAML file makes all of the keys of the resulting map lowercase, which is unexpected and was not the previous behavior.
GetStringMapStringSlice() while loading a YAML file
When do you call that method while loading a YAML file?
If you have:
yaml_key:
Case Sensitive Key:
- value1
- value2
GetStringMapStringSlice("yaml_key") now gives a map with a case sensitive key key.
To conclude: Viper have always tried to be case-insensitive by doing a lower-casing of the keys given. It was however not being doing that deeply for maps. Now that is fixed and we are seeing some breakage in the wild.
The main problem seems to be the expectation that you should be able to unmarshal the config map into a struct or similar.
Because for plain lookups, like viper.Get("a.b.A.d."), this is clearly an improvement.
@spf13 should chime in here.
GetStringMapStringSlice("yaml_key") now gives a map with a "case sensitive key" key.
No, it gives a lower-cased key.
No, it gives a lower-cased key.
Yes, I meant it gives the value: case sensitive key in my example.
I understand your point of view that this is the intended behavior, but I find it surprising for YAML. And I'm not expecting to unmarshall the config map into anything, I'm happy dealing with a map[string][]string. I was just relying on the keys to have the same case I put in the YAML file.
My "top" keys are lowercase though, so I didn't realize viper was already returning lowercased versions for anything but maps.
I understand your situation -- I have a similar breakage in one of my apps. We should probably have handled this in a better way, but I believe it is the intended behavior of Viper from the start. @spf13 should decide.
The way I think it should work is a bit of a hybrid of case sensitivity.
Keys should be treated as case insensitive when getting, setting or
merging. I can't see another approach that would make sense here.
Returning keys would ideally be in the case originally defined.
The challenge here is that while yaml is case sensitive, not all of the
configuration options are (flags, env vars, etc). Even Go itself has odd
sensitivity behaviors as the initial letter is loaded with exportability.
On Fri, Oct 14, 2016 at 7:06 PM Bjørn Erik Pedersen <
notifications@github.com> wrote:
I understand your situation -- I have a similar breakage in one of my
apps. We should probably have handled this in a better way, but I believe
it is the intended behavior of Viper from the start. @spf13
https://github.com/spf13 should decide.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#260 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAKlZFU7xGn_CZg1KReWeJ9rlKudEy1nks5q0ArYgaJpZM4KXgSN
.
Returning keys would ideally be in the case originally defined.
Which is a very expensive requirement, and in many cases not possible.
As I said earlier, this change was pushed earlier but then I did a revert. Viper is getting pretty complex -- it would be nice to have a simple and agreeable understanding of the casing of strings.
I am not familiar with many configuration use cases and the requirements for case sensitive keys, but if I can chime in, I really think the consistent conversion to lower case keys is the way to go. Feel free to point out some cases that I have no vision on, but I don't really see an upside to complicating the code with some hybrid handling.
I agree with @awishformore -- I suggest we just ride off the storm.
I am not familiar with many configuration use cases and the requirements for case sensitive keys
Our application uses viper for describing environment variables in a YAML file:
environment:
FIRST_THING: yes
secondThing: no
These could have arbitrary names/values since the user defines them. But environment variables are case sensitive, so this means we can no longer represent them naturally as a map. I suppose we will have to switch to something like a list of key=value if we want to continue using viper.
Hi, to follow up on this issue, I encountered a bug with my configuration, with some Keys in lists and some other unlisted:
package main
import (
"bytes"
"fmt"
"github.com/spf13/viper"
)
var yamlExample = []byte(`
Unlisted:
Test: true
Listed:
- Test: true
`)
func main() {
viper.SetConfigType("yaml")
viper.ReadConfig(bytes.NewBuffer(yamlExample))
// Will output "test:true"
var a interface{}
viper.UnmarshalKey("Unlisted", &a)
fmt.Println(a)
// Will output "Test:true"
var b interface{}
viper.UnmarshalKey("Listed", &b)
fmt.Println(b)
}
Shouldn't all keys be treated the same way ? These two different behaviors look quite error-prone to me.
It seems to be intentional
func insensitiviseMap(m map[string]interface{}) {
for key, val := range m {
switch val.(type) {
case map[interface{}]interface{}:
// nested map: cast and recursively insensitivise
val = cast.ToStringMap(val)
insensitiviseMap(val.(map[string]interface{}))
case map[string]interface{}:
// nested map: recursively insensitivise
insensitiviseMap(val.(map[string]interface{}))
}
lower := strings.ToLower(key)
if key != lower {
// remove old key (not lower-cased)
delete(m, key)
}
// update map
m[lower] = val
}
}
+1
For what it is worth I recently discovered that the go-yaml yaml.Marshal() method automatically forces names to lowercase unless they are explicitly tagged, which I personally find really annoying. I was expecting the same json.Marshal() behavior (as many others have as well):
type Position struct {
Row int
Col int
Byt int
SrC string `yaml:",omitempty"`
}
func (p Position) YAML() string {
byt, err := yaml.Marshal(p)
if err != nil {
return ""
}
return string(byt)
}
func ExamplePosition_YAML() {
p := Position{1,2,3,"foo"}
fmt.Print(p.YAML())
// Output:
// Row: 1
// Col: 2
// Byt: 3
// SrC: foo
}--- FAIL: ExamplePosition_YAML (0.00s)
got:
row: 1
col: 2
byt: 3
src: foo
want:
Row: 1
Col: 2
Byt: 3
SrC: foo
FAIL
The only way around it is to explicitly tag them the way you want them:
type Position struct {
Row int `yaml:"Row"`
Col int `yaml:"Col"`
Byt int `yaml:"Byt"`
SrC string `yaml:"SrC,omitempty"`
}Which gets doubly annoying if you want to json.Marshal() as well. (I have pos.String(), pos.JSON(), and pos.YAML()):
type Position struct {
Row int `json:"Row" yaml:"Row"`
Col int `json:"Col" yaml:"Col"`
Byt int `json:"Byt" yaml:"Byt"`
SrC string `json:"SrC,omitempty" yaml:"SrC,omitempty"`
}Of course adding TOML to the mix would get really fun. I really like the idea of being able to represent structures in multiple formats, but this can get tedious fast.
Which led me to research what the Kubernetes folks are doing because they have such a massive dependency on YAML and JSON working in what appears to be a case-sensitive way.
How Kubernetes Does It
First of all, under the hood k8s use https://github.com/ghodss/yaml to get rid of YAML marshaling tags altogether even thought it uses the same go-yaml module.
Second, they use the explicit name in the struct tags:
./staging/src/k8s.io/client-go/tools/clientcmd/api/v1/types.go: APIVersion string `json:"apiVersion,omitempty"`
Viper Thwarted by go-yaml?
I'm betting viper is facing this issue because of the implied names from marshaling structs that are not explicitly named with tags. As long as go-yaml is being used (which is a port of a C lib) this will likely remain.
Anyone feel like writing a Go YAML library that actually respects the same marshaling naming tags like JSON does? I bet a fork of go-yaml could do it. 😁
+1 for adding a setting to preserve casing.
I wanted to document my use case:
My app looks up EC2 Instances by tags, but ec2 tags are case sensitive in the aws sdk. I was hoping to store my tags in a yaml file, like so:
tags:
Service: myapp
Owner: Dylan
Environment: dev
Due to this, I'm not going to be able to use viper.
This problem seems to be fixed already, but I don't see new releases since end of May 2019. Does anyone know when next release is scheduled?
I added it to the 1.5 milestone. New release will happen soon.
Releasing 1.5.0
Can someone give a tl;dr summary of the solution present in 1.5.0?
Github release only states Documentation and other fixes, would be nice to have a pointer for everyone who visits this page
I assumed this has been fixed, but looking through the issue I'm not so sure anymore. I'm reopening the issue until I figure out what's going on.
I confirm that the issue is still present. When using 1.5.0, keys in TOML configuration files (in my case) are still converted to lowercase.
This issue also happens in json configuration now. Is there anyone working on it?
Thanks for the pointer @sbourlon. From this comment it looks like Viper is not going to get case sensitivity any time soon, if ever.
If you have:
yaml_key: Case Sensitive Key: - value1 - value2GetStringMapStringSlice("yaml_key") now gives a map with a
case sensitive keykey.
not a workaround in 2020/06/02 master branch.
When my yaml like:
iocFilterSchemaForHeader:
<2 blanks>FileSize: [FileSize]
<2 blanks>FileMD5: [FileMD5]
<2 blanks>FileSHA1: [FileSHA1]
<2 blanks>FileSHA256: [FileSHA256]
<2 blanks>PPf: [PPf]
<2 blanks>Family: [Family]
<2 blanks>Score: [Score]
And read it with viper.GetStringMapStringSlice("iocFilterSchemaForHeader"), print it, it like:
get iocFilterSchemaForHeader:map[family:[Family] filemd5:[FileMD5] filesha1:[FileSHA1] filesha256:[FileSHA256] filesize:[FileSize] ppf:[PPf] score:[Score]]
The keys still case-insensitive!
UnmarshalKey
This is a workaround for case sensitive
+1
It is already 2024, and I still have not seen the v2 version of Viper. The corresponding issue seems to have been put on hold, which has forced me to abandon the use of Viper in some projects. This is because in some cases, I need to ensure that the configuration keys are not converted to lowercase. For instance, I want to place a map in the configuration, but when it is read back, it has changed its appearance.This is because it might be a name.
config.yaml:
users: {"Bob": 21, "Jeff": 18}
In English, there are also words where the meaning is different when written with different cases, isn’t there? For example, “China” and “china.”
any update?
+1