Support loading single keys from a file (e.g. secrets files)

[Jameel Al-Aziz]

I apologize if this has been asked and answered before, but I couldn't find anything.

These days (especially in Kubernetes), it's not atypical to have secrets mounted as individual files to be read from. Unlike a configuration, these files each represent a single secret. A good example of this is how secrets-store-csi-driver works.

While I realize this could be handled outside of koanf and the secret values merged into the config manually, it would be nice kind of convenient to have a provider and a parser that can load values from a file directly into a single key.
I also realize this may be fairly trivial to implement as a third party provider / parser.

If you believe this functionality merits inclusion in the default set of providers and parsers, I would be happy to give it a shot and submit a PR.

I was thinking of introducing:

1. A provider that loads a file into a single key + a parser that simple returns raw values as strings
2. A provider that loads all files in a directory into a map of values

Would that make sense?

[Kailash Nadh]

Hi @jalaziz. Do these secret files have a specific format? Could you point to a sample? I'm not familiar with them.

[Jameel Al-Aziz]

@knadh no specific format. They typically contain whatever the secret is (be it a string, JSON, a PEM-encoded key, etc).

For example, you might have a file named /run/secrets/db_password that contains a password in plaintext. Or you might have /run/secrets/db_creds that may contain a JSON blob of the username and password.

This is way of passing secrets is common in containerization and used by:

• Docker
• Kubernetes Secrets Store CSI Driver (which supports AWS Secrets Manager, Vault, and several other solutions)
• Google Cloud Run

[Kailash Nadh]

I'm a little confused by this. If secrets are just strings that can be in any file, JSON, YAML, TOML, or env, or even just a file with no format, how do you propose that a new provider handle that? How will the provider work with arbitrary data?

You can just use the existing providers and parses to read a file (YAML, JSON etc.) and just access that key, right? koanf.String("myapp.mysecret").

[Jameel Al-Aziz]

Maybe I'm mistaken, but the issue with the current providers and parsers is two fold:

1. All current parsers expect structured data.
2. The current file provider only supports a single file.

What I believe is needed is:

1. A new "directory" provider that reads files into map keyed by file name. This would koanf.String("db-secret") to map to the value of a file and allow mapping the value to a struct key when unmarshalling.
2. A new provider or parser that maps the value of a provider to a specific map key (rather than the config map itself).
3. If possible, some combination of a provider and parser that would allow you to map a directory of files to different "types" of values.

It actually be easier to talk about specific examples of what I'm trying to accomplish.

In the simplest case, I would like the value of koanf.String("myapp.foo") to be the contents of a file named /var/run/secrets/foo. I don't believe this is possible today because I can't seem to find a way to parse the contents of a file into a single key.

For example, if /var/run/secrets/foo contained the simple text content of secret-value. I would like koanf.String("foo") to return secret-value. The important detail here is that key in the config maps to a specific file on disk.

In the more complex case, given a directory (e.g. /run/secrets/) with the following files:

.
├── bar // Contains basic text API Key
└── foo // Contains JSON value

I would like to be able to unmarshal into the following structs:

type Nested struct {
    a string
    b int64
}

type Config struct {
    foo Nested
    bar string
}

I think this case would be a more challenging given the current structure of providers and parsers as the parser would have to be determined lazily based on the expected type. A simple version would be to load all the files in the directory into a config map that looks like:

{ foo: "{\"a\": \"one\", \"b\": 2}", bar: "secret-value"} 

This would then allow koanf.String("foo") and koanf.String("bar") to return the contents of the corresponding files and the end user would have to parse from there.

[Kailash Nadh]

This seems very opinionated and not necessarily generalisable.

For example, if /var/run/secrets/foo contained the simple text content of secret-value. I would like koanf.String("foo") to return secret-value.

Someone would want that to be secrets/foo, secrets/bar etc. Mapping file names to key is thus arbitrary.

I think you can write a provider that does this for usecase, but it wouldn't make sense to merge it into koanf's core. You can publish this on your own repo and we can add it to the 3rd party providers list in the koanf docs.

[Jameel Al-Aziz]

This seems very opinionated and not necessarily generalisable.

I'm a bit confused by this. It should be generalizable. The file names in the directory form the keys to the conf map and the values are the file content. The idea is to support any file name.

That being said, I'm likely doing a very poor job explaining the desired behavior. I'm happy to build a 3rd party provider to support this.

Thanks for the help.