Feature Request: Pull individual secrets from AWS SM secret objects
zachequi opened this issue · comments
AWS's default behavior is to store secrets as a JSON object with lots of secrets as key/values on that object. It'd be nice to be able to pull an individual value from that secret object. Something like
awssm:arn:aws:secretsmanager:us-west-2:1111111111:secret:mysecretobject.somepassword
Hi @zachequi, thanks for creating this issue. Fetching a specific field in a JSON secret is something we would like to support.
It is common practice on AWS to store secrets as JSON objects, although it is not mandatory. It is also possible with any secret storage system, technically. For that reason, I am reluctant to add support for JSON fields only for awssm: secrets. I would rather add more general support for JSON, and maybe even templating. I had something like this in mind:
awssm:arn:aws:secretsmanager:us-west-2:1111111111:secret:mysecretobject|template:{{ . | fromJson | .somepassword }}
I figure the |template: filter would serve even more use-cases, not just JSON.
What do you think? How does the |template:{{...}} syntax look?
If you're going for something very generic and powerful, that works. One concern would just be the aesthetics of it if, for example, I have 10 or 15 secrets all using the syntax it starts to be quite a lot of repetitive "code" inside an envvar. Some syntactic sugar to make it a bit simpler would help. Since JSON is so common maybe use a json-like syntax? secret:mysecretobject['key']
Separately, if there are multiple secrets loading the same object and pulling different keys, would whisper download the secret multiple times or cache it?
Currently, whisper does not cache anything. I can see how this can lead to a large number of unnecessary requests. I think this should be a separate issue however.
Regarding the heaviness of the template:{{ ... }} syntax, I agree. A more concise way to extract a single field in a JSON object would be nice.
While secret:mysecretobject['key'] is indeed concise, I fear it may be complex to implement nested field lookups this way. What I mean by nested fields: secret:mysecretobject['key']['other_key'].
I feel like a jq-like syntax may be ideal here. For example: secret:mysecretobject|jq:.key.other_key. Although this may suffer from feature bloat: whisper would suddenly include jq.
Alternatively, JSONPath could also work: secret:mysecretobject|jsonpath:$.key.other_key.
Or Kubernetes' jsonpath/template combo: secret:mysecretobject|jsonpath:{.key.other_key}.
What do you think?
Either way, I think whisper's internals need to support filters, be it template or jsonpath or both. I think there should be a clear delimiter between a reference to a secret in a secret store (awssm:my-secret) and any applied filter (jsonpath:{.key}). A pipe | symbol is often used for this purpose, so we might want to opt for that.
I like both of your most recent suggestions! Aesthetically I'd go for the Kubernetes style |jsonpath:{path.to.key}. I'll open another issue re: caching. Thanks for all your hard work and receptiveness to feedback!
@zachequi quick update on this: support for the jsonpath filter is merged into the main branch 👍
The change will be included in release v0.4. I also plan on adding concurrent fetching of secrets (#41) and caching (#30) in the same release. Once those changes are added in, I'll release v0.4 and close this issue.
Whisper now does this with release v0.4.0.