Users should be able to persist credentials for third-party services

[Andrew Louis]

Connectors that interact with third-party services often require access tokens, user ids, or other secrets. Chronicle-ETL needs a lightweight secret management system so that jobs don't have to have these specified manually each time.

At minimum, we need namespaced key/value pairs. We can use one yml file per namespace, stored in ~/.config/chronicle/etl/secrets (or $XDG_CONFIG_HOME). By convention, we can use one namespace per provider.

Proposed UX

chronicle-etl secrets:set namespace key
chronicle-etl secrets:unset namespace key
chronicle-etl secrets:list namespace

# set with interactive prompt
chronicle-etl secrets:set pinboard access_token

# set from stdin
echo -n 'FOO' | chronicle-etl secrets:set pinboard access_token

# set from cli option + environment variable
chronicle-etl secrets:set pinboard access_token --body "$PINBOARD_ACCESS_TOKEN"

When running a job, secrets will be merged into the connector's option hash. Secrets can come from a few places. This is the load order:

• secrets from the namespace matching the plugin provider's name
• secrets from a custom namespace (specified in job options with something like secrets: pinboard-personal
• via cli flags for common secrets (--access_token FOO)

Decisions

• base64 encode secrets?
• encrypt secrets?
• save timestamps for each key/value pair?
• save a chronicle-etl version in the yml file to make future migrations easier