iesahin / xvc

A robust (🐢) and fast (🐇) MLOps tool for managing data and pipelines in Rust (🦀)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add hierarchical credentials for S3 based storages

iesahin opened this issue · comments

What I mean is something like:

For example, for S3,

  • If there is an environment variable XVC_STORAGE_ACCESS_KEY_ID_remote1, it's considered to have more priority over other credentials.
  • Q: Should we continue to support XVC_STORAGE_ACCESS_KEY_ID (without remote1 per storage)? I think for multiple storage types, this can cause difficulties. If I want to access GCS with its credentials and S3 with its credentials, having a single environment variable makes everything complex. OTOH if there is such a single remote storage, that might work. But for single storage, the above scheme might also work. There is no need for a common credentials scheme for all storage types. If user wants to set credentials for remote1 specifically, they can do using the above environment variable.

The following variables are recognized by Xvc for credentials

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

DIGITAL_OCEAN_ACCESS_KEY_ID
DIGITAL_OCEAN_SECRET_ACCESS_KEY

WASABI_ACCESS_KEY_ID
WASABI_SECRET_ACCESS_KEY

GCS_ACCESS_KEY_ID
GCS_SECRET_ACCESS_KEY

R2_ACCESS_KEY_ID
R2_SECRET_ACCESS_KEY

MINIO_ACCESS_KEY_ID
MINIO_SECRET_ACCESS_KEY