Fetch file from private S3 bucket with AWS keys
Schaechtle opened this issue · comments
I am trying to fetch a file from a private s3 bucket that was pushed with DVC. It's not clear to me how I make my AWS keys available to pkgs.dvc-nix.fetch-dvc and if that's even possible. I've got some feedback from the Nix community saying that nothing in Nix supports the AWS Sig 4 protocol which is required to authenticate. On the other hand, there seem to be Nix users who have use cases for making their AWS keys available to Nix. So I find it hard to believe that there are no workarounds.
Is there any way to do this with dvc.nix?
Providing credentials to be accessible during a build in nix is really tricky. This unfortunately gets even worse if you're running with a multi-user installation, because in that case using impureEnvVars isn't really possible, since then the environment variables need to be present when the nix daemon is started, not when you run nix on the command line. One possible way to provide credentials to a build is to a nix build is to pass the --extra-sandbox-paths ... option with a path to your credentials file and then refer to that same path as a string inside the build.
The fetchs3 fetcher in nixpkgs seems to use the aws cli to download the files. This could be a reasonable approach to download the files but this requires either passing credentials that will be stored as plain text in the world-readable store or have a credentials service running on the same machine that the aws s3 cp references during execution.
Moreover, I'll have to update the flake a bit because the packages are using the let bindings and not taking the values from final. This means that overriding fetch-md5-file will not be enough in your case.
In conclusion: There's currently no easy way to support fetching DVC files from a private S3 bucket using dvc.nix but if you're interested in supporting this use case I would be willing to assist you in implementing this.