Sync two S3 buckets in two different AWS accounts using AWS Profile + assumeRole to access S3 API?
frittentheke opened this issue · comments
Thanks for creating this great tool!
I attempted to use s3sync with a setup that requires an assumeRole to access the target bucket but apparently this is not supported (yet)?
More particular the setup looks like this:
- Two AWS account, each containing an S3 bucket:
Account 12345: source-bucket
Account 67890: destination-bucket
- Source bucket received a policy like:
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::DESTINATION_ACCOUNT_ID:root"
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::destination-bucket/*",
"arn:aws:s3:::destination-bucket"
]
}
to allow the destination bucket to list and pull the files.
- I then used the aws cli tool to verify things work:
export AWS_PROFILE=my.account.test
aws s3 sync s3://source-bucket s3://destination-bucket
and it was syncing the objects quite happily ... just not as fast as I'd like to ;-)
My questons or potential feature request now is to be able to use s3sync in this setup.
Hello.
Thanks for feedback;).
Are you checked the s3sync with this setup? AWS SDK supports AWS_PROFILE
and should read credentials from it.
So you try to run s3sync and report a result.
Please use latest version (2.15). I was build create this release few min ago. Prev release was not contains latest changes for some auth methods.
@larrabee thanks for the quick reply. Sorry for not mentioning this in my initial post - of course I did test the setup with s3sync, but it did not work.
I now used the new release, but that also fails:
INFO[0000] Starting sync
DEBU[0006] S3 listing failed with error: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
DEBU[0006] Pipeline step: ListSource finished
DEBU[0006] Recv pipeline err: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
ERRO[0006] Sync error: pipeline step: 0 (ListSource) failed with error: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors, terminating
DEBU[0006] Pipeline step: ACLUpdater finished
DEBU[0006] Pipeline step: LoadObjData finished
DEBU[0006] Pipeline step: UploadObj finished
DEBU[0006] Pipeline step: Terminator finished
DEBU[0006] All pipeline steps finished
DEBU[0006] Pipeline terminated
INFO[0006] 0 ListSource: Input: 0; Output: 0 (0 obj/sec); Errors: 1
INFO[0006] 1 LoadObjData: Input: 0; Output: 0 (0 obj/sec); Errors: 0
INFO[0006] 2 ACLUpdater: Input: 0; Output: 0 (0 obj/sec); Errors: 0
INFO[0006] 3 UploadObj: Input: 0; Output: 0 (0 obj/sec); Errors: 0
INFO[0006] 4 Terminator: Input: 0; Output: 0 (0 obj/sec); Errors: 0
INFO[0006] Duration: 6.27037978s
ERRO[0006] Sync Failed
Hello again;)
Are you using AWS_PROFILE
correctly?
AWS_PROFILE
should contain account name from credentials file. File location can be specified by variable AWS_SHARED_CREDENTIALS_FILE
.
For example:
export AWS_PROFILE="default"
export AWS_SHARED_CREDENTIALS_FILE="/home/test/aws_creds"
>> cat /home/test/aws_creds
[default]
aws_access_key_id = KEY
aws_secret_access_key = SECRET
Is AWS regions correct? By default it's connecting to us-east-1
region. You can set region by args --sr
and --tr
.
@larrabee yes, I did all that:
$ env | grep AWS
AWS_PROFILE=my.account.test
AWS_SHARED_CREDENTIALS_FILE=/home/myuser/.aws/credentials
The credentials file (/home/myuser/.aws/credentials
) looks like this:
[default]
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED
The difference in my case is that I want to "assume a role". The credentials file contains my access credentials, but then a file ~/.aws/config
exists containing:
[profile my.account.test]
role_arn = arn:aws:iam::1234567890:role/myRoleInTheOtherAccount
source_profile = default
region = eu-central-1
So I assume the role of myRoleInTheOtherAccount
which actually allows access to the S3 bucket, not the role / account I have when using the aws_access_key_id/aws_secret_access_key.
In short, the aws_secret_access_key authenticate me as a human, the role is the role I have related to the particular account. This is quite common in larger / enterprise usage of AWS accounts.
I'm commit changes to branch assumeRole_debug
.
Can you build and test it?
Hey @larrabee thanks for really digging into this! Really appreciate it.
Are we talking about this commit: ded3296 ? What is that intended to do as the line ded3296#diff-44bbcc9d983da65f32aa64529eb190e2R50 is commented out ...
I built s3sync in branch assumeRole_debug with the commit ... but nothing has changed unfortunately.
Please try new version from branch assumeRole_debug
, it's should works (It's working for me).
Yeah! It works splendidly.
Changes merged to master.