AuthFailure with IAM role and signature version 4
apptaro opened this issue · comments
Net::Amazon::EC2 0.33 have an auth issue with IAM role and signature version 4.
$ cat test.pl
#!/usr/bin/perl
use strict;
use Net::Amazon::EC2;
my ($signature_version) = @ARGV;
if (not defined $signature_version) {
die "No signature version specified\n";
}
my $ec2 = Net::Amazon::EC2->new(
region => 'ap-northeast-1',
signature_version => $signature_version
);
my $reservations = $ec2->describe_instances;
foreach my $reservation (@$reservations) {
foreach my $instance ($reservation->instances_set) {
print $instance->instance_id . "\n";
}
}
$ chmod u+x test.pl
$ ./test.pl 2
i-0066cd49489e4faa9
i-05eaa8b03dd55d22b
$ ./test.pl 4
Amazon EC2 Errors [Request 08ecf962-c346-4f75-afc2-837eaec2f78f]:
[AuthFailure] AWS was not able to validate the provided access credentials
$
I'm not sure if this issue is region dependent.
This issue has been reported here about two years ago:
https://rt.cpan.org/Public/Bug/Display.html?id=107491
OK, I have a PR up which - I think - addresses this issue. You can apply it on your test system by:
- Grabbing the diff
curl -LO https://patch-diff.githubusercontent.com/raw/mrallen1/net-amazon-ec2/pull/60.patch - applying it using
patch -p1 < 60.patch perl Makefile.PLmake install
And re-running your test script with signature v4.
Or you could git clone --branch fix_iam_v4_sigs https://github.com/mrallen1/net-amazon-ec2 and do the usual Makefile nonsense.
It'd be super helpful if you could enable debugging in net-amazon-ec2 by setting debug => 1 when you initialize the ec2 object. Thanks.
Obviously, you'll want to leave out your secrets if you post the debug output. :)
Hi, I applied the patch and the issue is gone!
Great. I'll merge this PR and get a new release up on CPAN
OK, I released 0.35 to CPAN today. Open a new ticket if there's more trouble.
Sorry for commenting on closed issue but I have the same issue with 0.36.
It is just like 25% of tries actually succeed. Any ideas how to troubleshoot it ?
It happens also with version 2 signature too.
Failure output:
/ec2-consistent-snapshot.bin --use-iam-role --no-freeze-filesystem /etc/host
name --region=us-west-1 --debug --signature-version 4
ec2-consistent-snapshot.bin: Authenticating with IAM role
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: No volume ids specified; discovering volume ids
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: Discovering volume ids for: /etc/hostname
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: Determining instance id
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: create EC2 object
ec2-consistent-snapshot.bin: Endpoint: https://ec2.us-west-1.amazonaws.com
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: Fetching instance description for i-0c89d93e5ad619faf
Amazon EC2 Errors [Request c027f6ea-b2b6-47f1-be3d-31d1e4027578]:
[AuthFailure] AWS was not able to validate the provided access credentials
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: done
Success output:
# /ec2-consistent-snapshot.bin --use-iam-role --no-freeze-filesystem /etc/host
name --region=us-west-1 --debug --signature-version 4
ec2-consistent-snapshot.bin: Authenticating with IAM role
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: No volume ids specified; discovering volume ids
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: Discovering volume ids for: /etc/hostname
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: Determining instance id
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: create EC2 object
ec2-consistent-snapshot.bin: Endpoint: https://ec2.us-west-1.amazonaws.com
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: Fetching instance description for i-0c89d93e5ad619faf
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: Found EBS block devices for i-0c89d93e5ad619faf:
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: vol-0cd3b94c4fbfc6da0 /dev/sda1
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: vol-073e02d9776bc83e2 /dev/sdm
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: vol-0b5879ae4bd7672c2 /dev/sdn
--> Working on Net::Amazon::EC2
Fetching http://www.cpan.org/authors/id/M/MA/MALLEN/Net-Amazon-EC2-0.36.tar.gz ... OK
Configuring Net-Amazon-EC2-0.36 ... OK
My solution is to keep trying util it managed to generate signature properly for both AWS calls :)
until trigger_snapshot; do
sleep 3
done
Huh. That it "sometimes works" indicates to me there's some kind of race condition going on in the code somewhere (either yours or mine) which messes up the v4 signing code. Could you set the debug flag on your Net::Amazon::EC2 object? It will be very spammy, but it would help isolate the failure. (Also, if you paste in the output, please make sure you remove your AWS credentials 😄)
It is iam role attached to instance to all creds are temporary :) but thanks .
That is actually could be part of problem it seems.
Here is failure of describeInstance API Call:
ec2-consistent-snapshot.bin: Thu May 24 21:07:52 2018: create EC2 object
ec2-consistent-snapshot.bin: Endpoint: https://ec2.us-west-1.amazonaws.com
ec2-consistent-snapshot.bin: Thu May 24 21:07:52 2018: Fetching instance description for i-0887f8a21c3ce1ccd
$VAR1 = {
'Errors' => [
{
'Error' => {
'Message' => 'AWS was not able to validate the provided access credentials',
'Code' => 'AuthFailure'
}
}
],
'RequestID' => 'a6968303-9ac8-4737-9faf-72537f8b018a'
};
Last error was: AWS was not able to validate the provided access credentials at Net/Amazon/EC2.pm line 484.
Net::Amazon::EC2::_parse_errors(Net::Amazon::EC2=HASH(0x559ca3d4c120), HASH(0x559ca4e13c90)) called at Net/Amazon/EC2.pm line 2017
Net::Amazon::EC2::describe_instances(undef, "InstanceId", "i-0887f8a21c3ce1ccd") called at script/ec2-consistent-snapshot line 414
main::ec2_instance_description("https://ec2.us-west-1.amazonaws.com") called at script/ec2-consistent-snapshot line 440
main::discover_volume_ids("https://ec2.us-west-1.amazonaws.com") called at script/ec2-consistent-snapshot line 147
require main called at /usr/local/share/perl5/site_perl/PAR.pm line 645
PAR::_run_member(Archive::Zip::ZipFileMember=HASH(0x559ca17f4340), 1) called at script/main.pl line 26
require main called at /usr/local/share/perl5/site_perl/PAR.pm line 645
PAR::_run_member(Archive::Zip::ZipFileMember=HASH(0x559ca17f4628)) called at /usr/local/share/perl5/site_perl/PAR.pm line 432
PAR::import("PAR") called at -e line 606
eval {...} called at -e line 41
__par_pl::BEGIN() called at script/ec2-consistent-snapshot line 0
eval {...} called at script/ec2-consistent-snapshot line 0
ec2-consistent-snapshot.bin: Thu May 24 21:07:52 2018: done
CONTENT: Action=DescribeInstances&InstanceId=i-0887f8a21c3ce1ccd&Version=2014-06-15
CANONICAL REQUEST: POST
/
content-type:application/x-www-form-urlencoded
host:ec2.us-west-1.amazonaws.com
x-amz-date:20180524T210752Z
content-type;host;x-amz-date
519707a9e49b186fae6ed30297ecfd4359032677634a29dd9a01b5a899a457f5
STRING TO SIGN: AWS4-HMAC-SHA256
20180524T210752Z
20180524/us-west-1/ec2/aws4_request
4cc5c0e691e58b816b2abf58014ef1fbbab8beb636a7c955e9d433b7792a63b1
HTTP REQUEST: POST https://ec2.us-west-1.amazonaws.com
Authorization: AWS4-HMAC-SHA256 Credential=ASIAxxxxxx/20180524/us-west-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=db84ca08fbf1a51eb616ab755f31313aababc3c727fc8ef9dfa6575090bc15a5
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20180524T210752Z
Action=DescribeInstances&InstanceId=i-0887f8a21c3ce1ccd&Version=2014-06-15
ERROR CODE: AuthFailure MESSAGE: AWS was not able to validate the provided access credentials FOR REQUEST: a6968303-9ac8-4737-9faf-72537f8b018a
It is also can fail on CreateSnapshot action too.
Oh. And I use PAR/pp to build binary (well in fact it extracts all that stuff in /tmp)
I also experience this issue when using Net::Amazon::EC2 via ec2-consistent-snapshot with IAM Role authentication. Typically, the first authorization call falls and a follow-up call succeeds. I'm using v4 signatures.
I may switch to using Paws or static credentials as a workaround.
That sort of sounds like the code isn't getting the token at first and then when you try a second time it works.
I'm also having this problem and am currently working on the idea that it relates to time sync differences between the instance and the AWS IAM auth server based on these:
aws/aws-cli#2205
aws/aws-cli#1948
It may the the wrong track and have had no success after trying a few things, but I'll offer this here in case it's helpful or if someone else can offer more informed advice.
Edit: Though as I can repeatedly run aws cli commands with no issues, and the ec2-consistent-snapshot fails about 60% of the time, this is quite possibly the wrong track.
I was in contact with AWS Premium Support about this issue and they also suspected it was a time sync issue. But I compared the server in question to others and found it was accurate within a second. Also, AWS Premium Support mentioned the difference needed to be 5 minutes for this to matter, but in the second linked issue above, someone reported the problem with only about a one minute difference.
So, time skew is certainly one cause of this issue, but I'm not convinced it's the only cause.
Certainly nothing I've done with time sync has made any difference to IAM auth problems (called from ec2-consistent-snapshot as I think you've noticed). I see why you've switched to static credentials and I'm thinking that need to be my interim solution too.
And static credentials are also failing, just based on a simple test of:
perl -de1
DB<1> use Net::Amazon::EC2; my $ec2 = Net::Amazon::EC2->new(AWSAccessKeyId => 'XXXXXXXXX', SecretAccessKey => 'XXXXXXXX', signature_version => 4, debug => 1); $ec2->describe_instances;
(removed output)
Last error was: AWS was not able to validate the provided access credentials at /usr/share/perl5/Net/Amazon/EC2.pm line 484.
Net::Amazon::EC2::_parse_errors(Net::Amazon::EC2=HASH(0x55ebbbc2f340), HASH(0x55ebbbc32618)) called at /usr/share/perl5/Net/Amazon/EC2.pm line 2017
This suggests that it isn't solely an IAM issue, and the end result is that it will fail 2/3 or 3/4 times but work sometimes. However this test is using 0.35-1 which is all I have available on a production server.
And can this ticket be reopened?
We are also seeing this issue across several different AWS regions, can we reopen this issue?
We have observed this issue occurring in Net::Amazon::EC2 0.36 on ActivePerl-5.26 and ActivePerl-5.24
Oddly enough the issue doesn't manifest when using Perl v5.16.3 installed by default in Centos 7 (with Net::Amazon::EC2 force installed via CPAN)
@tavisma I gave up on getting this solved in the Perl stack, which is not supported by AWS. I believe you happen to experiencing this via ec2-consistent-snapshot. I rewrote that tool in Bash, which in turn uses the officially supported aws CLI tool.
https://github.com/RideAmigosCorp/ec2-consistent-snapshot.sh
Thanks, i'll check it out