check-cloudwatch-metric.rb is stuck at AWS/EC2 namespace. No other metrics available

Question

check-cloudwatch-metric.rb is stuck at AWS/EC2 namespace. No other metrics available

ay-b opened this issue 7 years ago · comments

Alexander Yankovskiy-Betcher commented 7 years ago

Hi.
check-cloudwatch-metric.rb seems bugged with sticking to AWS/EC2 namespace so no other metrics are available to be monitored.
Attention to namespaces, I've marked them with bold.

This works:

$ check-cloudwatch-metric.rb -r us-east-1 -n AWS/EC2 -m CPUUtilization -d ImageId=ami-275dde31 -p 60 -w 70 -c 90
CloudWatchMetricCheck OK: AWS/EC2-CPUUtilization(ImageId=ami-275dde31) is 5.29: comparison=greater, will alarm at 70.0

But this — don't:

$ check-cloudwatch-metric.rb -r us-east-1 -n AWS/ApplicationELB -m HTTPCode_ELB_5XX_Count -d LoadBalancer=app/my-balancer/3c8de7bacef1894f,AvailabilityZone=us-east-1b -p 60 -w 70 -c 90
CloudWatchMetricCheck OK: AWS/EC2-HTTPCode_ELB_5XX_Count(LoadBalancer=app/my-balancer/3c8de7bacef1894f&AvailabilityZone=us-east-1b) returned no data but that's ok

UPDATE

Definitely there is something wrong with namespaces:

$ check-cloudwatch-metric.rb -r us-east-1 -n AWS/ECS -m CPUReservation -d ClusterName=mydevcluster -p 60 -w 70 -c 90
CloudWatchMetricCheck OK: AWS/EC2-CPUReservation(ClusterName=mydevcluster) returned no data but that's ok

$ check-cloudwatch-metric.rb -r us-east-1 --namespace AWS/ECS -m CPUReservation -d ClusterName=mydevcluster -p 60 -w 70 -c 90
CloudWatchMetricCheck OK: AWS/ECS-CPUReservation(ClusterName=mydevcluster) is 0.0: comparison=greater, will alarm at 70.0

Also there are problems with dimensions: plugins returns an error if there are more than one key-pair.

$ check-cloudwatch-metric.rb -r us-east-1 --namespace AWS/ApplicationELB -m HTTPCode_ELB_5XX_Count -d LoadBalancer=app/my-balancer/3c8de7bacef1894f,AvailabilityZone=us-east-1b -p 60 -w 70 -c 90
CloudWatchMetricCheck UNKNOWN: AWS/ApplicationELB-HTTPCode_ELB_5XX_Count(LoadBalancer=app/my-balancer/3c8de7bacef1894f&AvailabilityZone=us-east-1b) could not be retrieved

Ben Abrams · Answer 1 · Mon Aug 07 2017 01:06:34 GMT+0800 (China Standard Time)

Well one thing that I see that is definitely wrong is that there are conflicting short options for -n:

Ben Abrams · Answer 2 · Mon Aug 07 2017 01:14:23 GMT+0800 (China Standard Time)

have you tried quoting the dimension to avoid shell interpolation?

Alexander Yankovskiy-Betcher · Answer 3 · Mon Aug 07 2017 01:48:31 GMT+0800 (China Standard Time)

have you tried quoting the dimension to avoid shell interpolation?

Yep. No luck. Can MacOS be a reason?

$ check-cloudwatch-metric.rb -r us-east-1 --namespace AWS/ApiGateway -m 5XXError -d "ApiName=Imported on 2017-03-31T23:45:21Z,Stage=Deployment" -p 60 -w 70 -c 90

CloudWatchMetricCheck UNKNOWN: AWS/ApiGateway-5XXError(ApiName=Imported on 2017-03-31T23:45:21Z&Stage=Deployment) could not be retrieved

Ben Abrams · Answer 4 · Mon Aug 07 2017 02:15:48 GMT+0800 (China Standard Time)

I don't think so, I looked at the code really quick and added a quick print statement to validate that it's being passed properly. I feel like it's more likely that the queries are not correct and the output is expected. I honestly don't use this check so I might be missing with how to use it properly but I cant get it to work with a single dimension with a classic ELB. Can you give me a working example of a classic ELB with a single dimension? Just trying to start from a place where it is working before assuming that I am using it correctly.

Here is what I tried:

$ bundle exec ./bin/check-cloudwatch-metric.rb -r us-west-2 --namespace AWS/ELB -m HTTPCode_Backend_5XX -d 'LoadBalancer=dev-nginx-upload' -c 10
[{:name=>"LoadBalancer", :value=>"dev-nginx-upload"}]
CloudWatchMetricCheck UNKNOWN: AWS/ELB-HTTPCode_Backend_5XX(LoadBalancer=dev-nginx-upload) could not be retrieved

Alexander Yankovskiy-Betcher · Answer 5 · Mon Aug 07 2017 03:48:54 GMT+0800 (China Standard Time)

No luck :(

$ check-cloudwatch-metric.rb -r us-east-1 --namespace AWS/ELB -d LoadBalancerName=awseb-e-a-AWSEBLoa -m EstimatedProcessedBytes -p 60 -w 900000000 -c 1800000000000
CloudWatchMetricCheck UNKNOWN: AWS/ELB-EstimatedProcessedBytes(LoadBalancerName=awseb-e-a-AWSEBLoa) could not be retrieved

Could plugin suppress autorization issues? May be there is something wrong with my config and actually request isn't authorized?
UPD: No, I'm in doubt about it. Made the same query using AWS CLI and it passes successfully.

Ben Abrams · Answer 6 · Tue Aug 08 2017 00:17:14 GMT+0800 (China Standard Time)

Could plugin suppress authorization issues?

https://github.com/sensu-plugins/sensu-plugins-aws/blob/6.3.0/lib/sensu-plugins-aws/cloudwatch-common.rb#L3 includes the auth stuff and I can assure that this works as I use that with other checks.

Alexander Yankovskiy-Betcher · Answer 7 · Tue Aug 08 2017 01:59:12 GMT+0800 (China Standard Time)

I figured out the auth problem — it was it. I suggest adding the following to the docs:

Requirements

Exact AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID variables must be explicitly set in the environment where plugin is running. Otherwise expect error:
«CloudWatchMetricCheck UNKNOWN: [your settings data] could not be retrieved»

Ben Abrams · Answer 8 · Wed Aug 09 2017 10:13:21 GMT+0800 (China Standard Time)

@ay-b hmm I would have expected the error you get to be like this:

$ bundle exec ./bin/check-cloudwatch-metric.rb -r us-east-1 --namespace AWS/RDS -m DatabaseConnections -d '' -p 60 -w 5 -c 2
Check failed to run: unable to sign request without credentials set, ["/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/request_signer.rb:104:in `require_credentials'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/request_signer.rb:94:in `sign_authenticated_requests'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/request_signer.rb:87:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/retry_errors.rb:89:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/query/handler.rb:27:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/seahorse/client/plugins/endpoint.rb:41:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/seahorse/client/request.rb:70:in `send_request'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.10.22/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'", "/home/babrams/projects/personal/sensu/sensu-plugins-aws/lib/sensu-plugins-aws/cloudwatch-common.rb:87:in `check'", "./bin/check-cloudwatch-metric.rb:121:in `run'", "/home/babrams/.rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/sensu-plugin-2.1.0/lib/sensu-plugin/cli.rb:58:in `block in <class:CLI>'"]

I really need to write up some documentation on how the auth works.

Ben Abrams · Answer 9 · Wed Aug 09 2017 11:26:11 GMT+0800 (China Standard Time)

Btw I played around it looks like possibly some metrics and dimensions work and others do not:

$ bundle exec ./bin/check-cloudwatch-metric.rb -r us-west-2 --namespace AWS/ELB -m BackendConnectionErrors -d 'AvailabilityZone=us-west-2a' -c 10
CloudWatchMetricCheck OK: AWS/ELB-BackendConnectionErrors(AvailabilityZone=us-west-2a) is 1.0: comparison=greater, will alarm at 10.0

$ bundle exec ./bin/check-cloudwatch-metric.rb -r us-west-2 --namespace AWS/ELB -m HTTPCode_Backend_5XX -d 'AvailabilityZone=us-west-2a' -c 10
CloudWatchMetricCheck UNKNOWN: AWS/ELB-HTTPCode_Backend_5XX(AvailabilityZone=us-west-2a) could not be retrieved

This is quite troubling indeed, we should test each of these combinations and see which ones work and which ones are broken: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/elb-metricscollected.html#loadbalancing-metrics-clb

Ben Abrams · Answer 10 · Wed Aug 09 2017 11:34:27 GMT+0800 (China Standard Time)

regarding the namespace issue I should have that fixed as part of #230

Ben Abrams · Answer 11 · Sun Sep 10 2017 02:33:25 GMT+0800 (China Standard Time)

@ay-b is there anything that is still outstanding here?

Ben Abrams · Answer 12 · Sun Oct 08 2017 00:58:28 GMT+0800 (China Standard Time)

closing due to inactivity