Giters

Netflix / SimianArmy

Tools for keeping your cloud operating in top form. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ELB cleanup causing AWS rate exceeded exception

ebukoski opened this issue · comments

commented

The Janitor Monkey ELB cleanup feature is encountering rate limit exceptions when running in AWS.

A quick solution could be to add a sleep when deleting ELBs. This will slow down Janitor but speed isn't that important during ELB cleanups. Of larger concern is impacting ELB API calls for the entire organization.

2016-11-15 18:59:46,650 ERROR [com.netflix.simianarmy.janitor.AbstractJanitor] Failed to clean up the resource obiwan-cseqa-frontend of type ELB.
com.amazonaws.services.elasticloadbalancing.model.AmazonElasticLoadBalancingException: Rate exceeded (Service: AmazonElasticLoadBalancing; Status Code: 400; Error Code: Throttling; Request ID: ad215c19-ab65-11e6-8f94-d765cc723562)
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1386)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:939)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:714)
        at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:465)
        at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:427)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:376)
        at com.amazonaws.services.elasticloadbalancing.AmazonElasticLoadBalancingClient.doInvoke(AmazonElasticLoadBalancingClient.java:2172)
        at com.amazonaws.services.elasticloadbalancing.AmazonElasticLoadBalancingClient.invoke(AmazonElasticLoadBalancingClient.java:2142)
        at com.amazonaws.services.elasticloadbalancing.AmazonElasticLoadBalancingClient.deleteLoadBalancer(AmazonElasticLoadBalancingClient.java:996)
        at com.netflix.simianarmy.client.aws.AWSClient.deleteElasticLoadBalancer(AWSClient.java:576)
        at com.netflix.simianarmy.aws.janitor.ELBJanitor.cleanup(ELBJanitor.java:58)
        at com.netflix.simianarmy.janitor.AbstractJanitor.cleanupResources(AbstractJanitor.java:336)
        at com.netflix.simianarmy.basic.janitor.BasicJanitorMonkey.doMonkeyBusiness(BasicJanitorMonkey.java:135)
        at com.netflix.simianarmy.nf.janitor.NFJanitorMonkey.doMonkeyBusiness(NFJanitorMonkey.java:79)
        at com.netflix.simianarmy.Monkey.run(Monkey.java:134)
        at com.netflix.simianarmy.nf.janitor.NFJanitorMonkey.run(NFJanitorMonkey.java:65)
        at com.netflix.simianarmy.Monkey$1.run(Monkey.java:155)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)