alexcasalboni / aws-lambda-power-tuning

AWS Lambda Power Tuning is an open-source tool that can help you visualize and fine-tune the memory/power configuration of Lambda functions. It runs in your own AWS account - powered by AWS Step Functions - and it supports three optimization strategies: cost, speed, and balanced.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Timeout when running

rcoundon opened this issue · comments

I'm trying to tune a lambda that is usually invoked from API Gateway. The Power Tuning application is deployed to the same region as the lambda but when I call it I'm getting a timeout error:

{
  "lambdaARN": "arn:aws:lambda:eu-west-2:68xxxx66:function:myFunctionArn",
  "powerValues": [
    128,
    256,
    512,
    1024,
    2048,
    3008
  ],
  "num": 10,
  "payload": {
    "event": {
      "queryStringParameters": {
        "token": "someValue",
        "contactCentre": true
      }
    }
  },
  "parallelInvocation": true,
  "strategy": "cost",
  "error": {
    "Error": "Error",
    "Cause": "{\"errorType\":\"Error\",\"errorMessage\":\"Invocation error (running in parallel): {\\\"errorMessage\\\":\\\"2021-12-20T20:22:43.783Z 4607c3a0-0c91-4ec2-8f13-a5bfb815853d Task timed out after 10.01 seconds\\\"} with payload \\\"{\\\\\\\"event\\\\\\\":{\\\\\\\"queryStringParameters\\\\\\\":{\\\\\\\"token\\\\\\\":\\\\\\\"someValue\\\\\\\",\\\\\\\"contactCentre\\\\\\\":true}}}\\\"\",\"trace\":[\"Error: Invocation error (running in parallel): {\\\"errorMessage\\\":\\\"2021-12-20T20:22:43.783Z 4607c3a0-0c91-4ec2-8f13-a5bfb815853d Task timed out after 10.01 seconds\\\"} with payload \\\"{\\\\\\\"event\\\\\\\":{\\\\\\\"queryStringParameters\\\\\\\":{\\\\\\\"token\\\\\\\":\\\\\\\"someValue\\\\\\\",\\\\\\\"contactCentre\\\\\\\":true}}}\\\"\",\"    at /var/task/executor.js:93:19\",\"    at processTicksAndRejections (internal/process/task_queues.js:95:5)\",\"    at async Promise.all (index 2)\",\"    at async runInParallel (/var/task/executor.js:98:5)\",\"    at async Runtime.module.exports.handler (/var/task/executor.js:49:19)\"]}"
  }
}

Any thoughts on what could be going on?

I should say, when executed from an API Gateway request, this lambda typically takes less than 1s to complete

I've also tried with the input as follows with the same result:

{
    "lambdaARN": "arn:aws:lambda:eu-west-2:xxxxxxx:function:thefunctionarn",
    "powerValues": [128, 256, 512, 1024, 2048, 3008],
    "num": 10,
    "payload": {
        "queryStringParameters": {
            "token": "someValue",
            "contactCentre": true
         }
    },
    "parallelInvocation": true,
    "strategy": "cost"
}

I've also tried "parallelInvocation": false but the same issue occurs

Hi @rcoundon 👋 Thanks for reporting this!

Looking at the error, it sounds like the invocation payload is causing the handler to timeout. Let's try to figure out why :)

Have you tried testing the function in isolation locally and in the Lambda console with the same input?

Screenshot 2021-12-21 at 10 09 18

I'd recommend making sure that the input payload you're using for power-tuning works fine with your current implementation. It's possible that your function is expecting a different payload (with more API Gateway stuff?) and that the payload you're using isn't enough.

I mean this:

{
    "queryStringParameters": {
        "token": "someValue",
        "contactCentre": true
    }
}

Maybe your function expects additional routing parameters? Or headers?

Also, are you using callbacks or async/await? If your function is implemented in Node.js with callbacks, make sure that you're resolving all promises. If you don't, the invocation might time out.

Let me know when you manage to test the function in isolation both locally and in the Lambda console. If it runs fine with the same input payload, we'll have to dig deeper :)

Hi there - thanks for getting back to me.

If I try to run this using the console, everything works fine using payload:

{
  "queryStringParameters":{
      "token": "someVal",
      "contactCentre": "true"
  }
}

However, I think I'm going to redeploy the power tuning application as while debugging last night it looks like I've messed up the privileges as it now fails for a different reason:

User: arn:aws:sts::xxxxxx:assumed-role/xxxx is not authorized to perform: lambda:GetFunctionConfiguration on resource: arn:aws:lambda:eu-west-2:681524250866:function:someFunction because no identity-based policy allows the lambda:GetFunctionConfiguration action

We use async/await. This is a function that's been running fine for a while in production so we know it works in terms of resolving promises etc.

I've redeployed, ran exactly the same payload and it's now working...
No idea what it was, gotta hate those!

Thanks for sharing, I'm glad the issue is resolved now.