custom-resources: RedshiftData API call fails to find endpoint

Question

custom-resources: RedshiftData API call fails to find endpoint

rishiphatak opened this issue 2 months ago · comments

Describe the bug

I have a custom resource which calls the RedshiftData API executeStatement method. However, the lambda implementing the custom resource fails with the message Redshift endpoint doesn't exist in this region.

Expected Behavior

The RedshiftData API is actually called and executes the given SQL.

Current Behavior

We see this error upon deployment of the stack:

Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId: <request id here>)

Reproduction Steps

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import {AwsCustomResource, AwsCustomResourcePolicy, AwsSdkCall, PhysicalResourceId} from "aws-cdk-lib/custom-resources";
import {Duration} from "aws-cdk-lib";
import {ManagedPolicy, Role, ServicePrincipal} from "aws-cdk-lib/aws-iam";

export class CustomResourceTestAppStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const customResourceRole = new Role(this, 'customResourceRoleCdk', {
      assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
      roleName: 'customResourceRole',
      managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole')],
    });

    customResourceRole.addManagedPolicy(
        ManagedPolicy.fromAwsManagedPolicyName('AmazonRedshiftDataFullAccess'),
    );

    new AwsCustomResource(this, 'redshiftExecuteStmt', {
      resourceType: 'Custom::RedshiftExecuteStmt',
      onCreate: this.executeStmt(),
      onUpdate: this.executeStmt(),
      onDelete: this.executeStmt(),
      policy: AwsCustomResourcePolicy.fromSdkCalls({
        resources: AwsCustomResourcePolicy.ANY_RESOURCE,
      }),
      timeout: Duration.minutes(5),
      role: customResourceRole,
    });
  }
  private executeStmt(): AwsSdkCall {
    return {
      service: 'RedshiftData',
      action: 'executeStatement',
      parameters: {
        ClusterIdentifier: "my-redshift-cluster",
        Database: "MyRedshiftDatabase",
        DbUser: "testuser",
        Sql: "Create external schema hello_world_schema;",
      },
      physicalResourceId: PhysicalResourceId.of("physicalResource"),
    };
  }
}

Possible Solution

I think the cause of this bug is this change in the AWS JavaScript SDKv3: aws/aws-sdk-js-v3#5933

Additional Information/Context

No response

CDK CLI Version

2.145.0 (build fdf53ba)

Framework Version

No response

Node.js Version

v21.7.0

OS

macOS

Language

TypeScript

Language Version

Version 5.4.5

Other information

No response

Darshana Borade commented 2 months ago

+1

Anirudh Balaji commented 2 months ago

+1

Ashish Dhingra · Answer 1 · Fri Jun 14 2024 04:15:32 GMT+0800 (China Standard Time)

Reproducible. CloudFormation deployment gives the below error:

1:13:57 PM | CREATE_FAILED        | Custom::RedshiftExecuteStmt | redshiftExecuteStmt6566668B
Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId:
c9cf68cb-96dc-48f2-85dc-7b96cdc8e7ca)

1:14:01 PM | DELETE_FAILED        | Custom::RedshiftExecuteStmt | redshiftExecuteStmt6566668B
Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId:
b4c6f15f-c276-4d59-8bba-59ea15b9e0d6)


 ❌  Issue30536RedshiftStack failed: Error: The stack named Issue30536RedshiftStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_FAILED (The following resource(s) failed to delete: [redshiftExecuteStmt6566668B]. ): Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId: c9cf68cb-96dc-48f2-85dc-7b96cdc8e7ca), Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId: b4c6f15f-c276-4d59-8bba-59ea15b9e0d6)
    at FullCloudFormationDeployment.monitorDeployment (/usr/local/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:454:199515)
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:454:181237

 ❌ Deployment failed: Error: The stack named Issue30536RedshiftStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_FAILED (The following resource(s) failed to delete: [redshiftExecuteStmt6566668B]. ): Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId: c9cf68cb-96dc-48f2-85dc-7b96cdc8e7ca), Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId: b4c6f15f-c276-4d59-8bba-59ea15b9e0d6)
    at FullCloudFormationDeployment.monitorDeployment (/usr/local/lib/node_modules/aws-cdk/lib/index.js:451:10568)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/usr/local/lib/node_modules/aws-cdk/lib/index.js:454:199515)
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:454:181237

The stack named Issue30536RedshiftStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_FAILED (The following resource(s) failed to delete: [redshiftExecuteStmt6566668B]. ): Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId: c9cf68cb-96dc-48f2-85dc-7b96cdc8e7ca), Received response status [FAILED] from custom resource. Message returned: Redshift endpoint doesn't exist in this region. (RequestId: b4c6f15f-c276-4d59-8bba-59ea15b9e0d6)

Pahud Hsieh · Answer 2 · Fri Jun 14 2024 08:23:58 GMT+0800 (China Standard Time)

I didn't see how you created your provisioned redshift cluster or redshift serverless so it is very likely your custom resource just can't connect to it correctly. I just figured out a working sample with redshift serverless FYR:

export class CustomResourceTestAppStack extends Stack {
  readonly wg: redshiftserverless.CfnWorkgroup;
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const customResourceRole = new iam.Role(this, 'CrRole', {
      assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
      managedPolicies: [iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonRedshiftDataFullAccess')],
    });
    customResourceRole.addToPolicy(new iam.PolicyStatement({
      actions: ['redshift-serverless:*'],
      resources: ['*'],
    }));
    // create a redshift serverless namespace
    const ns = new redshiftserverless.CfnNamespace(this, 'NS', {
      namespaceName: `${Stack.of(this).stackName}-ns`,
      dbName: 'MyRedshiftDatabase',
    });

    // create a redshift serverless workgroup
    this.wg = new redshiftserverless.CfnWorkgroup(this, 'WG', {
      workgroupName: `${Stack.of(this).stackName}-wg`,
      publiclyAccessible: true,
      namespaceName: ns.namespaceName,
    });

    new CfnOutput(this, 'WorkGroupName', { value: this.wg.attrWorkgroupWorkgroupName });
    
    this.wg.addDependency(ns);

    const testCR = new cr.AwsCustomResource(this, 'redshiftExecuteStmt', {
      resourceType: 'Custom::RedshiftExecuteStmt',
      onCreate: this.executeStmt(),
      onUpdate: this.executeStmt(),
      onDelete: this.executeStmt(),
      role: customResourceRole,
      timeout: Duration.minutes(5),
    });

    // only execute the statement after wg is created
    testCR.node.addDependency(this.wg);

    const dbUser = testCR.getResponseField('DbUser');
    const database = testCR.getResponseField('Database');
    
    new CfnOutput(this, 'DbUser', { value: dbUser });
    new CfnOutput(this, 'Database', { value: database });
  }
  private executeStmt(): cr.AwsSdkCall {
    return {
      service: 'RedshiftData',
      action: 'executeStatement',
      parameters: {
        WorkgroupName: this.wg.workgroupName,
        Database: "MyRedshiftDatabase",
        Sql: "SHOW TABLES;",
      },
      physicalResourceId: cr.PhysicalResourceId.of("physicalResource"),
    };
  }
}

cdk deploy

Outputs:
dummy-stack7.Database = MyRedshiftDatabase
dummy-stack7.DbUser = IAMR:dummy-stack7-CrRole22B01441-FnWKISKf3Qu2
dummy-stack7.WorkGroupName = dummy-stack7-wg

verify using CLI

 % aws redshift-data execute-statement \
  --region us-east-1 \
  --workgroup-name "dummy-stack7-wg" \
  --database "MyRedshiftDatabase" \
  --sql "SHOW TABLES;"
{
    "CreatedAt": "2024-06-13T20:21:13.111000-04:00",
    "Database": "MyRedshiftDatabase",
    "DbUser": "IAMR:AWSReservedSSO_AdministratorAccess_c4188fabc1dd35a2",
    "Id": "d2444cd9-8d9b-4232-bb3a-d92f3f6fa7ed",
    "WorkgroupName": "dummy-stack7-wg"
}

Verified!

Tips:

Do not create the custom resource in the beginning, just create the redshift serverless or cluster first.
Make sure you can connect and execute the statement using data API with AWS CLI. This ensures your endpoint is working. After that, start authoring your CDK code.
Ensure the dependency: NS->WG->CR. Use addDependency to ensure their dependencies.
Try build your custom resource in addition to your redshift code in CDK.

Let me know if it works for you.

github-actions · Answer 3 · Sun Jun 16 2024 12:04:49 GMT+0800 (China Standard Time)

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

rishiphatak · Answer 4 · Wed Jun 19 2024 08:20:58 GMT+0800 (China Standard Time)

Hi Pahud,

Thanks for your quick reply. The suggested solution worked! Closing this issue.

Best,
Rishi

github-actions · Answer 5 · Wed Jun 19 2024 08:21:22 GMT+0800 (China Standard Time)

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

AWS CDK Automation · Answer 6 · Fri Jul 26 2024 01:46:37 GMT+0800 (China Standard Time)

Comments on closed issues and PRs are hard for our team to see. If you need help, please open a new issue that references this one.