Cross region SNS trigger

Question

Cross region SNS trigger

matthisk opened this issue 7 years ago · comments

Matthisk Heimensen commented 7 years ago

This is a (Bug Report)

Description

When trying to deploy a lambda with an SNS trigger in a different region the cloudformation script fails.

What went wrong?

Cloudformation fails to create SNS event subscription for lambda with error:

invalid parameter: TopicArn

What did you expect should have happened?

Expected the lambda to deploy (via cloudformation) with a trigger on an SNS topic in a different region.

What was the config you used?

service: testservice
provider:
  name: aws
  runtime: nodejs6.10
  region: us-west-1

functions:
  hello:
    handler: handler.run
    events: 
      - sns: arn:aws:sns:eu-west-1:xxxxxxxxxxx:stresstest

What stacktrace or error message from your provider did you see?

Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (2.41 MB)...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - stresser-dev
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - HelloLogGroup
CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - HelloLogGroup
CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - HelloLogGroup
CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - HelloLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - HelloLambdaFunction
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - HelloLambdaFunction
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - HelloLambdaPermissionStresstestSNS
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - HelloLambdaVersionOVTh8yMAX1zsFvyQcL4UekM3pyWBiYTwE71y8M
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - HelloLambdaPermissionStresstestSNS
CloudFormation - CREATE_IN_PROGRESS - AWS::SNS::Subscription - HelloSnsSubscriptionStresstest
CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - HelloLambdaVersionOVTh8yMAX1zsFvyQcL4UekM3pyWBiYTwE71y8M
CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - HelloLambdaVersionOVTh8yMAX1zsFvyQcL4UekM3pyWBiYTwE71y8M
CloudFormation - CREATE_FAILED - AWS::SNS::Subscription - HelloSnsSubscriptionStresstest
CloudFormation - CREATE_FAILED - AWS::Lambda::Permission - HelloLambdaPermissionStresstestSNS
CloudFormation - UPDATE_ROLLBACK_IN_PROGRESS - AWS::CloudFormation::Stack - stresser-dev
CloudFormation - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - stresser-dev
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Permission - HelloLambdaPermissionStresstestSNS
CloudFormation - DELETE_COMPLETE - AWS::SNS::Subscription - HelloSnsSubscriptionStresstest
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Version - HelloLambdaVersionOVTh8yMAX1zsFvyQcL4UekM3pyWBiYTwE71y8M
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Permission - HelloLambdaPermissionStresstestSNS
CloudFormation - DELETE_IN_PROGRESS - AWS::Lambda::Function - HelloLambdaFunction
CloudFormation - DELETE_COMPLETE - AWS::Lambda::Function - HelloLambdaFunction
CloudFormation - DELETE_IN_PROGRESS - AWS::Logs::LogGroup - HelloLogGroup
CloudFormation - DELETE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - DELETE_COMPLETE - AWS::Logs::LogGroup - HelloLogGroup
CloudFormation - DELETE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution
CloudFormation - UPDATE_ROLLBACK_COMPLETE - AWS::CloudFormation::Stack - stresser-dev

Similar or dependent issues:

#2183

Additional Data

  Your Environment Information -----------------------------
     OS:                 darwin
     Node Version:       7.10.0
     Serverless Version: 1.14.0

Philipp Muens · Answer 1 · Thu May 25 2017 02:33:58 GMT+0800 (China Standard Time)

Thanks for opening @matthisk 👍

🤔 right now Serverless assumes that everything will be deployed into one region. @eahefnawy do you have any idea if there's a possible solution / workaround available for this?

orasis · Answer 2 · Wed May 31 2017 10:49:05 GMT+0800 (China Standard Time)

I have the same problem. I need to subscribe to an us-east-1 ARN for the AWS marketplace but my whole infrastructure runs in us-west-2

Jeremy Thomerson · Answer 3 · Tue Jul 11 2017 03:12:21 GMT+0800 (China Standard Time)

@pmuens I don't think this is actually a Serverless issue. I looked into this once before and IIRC it is purely a CloudFormation issue. That seems to also be indicated by the error coming back from CloudFormation. I think the issue will need to be raised with AWS directly.

Jeremy Thomerson · Answer 4 · Tue Jul 11 2017 03:17:47 GMT+0800 (China Standard Time)

Interestingly, even when I manually subscribed a Lambda function in a different region than the SNS topic, I had weird behavior. Here's what happened:

$ aws sns subscribe \
   --topic-arn arn:aws:sns:us-east-1:123456789012:TOPIC_NAME \
   --protocol lambda \
   --notification-endpoint arn:aws:lambda:us-west-2:987654321098:function:function-name

# That worked:
{
    "SubscriptionArn": "arn:aws:sns:us-east-1:123456789012:TOPIC_NAME:c9b553d2-3d77-4a99-ad22-52bdb744bc06"
}

When I went to the SNS topic in the web console, I could see the cross-account, cross-region subscription.

But then when I went to the Lambda function in the web console, it showed me this:

Do we even know for sure if AWS supports cross-region SNS->Lambda subscriptions? I could not find any documentation saying they did.

Philipp Muens · Answer 5 · Tue Jul 11 2017 14:08:01 GMT+0800 (China Standard Time)

Interesting 🤔 Thanks for the update and investigation on this @jthomerson 👍

Do we even know for sure if AWS supports cross-region SNS->Lambda subscriptions?

Not entirely sure about that. The first time I encountered it was this issue.

Deleted user · Answer 6 · Wed Aug 16 2017 01:38:22 GMT+0800 (China Standard Time)

@jthomerson
Q: Do my AWS Lambda functions need to be in the same region as my Amazon SNS usage?
You can subscribe your AWS Lambda functions to an Amazon SNS topic in any region.
https://aws.amazon.com/sns/faqs/

We need this on our project. What are the next steps for follow up with AWS? I'm sure you guys can pull more strings than we can to get this worked out with CloudFormation

Jeremy Thomerson · Answer 7 · Wed Aug 16 2017 01:42:06 GMT+0800 (China Standard Time)

@srg-avai Yes, you can subscribe cross-region like the FAQ says. However, you can not do it (or could not at the time that I wrote that post) via CloudFormation. Thus, I ended up having to make a custom resource that did the cross-region subscription, and using that custom CloudFormation resource rather than using the built-in CF SNS subscription resource. Obviously, that's not an option for the SLS team (since we'd all have to deploy custom resources to use SLS), but maybe their connections / weight with AWS could get the issue noticed if someone on their team reported it to AWS.

Deleted user · Answer 8 · Wed Aug 16 2017 01:45:56 GMT+0800 (China Standard Time)

@jthomerson thank you, do you mind providing some more detail? i'm interested in the custom resource approach... could you share an example to head start me (and anyone in the future with this prob) on this?

Deleted user · Answer 9 · Wed Aug 16 2017 07:07:36 GMT+0800 (China Standard Time)

fwiw anyone else reading this... the approach jthomerson describes above is as follows:

implement a custom resource - use the resources example on the serverless docs and then search AWS docs for custom resources
the custom resource is a lambda which cloud formation calls mid-deployment, you do what you need to do in the lambda (e.g. subscribe a lambda to a topic in another region), and then you post to an endpoint (provided by cloudfront on the request) when you're done - with a success or failure property (all described in AWS custom resources docs)

John Scattergood · Answer 10 · Sat Oct 14 2017 03:22:38 GMT+0800 (China Standard Time)

I have created a Gist showing how to implement the custom resource with SLS:

https://gist.github.com/jscattergood/00a2ea6a80fe41a74c5c7efed1b238b4

Adil Hindistan · Answer 11 · Sat Oct 13 2018 02:11:57 GMT+0800 (China Standard Time)

This is now supported in cloudformation, however, you have to add target region within cfn.
Example:
` SNSSampleSubscription:
Type: AWS::SNS::Subscription

Properties:
  Endpoint: !Sub 'arn:aws:lambda:us-east-2:2222222:function:sample_lambda'
  Protocol: lambda
  TopicArn: !Sub 'arn:aws:sns:us-east-1:1111111:osbf-pull-ami'
  Region: !Sub 'us-east-1'`

In above example, lambda is in us-east-2, and it is subscribing to an sns in us-east-1. The "Region" allows you to do that.

Imran · Answer 12 · Tue Oct 16 2018 09:10:38 GMT+0800 (China Standard Time)

This is now supported in cloudformation, however, you have to add target region within cfn.

I was able to modify my serverless.yml with the Region parameter and I can see that the subscription is created successfully against the correct topic.

serverless.yml

resources:
  Resources
    SubNameFromGeneratedCfn:
      Properties:
        Region: eu-west-1

I also had to ensure that serverless-psuedo-parameters didn't auto replace the region so just a heads up if you are using that:

custom:
  pseudoParameters:
    skipRegionReplace: true

Frank zomerdijk · Answer 13 · Thu Nov 08 2018 17:30:59 GMT+0800 (China Standard Time)

Hi,
I can confirm the "Region" option is working, although I have not seen this option in the documentation.

My "LambdaFunction" is created in a EU region and I am subscribing tot the "AmazonIpSpaceChanged" SNS notification in "us-east-1"

See CFN json sniped below.

Regards,
Frank

`
"LambdaAmazonIpSpaceChangedSubscription" : {

  "Type" : "AWS::SNS::Subscription",

  "Properties" : {

    "Endpoint" : {"Fn::GetAtt" : ["LambdaFunction", "Arn"] },

    "Protocol" : "lambda",

    "TopicArn" : "arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged",

    "Region": "us-east-1"

  }

}

`

Imran · Answer 14 · Fri Nov 09 2018 14:28:37 GMT+0800 (China Standard Time)

Whilst there is now a workaround available it would be great if there was first class support for this on the sns event in serverless.

prernab · Answer 15 · Fri Jan 04 2019 15:03:34 GMT+0800 (China Standard Time)

The sns topic is in the region eu-west-1 (not created using cloud formation) so permission is added manually in topic policy and lambda is in different region us-west-2 and using cloud formation. How to provide permission in this case? As I am getting same error "invalid parameter: TopicArn".

{
"Sid": "Subscribe-to-topic",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxx:root"
},
"Action": "sns:Subscribe",
"Resource": "arn:aws:sns:eu-west-1:xxxxx:publish-notifier"
}

ZhangWei · Answer 16 · Thu Apr 11 2019 11:58:46 GMT+0800 (China Standard Time)

@Imran99 Could you please share you serverless.yml ?

patrusso2 · Answer 17 · Wed May 22 2019 21:53:28 GMT+0800 (China Standard Time)

Hey all - sorry if this is just me being dense, but this took me a while to figure out and I want to add a couple clarifications in case they help out others who run into this issue.

My lambda functions are in us-east-2, but I wanted to use a SNS queue in us-east-1 as a trigger.

I originally tried this in my serverless.yml file, but I got the "invalid parameter: TopicArn" error (like the original poster).

handleBounceOrComplaintEmailSNS:
  handler: server.handleBounceOrComplaintEmailSNS
  events:
    - sns:
        arn: arn:aws:sns:us-east-1:0005555555555:email-bounces-complaints

I didn't know enough about serverless or cloudformation to know what @Imran99 meant by SubNameFromGeneratedCfn, so it took a lot of head scratching and searching.
After staring at my .serverless/cloudformation-template-update-stack.json for a while (and several failed attempts), I finally sorted it out. By searching for my SNS's ARN in .serverless/cloudformation-template-update-stack.json, I found that the SubNameFromGeneratedCfn Imran99 was referring to was HandleBounceOrComplaintEmailSNSSnsSubscriptionemailbouncescomplaints (in my template). By using that name under resources, I was able get it working! So these two components of my serverless.yml file are:

handleBounceOrComplaintEmailSNS:
  handler: server.handleBounceOrComplaintEmailSNS
  events:
    - sns:
        arn: arn:aws:sns:us-east-1:0005555555555:email-bounces-complaints

resources:  
  Resources:
    HandleBounceOrComplaintEmailSNSSnsSubscriptionemailbouncescomplaints:
      Type: AWS::SNS::Subscription
      Properties:
        Region: us-east-1

Bottom line: use Imran99's workaround, but note you'll need to search .serverless/cloudformation-template-update-stack.json to figure out SubNameFromGeneratedCfn. Thank you all for the workaround!

Mithun Ayachit · Answer 18 · Wed Jul 10 2019 01:42:10 GMT+0800 (China Standard Time)

Whilst there is now a workaround available it would be great if there was first class support for this on the sns event in serverless.

@dschep why was this issue closed? I do not see any messages from maintainers as to whether this issue will be addressed. There isn't a pending PR either.

devops hipster in training. · Answer 19 · Fri Jul 12 2019 09:45:18 GMT+0800 (China Standard Time)

I've raised PR #6366 that should add this functionality.

isaacList · Answer 20 · Tue Jul 30 2019 15:23:42 GMT+0800 (China Standard Time)

Hey all - sorry if this is just me being dense, but this took me a while to figure out and I want to add a couple clarifications in case they help out others who run into this issue.

My lambda functions are in us-east-2, but I wanted to use a SNS queue in us-east-1 as a trigger.

I originally tried this in my serverless.yml file, but I got the "invalid parameter: TopicArn" error (like the original poster).
handleBounceOrComplaintEmailSNS:
  handler: server.handleBounceOrComplaintEmailSNS
  events:
    - sns:
        arn: arn:aws:sns:us-east-1:0005555555555:email-bounces-complaints
I didn't know enough about serverless or cloudformation to know what @Imran99 meant by SubNameFromGeneratedCfn, so it took a lot of head scratching and searching.
After staring at my .serverless/cloudformation-template-update-stack.json for a while (and several failed attempts), I finally sorted it out. By searching for my SNS's ARN in .serverless/cloudformation-template-update-stack.json, I found that the SubNameFromGeneratedCfn Imran99 was referring to was HandleBounceOrComplaintEmailSNSSnsSubscriptionemailbouncescomplaints (in my template). By using that name under resources, I was able get it working! So these two components of my serverless.yml file are:
handleBounceOrComplaintEmailSNS:
  handler: server.handleBounceOrComplaintEmailSNS
  events:
    - sns:
        arn: arn:aws:sns:us-east-1:0005555555555:email-bounces-complaints

resources:  
  Resources:
    HandleBounceOrComplaintEmailSNSSnsSubscriptionemailbouncescomplaints:
      Type: AWS::SNS::Subscription
      Properties:
        Region: us-east-1
Bottom line: use Imran99's workaround, but note you'll need to search .serverless/cloudformation-template-update-stack.json to figure out SubNameFromGeneratedCfn. Thank you all for the workaround!

Why would I get the following error even if I follow your example? My lambda function is deployed to us-west-1

  Serverless Error ---------------------------------------
 
  An error occurred: HandleEmailBounceSnsSubscriptionsesbouncestopic - Property Protocol cannot be empty..

functions:
  handleEmailBounce:
    handler: handler.handleEmailBounce
    # notificationArn: ${self:provider.TopicArn}
    events:
      - sns: ${self:provider.TopicArn}  # arn:aws:sns:us-east-1:${self:provider.awsAccountId}:ses-bounces-topic

resources:
  Resources:
    HandleEmailBounceSnsSubscriptionsesbouncestopic:
      Type: AWS::SNS::Subscription
      Properties:
        Region: us-east-1
       # TopicArn: arn of lambda
       # Protocol: lambda

Even if you add TopicArn and Protocol properties, I'm still getting errors....

devops hipster in training. · Answer 21 · Tue Jul 30 2019 17:08:42 GMT+0800 (China Standard Time)

@isaacList You no longer need the cloudformation override (I added this in my PR above). Can you try again with the latest version of serverless without the resources override.

isaacList · Answer 22 · Tue Jul 30 2019 18:15:48 GMT+0800 (China Standard Time)

@herebebogans Yeah I have updated to the latest version. I can successfully deploy the application without any errors, but I can't find SNS trigger when I go to AWS lambda trigger or find Subscription in the SNS console of us-east-1 region..... The AWS account is the same

devops hipster in training. · Answer 23 · Tue Jul 30 2019 20:01:57 GMT+0800 (China Standard Time)

Try the longer form for the event

    events:
      - sns: 
          arn: ${self:provider.TopicArn}

I'll look into why the short form is not adding it.

isaacList · Answer 24 · Tue Jul 30 2019 21:11:11 GMT+0800 (China Standard Time)

@herebebogans I am exactly this longer form... But I can't see SNS trigger in Lambda console...

devops hipster in training. · Answer 25 · Wed Jul 31 2019 07:56:47 GMT+0800 (China Standard Time)

@isaacList Can you post the relevant snippets from serverless.yml .. eg your provider section + the function definition?

isaacList · Answer 26 · Wed Jul 31 2019 10:12:48 GMT+0800 (China Standard Time)

@herebebogans Here you are....

service:
  name: email-bounce-receiver

provider:
  name: aws
  config: ${file(serverless/${env:NODE_ENV}.yml)}
  vpcSettings: ${file(vpc_settings.yml)}
  vpc: ${self:provider.vpcSettings.vpc} #securityGroupIds: [xxx] subnetIds: [xxx]
  runtime: provided
  stage: ${env:NODE_ENV} #dev
  awsAccountId: ${self:provider.config.awsAccountId} 
  region: ${self:provider.config.region} # us-west-1
  stack: ${self:provider.config.stack} #somestring
  TopicArn: arn:aws:sns:us-east-1:${self:provider.awsAccountId}:ses-bounces-topic

package:
  artifact: ./artifact.zip

functions:
  handleEmailBounce:
    handler: handler.handleEmailBounce
    events:
      - sns:
        arn: ${self:provider.TopicArn}

    layers:
      - arn:aws:lambda:${self:provider.region}:${self:provider.awsAccountId}:layer:nodejs-10:1

    environment:
      NODE_ENV: ${env:NODE_ENV}
      NODE_APP_INSTANCE: ${env:NODE_APP_INSTANCE}

    vpc: ${self:provider.vpc}
    reservedConcurrency: 1

devops hipster in training. · Answer 27 · Wed Jul 31 2019 10:27:58 GMT+0800 (China Standard Time)

Indent arn in the event two more spaces :-)

isaacList · Answer 28 · Wed Jul 31 2019 11:22:24 GMT+0800 (China Standard Time)

Ok You have corrected the sample code. It works perfectly now. Thanks a lot. 😆

Try the longer form for the event
    events:
      - sns: 
        arn: ${self:provider.TopicArn}
I'll look into why the short form is not adding it.

Haiyang Zhao · Answer 29 · Wed Nov 20 2019 18:11:28 GMT+0800 (China Standard Time)

Service	Account	Region
Lambda	Account1	us-west-1
SNS	Account2	us-east-1

In Account1 we subscribe SNS in region us-west-1, it will fail with error "Error code: InvalidParameter - Error message: Invalid parameter: TopicArn"

We need to switch Account1 to region us-east-1 to do the subscription, it will success.

eric-dorsey · Answer 30 · Thu Mar 16 2023 22:24:32 GMT+0800 (China Standard Time)

Seems like this other workaround of adding a topicName (and the topicName doesn't even have to match the actual topic name), which was used for a different problem (resources with the same name - though in our use case we did have SNS topics with the same name across different regions), also resolves not being able to subscribe to SNS topics in other regions (ie, always getting the invalid parameter: TopicArn output):

Repeating working snippet from #4742 (comment) here in case it helps anyone since this issue #3676 seems to come up when searching this problem:

functions:
  SomeFunction:
    handler: main.lambda_handler
    events:
      - sns:
          arn: arn:aws:sns:us-east-1:123456789012:SomeSNSTopic
          topicName: us-east-1
      - sns:
          arn: arn:aws:sns:us-east-2:123456789012:SomeSNSTopic
          topicName: us-east-2