ashishmohite / terraform-aws-lambda

Terraform module which creates AWS Lambda resources

AWS Lambda Terraform module

Terraform module, which creates almost all supported AWS Lambda resources as well as taking care of building and packaging of required Lambda dependencies for functions and layers.

These types of resources supported:

• Lambda Function
• Lambda Layer
• Lambda Alias
• Lambda Provisioned Concurrency
• Lambda Async Event Configuration

Not supported, yet:

• Lambda Permission
• Lambda Event Source Mapping

This Terraform module is the part of serverless.tf framework, which aims to simplify all operations when working with the serverless in Terraform:

1. Build and package dependencies
2. Create and store deployment package
3. Create and update AWS Lambda Function and Lambda Layer
4. Publish and create aliases for AWS Lambda Function
5. and more

Features

• Build dependencies for your Lambda Function and Layer. Supports local builds, using Docker (with or without SSH agent support for private builds).
• Create deployment package or deploy existing (previously built package) from local, from S3, from URL.
• Store deployment packages locally or in the S3 bucket.
• Support almost all features of Lambda resources (function, layer, alias, etc.)
• Lambda@Edge
• Conditional creation for many types of resources.
• Control execution of nearly any step in the process - build, package, store package, deploy, update.
• Control nearly all aspects of Lambda resources (provisioned concurrency, VPC, dead-letter notification, tracing, and async events) and create of required IAM policies.

Usage

Lambda Function (store package locally)

module "lambda_function" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "my-lambda1"
  description   = "My awesome lambda function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  source_path = "../src/lambda-function1"

  tags = {
    Name = "my-lambda1"
  }
}

Lambda Function and Lambda Layer (store packages on S3)

module "lambda_function" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "lambda-with-layer"
  description   = "My awesome lambda function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"
  publish       = true

  source_path = "../src/lambda-function1"

  store_on_s3 = true
  s3_bucket   = "my-bucket-with-lambda-builds"

  layers = [
    module.lambda_layer_s3.this_lambda_layer_arn,
  ]

  environment_variables = {
    Serverless = "Terraform"
  }

  tags = {
    Module = "lambda-with-layer"
  }
}

module "lambda_layer_s3" {
  source = "terraform-aws-modules/lambda/aws"

  create_layer = true

  layer_name          = "lambda-layer-s3"
  description         = "My amazing lambda layer (deployed from S3)"
  compatible_runtimes = ["python3.8"]

  source_path = "../src/lambda-layer"

  store_on_s3 = true
  s3_bucket   = "my-bucket-with-lambda-builds"
}

Lambda Functions with existing package (prebuilt) stored locally

module "lambda_function_existing_package_local" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "my-lambda-existing-package-local"
  description   = "My awesome lambda function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  create_package         = false
  local_existing_package = "../existing_package.zip"
}

Lambda Function with existing package (prebuilt) stored in S3 bucket

module "lambda_function_existing_package_s3" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "my-lambda-existing-package-local"
  description   = "My awesome lambda function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  create_package      = false
  s3_existing_package = {
    bucket = "my-bucket-with-lambda-builds"
    key    = "existing_package.zip"
  }
}

Lambda Layers (store packages locally and on S3)

module "lambda_layer_local" {
  source = "terraform-aws-modules/lambda/aws"

  create_layer = true

  layer_name          = "my-layer-local"
  description         = "My amazing lambda layer (deployed from local)"
  compatible_runtimes = ["python3.8"]

  source_path = "../fixtures/python3.8-app1"
}

module "lambda_layer_s3" {
  source = "terraform-aws-modules/lambda/aws"

  create_layer = true

  layer_name          = "my-layer-s3"
  description         = "My amazing lambda layer (deployed from S3)"
  compatible_runtimes = ["python3.8"]

  source_path = "../fixtures/python3.8-app1"

  store_on_s3 = true
  s3_bucket   = "my-bucket-with-lambda-builds"
}

Lambda@Edge

Make sure, you deploy Lambda@Edge functions into US East (N. Virginia) region (us-east-1). See Requirements and Restrictions on Lambda Functions.

module "lambda_at_edge" {
  source = "terraform-aws-modules/lambda/aws"

  lambda_at_edge = true

  function_name = "my-lambda-at-edge"
  description   = "My awesome lambda@edge function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  source_path = "../fixtures/python3.8-app1"

  tags = {
    Module = "lambda-at-edge"
  }
}

Lambda Function in VPC

module "lambda_function_in_vpc" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "my-lambda-in-vpc"
  description   = "My awesome lambda function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  source_path = "../fixtures/python3.8-app1"

  vpc_subnet_ids         = module.vpc.intra_subnets
  vpc_security_group_ids = [module.vpc.default_security_group_id]
  attach_network_policy = true
}

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "my-vpc"
  cidr = "10.10.0.0/16"

  # Specify at least one of: intra_subnets, private_subnets, or public_subnets
  azs           = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  intra_subnets = ["10.10.101.0/24", "10.10.102.0/24", "10.10.103.0/24"]
}

Conditional creation

Sometimes you need to have a way to create resources conditionally but Terraform does not allow usage of count inside module block, so the solution is to specify create arguments.

module "lambda" {
  source = "terraform-aws-modules/lambda/aws"

  create = false # to disable all resources

  create_package  = false  # to control build package process
  create_function = false  # to control creation of the Lambda Function and related resources
  create_layer    = false  # to control creation of the Lambda Layer and related resources
  create_alias    = false  # to control creation of the Lambda Function Alias
  create_role     = false  # to control creation of the IAM role and policies required for Lambda Function

  attach_cloudwatch_logs_policy = false
  attach_dead_letter_policy     = false
  attach_network_policy         = false
  attach_tracing_policy         = false
  attach_async_event_policy     = false

  # ... omitted
}

Creation of deployment package

By default, this module creates deployment package and uses it to create or update Lambda Function or Lambda Layer.

Sometimes, you may want to separate build of deployment package (eg, to compile and install dependencies) from the deployment of a package into two separate steps.

When creating archive locally outside of this module you need to set create_package = false and then argument local_existing_package = "existing_package.zip". Alternatively, you may prefer to keep your deployment packages into S3 bucket and provide a reference to them like this:

  create_package      = false
  s3_existing_package = {
    bucket = "my-bucket-with-lambda-builds"
    key    = "existing_package.zip"
  }

Using deployment package from remote URL

This can be implemented in two steps: download file locally using CURL, and pass path to deployment package as local_existing_package argument.

locals {
  package_url = "https://raw.githubusercontent.com/terraform-aws-modules/terraform-aws-lambda/master/examples/fixtures/python3.8-zip/existing_package.zip"
  downloaded  = "downloaded_package_${md5(local.package_url)}.zip"
}

resource "null_resource" "download_package" {
  triggers = {
    downloaded = local.downloaded
  }

  provisioner "local-exec" {
    command = "curl -L -o ${local.downloaded} ${local.package_url}"
  }
}

data "null_data_source" "downloaded_package" {
  inputs = {
    id       = null_resource.download_package.id
    filename = local.downloaded
  }
}

module "lambda_function_existing_package_from_remote_url" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "my-lambda-existing-package-local"
  description   = "My awesome lambda function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  create_package         = false
  local_existing_package = data.null_data_source.downloaded_package.outputs["filename"]
}

How does building and packaging work?

This is one of the most complicated part done by the module and normally you don't have to know internals.

build.py is the script which does it. The main functions of the script are to generate a filename of zip-archive based on the content of the files, verify if zip-archive has been already created, and create the zip-archive only when it is necessary (during apply, not plan).

Hash of zip-archive created with the same content of the files is always identical which prevents unnecessary force-updates of the Lambda resources unless content modifies. If you need to have different filenames for the same content you can specify extra string argument hash_extra.

Build Dependencies

You can specify source_path in a variety of ways to specify path(s) and rules for building.

Note that files are not copying anywhere from the source directories when making packages, we use fast Python regular expressions to find matching files, which makes packaging very fast and easy to understand.

Simple build from single directory

When source_path is set to a path the content of that directory will be used as-is:

source_path = "src/function1"

Static build from multiple source directories

When source_path is set to a list of directories the content of each will be taken and one archive will be created.

Combine various options for extreme flexibility

This is the most complete way of creating a deployment package from multiple sources with multiple dependencies.

source_path = [
  "src/main-source",
  {
    path     = "src/function1-dep",
    patterns = [
      "!.*/.*\\.txt", # Skip all txt files recursively
    ]
  }, {
    path = "src/python3.8-app1",
    pip_requirements = true,     # Run "pip install" with requirements.txt in "path"
    prefix_in_zip = "foo/bar1"   # Put content from this target into "foo/bar" inside of deployment package
  }, {
    path = "src/python3.8-app2"
    pip_requirements = "requirements-large.txt" # Run "pip install" with this file
  }, {
    path = "src/python3.8-app3"
    commands = ["npm install"]   # Run these commands
    patterns = [
      "!.*/.*\\.txt", # Skip all txt files recursively
      "node_modules/.+", # Include all node_modules
    ],
  }, {
    path = "${path.module}/../fixtures/python3.8-app1"
    commands = ["npm install"]
    patterns = <<END
      !.*/.*\.txt
      node_modules/.*
      abc/def/.*
    END
  }, {
    path = "${path.module}/../fixtures/python3.8-app1"
    commands = ["npm install"]
    prefix_in_zip = "foo/bar",
    patterns = [".*"]  # default
  }

]

• prefix_in_zip - By default everything installs into the root of a zip-archive.
• patterns - Python regex (can support multiline heredoc). Default value is patterns = [".*"] which means "include everything". Some examples:

    !.*/.*\.txt        # Filter all txt files recursively
    node_modules/.*    # Include empty dir or with a content if it exists
    node_modules/.+    # Include full non empty node_modules dir with its content
    node_modules/      # Include node_modules itself without its content
                       # It's also a way to include an empty dir if it exists
    node_modules       # Include a file or an existing dir only

    !abc/.*            # Filter out everything in an abc folder
    abc/def/.*         # Re-include everything in abc/def sub folder
    !abc/def/hgk/.*    # Filter out again in abc/def/hgk sub folder

• commands - List of commands to run. If specified, this argument overrides pip_requirements.
• pip_requirements - Set to true to run pip install with requirements.txt found in path. Or set to another filename which you want to use instead.
• prefix_in_zip - If specified, will be used as a prefix inside zip-archive. By default, everything installs into the root of zip-archive.

Building in Docker

If your Lambda Function or Layer uses some dependencies you can build them in Docker and have them included into deployment package. Here is how you can do it:

build_in_docker   = true
docker_file       = "src/python3.8-app1/docker/Dockerfile"
docker_build_root = "src/python3.8-app1/docker"
docker_image      = "lambci/lambda:build-python3.8"
runtime           = "python3.8"    # Setting runtime is required when building package in Docker and Lambda Layer resource.

Using this module you can install dependencies from private hosts. To do this, you need for forward SSH agent:

docker_with_ssh_agent = true

Deployment package - Create or use existing

By default, this module creates deployment package and uses it to create or update Lambda Function or Lambda Layer.

Sometimes, you may want to separate build of deployment package (eg, to compile and install dependencies) from the deployment of a package into two separate steps.

When creating archive locally outside of this module you need to set create_package = false and then argument local_existing_package = "existing_package.zip". Alternatively, you may prefer to keep your deployment packages into S3 bucket and provide a reference to them like this:

  create_package      = false
  s3_existing_package = {
    bucket = "my-bucket-with-lambda-builds"
    key    = "existing_package.zip"
  }

Using deployment package from remote URL

This can be implemented in two steps: download file locally using CURL, and pass path to deployment package as local_existing_package argument.

locals {
  package_url = "https://raw.githubusercontent.com/terraform-aws-modules/terraform-aws-lambda/master/examples/fixtures/python3.8-zip/existing_package.zip"
  downloaded  = "downloaded_package_${md5(local.package_url)}.zip"
}

resource "null_resource" "download_package" {
  triggers = {
    downloaded = local.downloaded
  }

  provisioner "local-exec" {
    command = "curl -L -o ${local.downloaded} ${local.package_url}"
  }
}

data "null_data_source" "downloaded_package" {
  inputs = {
    id       = null_resource.download_package.id
    filename = local.downloaded
  }
}

module "lambda_function_existing_package_from_remote_url" {
  source = "terraform-aws-modules/lambda/aws"

  function_name = "my-lambda-existing-package-local"
  description   = "My awesome lambda function"
  handler       = "index.lambda_handler"
  runtime       = "python3.8"

  create_package         = false
  local_existing_package = data.null_data_source.downloaded_package.outputs["filename"]
}

FAQ

Q1: Why deployment package not recreating every time I change something?

A1: There can be tons of reasons. The most likely is that changes has happened inside of dependency which is not used in calculating content hash. You can force it by setting value of hash_extra or to delete previously built package locally.

Q2: How to force recreate deployment package?

A2: Delete an existing zip-archive from builds directory, or make a change in your source code. If there is no zip-archive for the current content hash, it will be recreated during terraform apply.

Notes

1. Creation of Lambda Functions and Lambda Layers is very similar and both support the same features (building from source path, using existing package, storing package locally or on S3)
2. Check out this Awesome list of AWS Lambda Layers

Examples

• Complete - Create Lambda resources in various combinations with all supported features
• Building and Packaging - Building and packaging deployment packages in various ways
• Async Invocations - Create Lambda Function with async event configuration (with SQS and SNS integration)
• With VPC - Create Lambda Function with VPC

Requirements

Name	Version
terraform	~> 0.12.6
aws	~> 2.46

Providers

Name	Version
aws	~> 2.46
external	n/a
null	n/a

Inputs

Name	Description	Type	Default	Required
alias_description	Description of the alias.	string	""	no
alias_function_name	The function ARN of the Lambda function for which you want to create an alias.	string	""	no
alias_function_version	Lambda function version for which you are creating the alias. Pattern: ($LATEST|[0-9]+).	string	""	no
alias_name	Name for the alias you are creating.	string	""	no
alias_routing_additional_version_weights	A map that defines the proportion of events that should be sent to different versions of a lambda function.	map(number)	{}	no
artifacts_dir	Directory name where artifacts should be stored	string	"builds"	no
attach_async_event_policy	Controls whether async event policy should be added to IAM role for Lambda Function	bool	false	no
attach_cloudwatch_logs_policy	Controls whether CloudWatch Logs policy should be added to IAM role for Lambda Function	bool	true	no
attach_dead_letter_policy	Controls whether SNS/SQS dead letter notification policy should be added to IAM role for Lambda Function	bool	false	no
attach_network_policy	Controls whether VPC/network policy should be added to IAM role for Lambda Function	bool	false	no
attach_tracing_policy	Controls whether X-Ray tracing policy should be added to IAM role for Lambda Function	bool	false	no
build_in_docker	Whether to build dependencies in Docker	bool	false	no
compatible_runtimes	A list of Runtimes this layer is compatible with. Up to 5 runtimes can be specified.	list(string)	[]	no
create	Controls whether resources should be created	bool	true	no
create_alias	Controls whether Lambda Alias resource should be created	bool	false	no
create_async_event_config	Controls whether async event configuration for Lambda Function/Alias should be created	bool	false	no
create_function	Controls whether Lambda Function resource should be created	bool	true	no
create_layer	Controls whether Lambda Layer resource should be created	bool	false	no
create_package	Controls whether Lambda package should be created	bool	true	no
create_role	Controls whether IAM role for Lambda Function should be created	bool	true	no
dead_letter_target_arn	The ARN of an SNS topic or SQS queue to notify when an invocation fails.	string	null	no
description	Description of your Lambda Function (or Layer)	string	""	no
destination_on_failure	Amazon Resource Name (ARN) of the destination resource for failed asynchronous invocations	string	null	no
destination_on_success	Amazon Resource Name (ARN) of the destination resource for successful asynchronous invocations	string	null	no
docker_build_root	Root dir where to build in Docker	string	""	no
docker_file	Path to a Dockerfile when building in Docker	string	""	no
docker_image	Docker image to use for the build	string	""	no
docker_with_ssh_agent	Whether to pass SSH_AUTH_SOCK into docker environment or not	bool	false	no
environment_variables	A map that defines environment variables for the Lambda Function.	map(string)	{}	no
function_name	A unique name for your Lambda Function	string	""	no
handler	Lambda Function entrypoint in your code	string	""	no
hash_extra	The string to add into hashing function. Useful when building same source path for different functions.	string	""	no
kms_key_arn	n/a	string	null	no
lambda_at_edge	Set this to true if using Lambda@Edge, to enable publishing, limit the timeout, and allow edgelambda.amazonaws.com to invoke the function	bool	false	no
lambda_role	IAM role attached to the Lambda Function. This governs both who / what can invoke your Lambda Function, as well as what resources our Lambda Function has access to. See Lambda Permission Model for more details.	string	""	no
layer_name	Name of Lambda Layer to create	string	""	no
layers	List of Lambda Layer Version ARNs (maximum of 5) to attach to your Lambda Function.	list(string)	null	no
license_info	License info for your Lambda Layer. Eg, MIT or full url of a license.	string	""	no
local_existing_package	The absolute path to an existing zip-file to use	string	null	no
maximum_event_age_in_seconds	Maximum age of a request that Lambda sends to a function for processing in seconds. Valid values between 60 and 21600.	number	null	no
maximum_retry_attempts	Maximum number of times to retry when the function returns an error. Valid values between 0 and 2. Defaults to 2.	number	null	no
memory_size	Amount of memory in MB your Lambda Function can use at runtime. Valid value between 128 MB to 3008 MB, in 64 MB increments.	number	128	no
policy	An additional policy to attach to the Lambda Function role	string	null	no
provisioned_concurrent_executions	Amount of capacity to allocate. Must be greater than or equal to 1.	number	-1	no
publish	Whether to publish creation/change as new Lambda Function Version.	bool	false	no
reserved_concurrent_executions	The amount of reserved concurrent executions for this Lambda Function. A value of 0 disables Lambda Function from being triggered and -1 removes any concurrency limitations. Defaults to Unreserved Concurrency Limits -1.	number	-1	no
role_description	Description of IAM role to use for Lambda Function	string	null	no
role_force_detach_policies	Specifies to force detaching any policies the IAM role has before destroying it.	bool	true	no
role_name	Name of IAM role to use for Lambda Function	string	null	no
role_path	Path of IAM role to use for Lambda Function	string	null	no
role_permissions_boundary	The ARN of the policy that is used to set the permissions boundary for the IAM role used by Lambda Function	string	null	no
role_tags	A map of tags to assign to IAM role	map(string)	{}	no
runtime	Lambda Function runtime	string	""	no
s3_bucket	S3 bucket to store artifacts	string	null	no
s3_existing_package	The S3 bucket object with keys bucket, key, version pointing to an existing zip-file to use	map(string)	null	no
s3_object_storage_class	Specifies the desired Storage Class for the artifact uploaded to S3. Can be either STANDARD, REDUCED_REDUNDANCY, ONEZONE_IA, INTELLIGENT_TIERING, or STANDARD_IA.	string	"ONEZONE_IA"	no
s3_object_tags	A map of tags to assign to S3 bucket object.	map(string)	{}	no
source_path	The absolute path to a local file or directory containing your Lambda source code	any	null	no
store_on_s3	Whether to store produced artifacts on S3 or locally.	bool	false	no
tags	A map of tags to assign to resources.	map(string)	{}	no
timeout	The amount of time your Lambda Function has to run in seconds.	number	3	no
tracing_mode	Tracing mode of the Lambda Function. Valid value can be either PassThrough or Active.	string	null	no
trusted_entities	Lambda Function additional trusted entities for assuming roles (trust relationship)	list(string)	[]	no
vpc_security_group_ids	List of security group ids when Lambda Function should run in the VPC.	list(string)	null	no
vpc_subnet_ids	List of subnet ids when Lambda Function should run in the VPC. Usually private or intra subnets.	list(string)	null	no

Outputs

Name	Description
lambda_role_arn	The ARN of the IAM role created for the Lambda Function
lambda_role_name	The name of the IAM role created for the Lambda Function
local_filename	The filename of zip archive deployed (if deployment was from local)
s3_object	The map with S3 object data of zip archive deployed (if deployment was from S3)
this_lambda_alias_arn	The ARN of the Lambda Function Alias
this_lambda_alias_invoke_arn	The ARN to be used for invoking Lambda Function from API Gateway
this_lambda_function_arn	The ARN of the Lambda Function
this_lambda_function_invoke_arn	The Invoke ARN of the Lambda Function
this_lambda_function_kms_key_arn	The ARN for the KMS encryption key of Lambda Function
this_lambda_function_last_modified	The date Lambda Function resource was last modified
this_lambda_function_name	The name of the Lambda Function
this_lambda_function_qualified_arn	The ARN identifying your Lambda Function Version
this_lambda_function_source_code_hash	Base64-encoded representation of raw SHA-256 sum of the zip file
this_lambda_function_source_code_size	The size in bytes of the function .zip file
this_lambda_function_version	Latest published version of Lambda Function
this_lambda_layer_arn	The ARN of the Lambda Layer with version
this_lambda_layer_created_date	The date Lambda Layer resource was created
this_lambda_layer_layer_arn	The ARN of the Lambda Layer without version
this_lambda_layer_source_code_size	The size in bytes of the Lambda Layer .zip file
this_lambda_layer_version	The Lambda Layer version

Authors

Module managed by Anton Babenko. Check out serverless.tf to learn more about doing serverless with Terraform.

License

Apache 2 Licensed. See LICENSE for full details.