Serverless Reference Architecture: IoT Backend
The Internet of Things (IoT) Backend reference architecture (diagram) demonstrates how to use AWS Lambda in conjunction with Amazon Kinesis, Amazon DynamoDB, Amazon Simple Storage Service (Amazon S3), and Amazon CloudWatch to build a serverless system for ingesting and processing sensor data. By leveraging these services, you can build cost-efficient applications that can meet the massive scale required for processing the data generated by huge deployments of connected devices.
This repository contains sample code for all the Lambda functions depicted in this diagram as well as a AWS CloudFormation template for creating the functions and related resources. There is also a simple webpage that you can run locally to publish sample events and query the data from DynamodDB.
Running the Example
The entire example system can be deployed in us-east-1 using the provided CloudFormation template and an S3 bucket. If you would like to deploy the template to a different region, you must copy the Lambda deployment packages under the iot-backend
prefix in the awslambda-reference-architectures
bucket to a new S3 bucket in your target region. You can then provide this new bucket as a parameter when launching the template.
Choose Launch Stack to launch the template in the us-east-1 region in your account:
Testing the Example
You can use the test webpage to test the system as follows:
- Save the testpage.html file to your local system.
- Open the downloaded file with a text editor and fill in the configuration values using the outputs of the CloudFormation stack launched in the previous section.
- Open your edited copy of testpage.html using a web browser of your choice.
After you launch the test page, you can simulate the submission of sensor data from multiple devices, as well as query the DynamoDB table for the historical data of a given device. In addition to using the test page to query DynamoDB, you can check the CloudWatch metrics published under the Sensor
namespace.
Cleaning Up the Example Resources
To remove all resources created by this example, do the following:
- Delete all objects from the
ArchiveBucket
created by the CloudFormation stack. - Delete the CloudFormation stack.
- Delete all CloudWatch log groups for each of the Lambda functions in the stack.
CloudFormation Template Resources
The following sections explain all of the resources created by the CloudFormation template provided with this example.
Lambda functions
-
ApiFunction - A Lambda function that provides a simple API for querying the sensor data stored in the
SensorDataTable
. -
DdbCloudWatchEventProcessorFunction - A Lambda function that processes events from
EventStream
and persists them to both theSensorDataTable
and a custom CloudWatch metric under theSensor
namespace. -
EventArchiverFunction - A Lambda function that processes events from
EventStream
and archives the raw data inArchiveBucket
.
Function roles
-
ApiExecutionRole - An AWS Identity and Access Management (IAM) role assumed by the
ApiFunction
. This role provides logging permissions and access to querySensorDataTable
. It also enables the function to callGetFunction
in order to read configuration data from the function's description. -
DdbCloudWatchProcessorRole - An IAM role assumed by the
DdbCloudWatchEventProcessorFunction
. This role provides permissions for logging, writing items toSensorDataTable
, and publishing custom CloudWatch metrics. It also enables the function to callGetFunction
in order to read configuration data from the function's description. -
EventArchiverRole - An IAM role assumed by the
EventArchiverRole
. This role provides logging permissions and access to put objects toArchiveBucket
. It also enables the function to callGetFunction
in order to read configuration data from the function's description.
Event source mappings
-
DdbCloudWatchProcessorSourceMapping - An event source mapping that enables
DdbCloudWatchEventProcessorFunction
to process records fromEventStream
. -
EventArchiverSourceMapping - An event source mapping that enables
EventArchiverFunction
to process records fromEventStream
.
IAM Users and Policies
-
TestClientUser - An IAM user used by the test webpage.
-
TestClientPolicy - An IAM policy attached to
TestClientUser
that grants access to put records on theEventStream
and invoke theApiFunction
. -
TestClientKeys - Access keys that enable the test webpage to sign API requests in order to simulate device events and query the
SensorDataTable
.
Other Resources
-
EventStream - An Amazon Kinesis stream to receive the raw sensor data.
-
SensorDataTable - A DynamoDB table to store the processed sensor data.
-
ArchiveBucket - An S3 Bucket for archiving the raw sensor data.
Configuration
-
ConfigTable - A DynamoDB table to hold configuration values read by the various Lambda functions. The name of this table, "IoTRefArchConfig", is hard coded into each function's code and cannot be modified without updating the code as well.
-
ConfigHelperStack - A sub-stack that creates a custom resource for writing entries to the
ConfigTable
. This stack creates a Lambda function and execution role that grants UpdateItem permission on theConfigTable
. -
SensorDataTableConfig - A configuration entry that identifies the
SensorDataTable
name. -
ArchiveBucketConfig - A configuration entry that identifies the
ArchiveBucket
name.
License
This reference architecture sample is licensed under Apache 2.0.