AWS IoT Greengrass V2 Community Component - The Things Stack LoRaWAN

The Things Stack is an open-source LoRaWAN stack. Users can deploy the stack as Open Source (free) or Enterprise (requires a license) from images available on Docker Hub.

This repository facilitates packaging The Things Stack into an AWS IoT Greengrass V2 component named aws.greengrass.labs.TheThingsStackLoRaWAN. This component enables customer use cases that require a private LoRaWAN Network Server (LNS) at the edge. For Enterprise deployments, ingestion into AWS IoT Core can then be achieved by deploying The Things Stack AWS IoT default integration into your AWS account.

The Things Stack is also offered as an AWS Marketplace AMI. This Greengrass component is a logical equivalent, but pitching at the edge.

Architecture
Repository Contents
Requirements and Prerequisites
Getting Started
- Quickstart
- Slowstart
The Things Stack Configuration Tips
Development
- Static Analysis
- Unit Tests

Architecture

An overview of the system architecture is presented below.

The aws.greengrass.labs.TheThingsStackLoRaWAN component is a thin wrapper around a conventional The Things Stack deployment. The Things Stack is composed of multiple Docker containers, using Docker Compose. Subject to configuration, there may be a Redis container and either a PostreSQL or CockroachDB container, in addition to The Things Stack container.

The Things Stack is delivered as Docker images on Docker Hub. This component downloads Docker images from Docker Hub with the help of the Docker application manager managed component.

The Things Stack configuration includes a considerable amount of sensitive information. Accordingly, the configuration is stored as a secret in AWS Secrets Manager. This component gets its configuration from Secrets Manager with the help of the Secret manager managed component.

For Enterprise deployments, ingestion into AWS IoT Core can be best achieved by deploying The Things Stack AWS IoT default integration into your AWS account.

The Things Stack offers an API and various integrations that can be used for remote ingestion or control, or can be used by other Greengrass components to achieve local data processing and actions. The Pub/Sub integration is a good option for achieving ingestion into AWS IoT Core from the Open Source edition of The Things Stack.

Repository Contents

Item	Description
/artifacts	Greengrass V2 component artifacts that run on the Greengrass edge runtime.
/cicd	CDK Typescript app for a CodePipeline CI/CD pipeline.
/images	Images for README files.
/libs	Python libraries shared by Python scripts.
/recipes	Greengrass V2 component recipe templates.
/robot	Robot Framework integration tests.
/tests	Pytest unit tests
/tts-config	The Things Stack configuration files.
create_certs.sh	Creates self-signed TLS certificates for a given domain name or IP address.
create_component_version.py	Creates a new version of The Things Stack LoRaWAN component in the Greengrass cloud service.
create_config_minimal.py	Creates a minimal configuration needed to deploy the component for a given domain name or IP address.
create_config_secret.py	Creates or updates The Things Stack configuration secret in Secrets Manager.
deploy_component_version.py	Deploys a component version to the Greengrass core device target.
quickstart.sh	Creates and deploys a component with default configuration for a given domain name or IP address.

Requirements and Prerequisites

Greengrass Core Device

Platform

This component requires that the Greengrass device be running a Linux operating system. It supports all architectures supported by Greengrass itself.

Edge Runtime

The Greengrass edge runtime needs to be deployed to a suitable machine, virtual machine or EC2 instance. Please see The Things Stack stated prerequisites for guidance on the resources required. For a small number of devices and gateways, the processing and memory requirements are small.

The Greengrass machine or instance should not be running any process that uses any ports that may be needed by The Things Stack. The specific ports used depends on your particular configuration of The Things Stack. The configuration generated by quickstart.sh requires ports 443, 1700, 6379, 8881, 8882, 8883, 8884, 8885, 8887, 26256 and 26257 to be available.

Docker Requirements

The Things Stack is generally composed as an application with multiple Docker containers, using Docker Compose. Images are obtained from Docker Hub.

Therefore your core device must meet the requirements to run Docker containers using Docker Compose and Docker Hub.

Not all releases of these third-party container images support all Greengrass architectures. Care must be taken to select appropriate images and releases to suit your target architecture.

Python Requirements

This component requires both python3 and pip3 to be installed on the core device.

Greengrass Cloud Services

Core Device Role

This component downloads artifacts from an S3 bucket named greengrass-tts-lorawan-ACCOUNT-REGION. Therefore your Greengrass core device role must allow the s3:GetObject permission for this bucket. For more information: https://docs.aws.amazon.com/greengrass/v2/developerguide/device-service-role.html#device-service-role-access-s3-bucket

Additionally, this component downloads sensitive The Things Stack configuration from Secrets Manager. Therefore your Greengrass core device role must also allow the secretsmanager:GetSecretValue permission for the greengrass-tts-lorawan-ID secret.

Policy template to add to your device role (substituting correct values for ACCOUNT, REGION and ID):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::greengrass-tts-lorawan-ACCOUNT-REGION/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:greengrass-tts-lorawan-ID"
    }
  ]
}

Developer Machine

AWS CLI

It may be necessary to upgrade your AWS CLI if you wish to use any greengrassv2 commands, as these are relatively recent additions to the CLI.

cfssl

If you wish to generate your own self-signed TLS certificates for your domain, or use quickstart.sh, then cfssl is required. Example installation on an Ubuntu or Debian machine:

sudo apt update
sudo apt install -y golang-cfssl

Or using Homebrew (Mac or Linux):

brew install cfssl

Alternatively consider installing from released binaries available in the cfssl GitHub repository. Windows installation executables are available.

Python

Most of the scripts in this repository are Python scripts. They are Python 3 scripts and hence python3 and pip3 are required.

Package dependencies can be resolved as follows:

pip3 install -r requirements.txt
pip3 install -r robot/requirements.txt

Please consider to use a virtual environment.

Bash

The quickstart.sh and create_certs.sh scripts are Bash scripts. If using a Windows machine, you will need a Bash environment.

Getting Started

Here we define two ways to get started: Quickstart or Slowstart.

All scripts used should be compatible with Linux, Mac or Windows operating systems, provided a Bash environment is available.

Quickstart

The quickstart.sh bash script is supplied to help you get going fast. For experimentation in non-production settings.

Before running the script, users must:

Deploy Greengrass V2 to an x86-compatible Linux machine, virtual machine or EC2 instance. The default The Things Stack configuration settings will pull amd64 images, hence the x86-compatibility requirement.
Initialize the Greengrass core device to satisfy the the requirements to run Docker containers using Docker Compose and Docker Hub.
Install cfssl on the developer machine.

The Quickstart script will:

Install required Python packages.
Create self-signed certificates for a given domain name or IP address.
Create a minimal The Things Stack configuration for the given domain name or IP address.
Upload the configuration to a configuration secret in Secrets Manager.
Create a new component version in Greengrass cloud services, with artifacts in an S3 bucket.
Prompt you to add permissions for the configuration secret and artifacts bucket to the Greengrass core device role.
Deploy the new component version to the Greengrass core (creating The Things Stack admin user in the process).
Run Robot Framework integration tests to confirm The Things Stack is running under Greengrass.

The script accepts 6 arguments:

Domain name or IP address (this can be an EC2 default domain name).
AWS region.
The password of The Things Stack admin user that will be created.
The email of The Things Stack admin user that will be created.
Component version. (Semantic major.minor.patch.)
Greengrass Core device name.

Example execution:

bash quickstart.sh example.com ap-southeast-1 mypassword user@example.com 1.0.0 GGTheThingsStackLoRaWAN

Slowstart

For any serious use of the component, Quickstart shall not be appropriate.

Manual Deployment

If not using Quickstart, you must perform the following steps:

Deploy the Greengrass runtime to your machine, virtual machine or EC2 instance. This should be a Linux machine, but can be of any architecture.
Configure the LoRaWAN stack as per https://www.thethingsindustries.com/docs/getting-started/installation/configuration/. Take care to select Docker images with the correct architecture.
Configure certificates as per https://www.thethingsindustries.com/docs/getting-started/installation/certificates/.
If BYO/generating custom TLS certificates (rather than using Lets Encrypt), then add them to the /tts-config directory.
If using an Enterprise license file, add it to the /tts-config directory.
Run create_config_secret.py to create the configuration secret in Secrets Manager.
Run create_component_version.py to create a component version in Greengrass cloud service, and upload artifacts to S3.
Add permissions for the configuration secret and artifacts bucket to the Greengrass core device role.
Run deploy_component_version.py to deploy the new component version to your Greengrass device.

For iterative configuration changes, repeat steps 2, 6, 7 and 9.

Example Execution

Example of steps 6, 7 and 9:

python3 create_config_secret.py ap-southeast-1 mypassword user@example.com
python3 create_component_version.py 1.0.0 ap-southeast-1
python3 deploy_component_version.py 1.0.0 ap-southeast-1 MyCoreDeviceThingName

This example:

Operates in AWS region ap-southeast-1.
Creates and deploys component version 1.0.0 to Greengrass core device MyCoreDeviceThingName.
Creates an admin user for The Things Stack with password mypassword and email user@example.com.

CI/CD Pipeline

This repository offers a CodePipeline CI/CD pipeline as a CDK application. This can be optionally deployed to the same account as the Greengrass core.

This CI/CD pipeline automates steps 7 and 9. Following deployment, it performs automated smoke tests to ensure that The Things Stack has started correctly. With the pipeline deployed, users can make iterative configuration changes, update the configuration secret using create_config_secret.py, and then trigger the CI/CD pipeline to handle the rest.

Automated Testing

This repository includes an automated test suite built on top of Robot Framework. This can be run on demand from the command-line but it is also included as part of the CI/CD pipeline.

The Things Stack Configuration Tips

Configuration of the LoRaWAN stack is mainly an exercise in editing tts-config/docker-compose.yaml and tts-config/config/stack/ttn-lw-stack-docker.yml. Please consult The Things Stack configuration documentation for details.

Defaults

The tts-config/docker-compose.yaml and tts-config/config/stack/ttn-lw-stack-docker.yml files in this repository are The Things Stack Enterprise example configuration files, with the following divergence:

All Enterprise configuration elements are commented out so that it defaults to Open Source edition.
Acme Let's Encrypt TLS certificates are disabled and custom TLS certificates are enabled.
All non-TLS ports and endpoints are disabled.

This is to facilitate minimal effort in deploying this component to a server that only has an IP address, a local network DNS name or an EC2 default domain name.

Enterprise versus Open Source

As compared to Open Source, an Enterprise license adds:

Device Claim Server
Multi-tenant support
Storage Integration
AWS IoT Default Integration support
Support plan

Production Deployments

Per The Things Stack recommendations, production deployments should reference specific Docker image tags (not just latest) and consideration should also be given to using external managed instances of the databases.

Image Architecture

CockroachDB is not available for Arm architecture. You must use PostgreSQL instead.

The Things Stack images tagged latest on Docker Hub are not available for Arm archiecture. Only formal releases are.

Pub/Sub Integration

The Pub/Sub integration can be used to achieve ingestion into AWS IoT Core. This is particularly useful for the Open Source edition of the stack since the The Things Stack AWS IoT default integration is only available for Enterprise deployments. In configuring the Pub/Sub integration, the Server URL should define the AWS IoT Core endpoint as follows:

mqtts://ENDPOINTID-ats.iot.ap-REGION.amazonaws.com:8883

Development

Static Analysis

Static analysis is performed using Pylint. Example execution:

pylint artifacts libs tests *.py

Unit Tests

Unit tests are performed using pytest and moto.