textract-email-attachments-demo

Overview

This project is a simple Textract and Lambda demo. Incoming email messages are received by SES, persisted to S3 and any PDF PNG or JPG attachments have their text content extracted and stored to DynamoDB.

It uses:

SES for incoming emails
SNS for notifications
S3 for storage of raw emails as well as attachments
Textract to extract text from PDFs, PNGs and JPGs
DynamoDB for processed attachment
Lambda for processing
CloudFormation and SAM to define the infrastructure

Availability

Due to service availability limitations, this project can only be deployed in the following regions:

us-east-1
us-west-2
eu-west-1

Architecture

The demo is deployed using 3 CloudFormation stacks:

cf/incoming-emails.yaml: Sets up an SES rule set and rule to forward incoming emails from the specified domain to S3 and SNS
cf/compute-environment.yaml: Sets up a compute environment including a VPC, subnets and VPC endpoints for the required services
cf/functions.yaml: SAM template that sets up the Lambda functions and other dependencies needed for processing

In order to receive emails using SES, the domain must be verified and the proper MX record must be installed on the hosted zone. The utilities/verify_domain.py script can be used if your domain is hosted on Route 53.

Installation

Prerequisites

Python 3.7
pipenv (pip install --user pipenv) for development
AWS CLI (pip install --user awscli or see documentation)
SAM CLI (pip install --user aws-sam-cli or see documentation)
Configured AWS credentials (aws configure)
Access to add DNS records to the desired domain name
Access to an S3 bucket to deploy the SAM template. The S3 bucket must be in the same region as the stacks you are deploying.

Deployment

From the project root (replace values enclosed in brackets with appropriate values):

pipenv shell
# Warning: this will modify any existing MX record on the domain name!
python3 utilities/verify_domain.py --domain [domain-name] --region [region]
aws cloudformation deploy --stack-name textract-demo-incoming --template-file cf/incoming-emails.yaml --region [region] --parameter DomainName=[domain-name]
aws cloudformation deploy --stack-name textract-demo-vpc --template-file cf/compute-environment.yaml --region [region]
sam package --output-template-file /tmp/packaged.yaml --template-file cf/functions.yaml --s3-bucket [deployment-bucket]
sam deploy --template-file /tmp/packaged.yaml --capabilities CAPABILITY_NAMED_IAM --parameter IncomingEmailStack=textract-demo-incoming ComputeEnvStack=textract-demo-vpc --stack-name textract-demo-functions --region [region]

sarojdongol / textract-email-attachments-demo