subhamay-cloudworks / 0090-deutzia-cft

Creating an audit table for a DynamoDB table using CloudTrail, Kinesis Data Stream, Lambda, S3, Glue and Athena and CloudFormation

Project Deutzia: Creating a Audit Trail on a DynamoDB table using S3,CloudTrail Data Events, Lambda, CloudWatch Logs with Event Subscription Filters, Glue and Athena

This is a demo project on cretaing an Audit Trail of a DynamoDB table. All the data events are stored in a S3 bucket and CloudWatch Log Group and enabling viewing the data using Athena.

Description

This sample project demonstrate creating a audit trail of a DynamoDB table using CloudTrail data events. The data event API calls are logged into a S3 bucket and CloudWatch Log Group. Using Subescription Filter, the events are pushed to a Kinesis Data Stream, which invokes a Lambda and formats the required attributes and saved them to a S3 bucket. A Glue Crawler runs every 10 mins and checks for any schema change. The audit table can be queried using Athena.

Getting Started

Dependencies

• Create a Customer Managed KMS Key in the region where you want to create the stack..
• Modify the KMS Key Policy to let the IAM user encrypt / decrypt using any resource using the created KMS Key.

Installing

• Clone the repository.
• Create a S3 bucket and make it public.
• Create the folders - deutzia/cft/nested-stacks, deutzia/cft/cross-stacks, deutzia/code
• Upload the following YAML templates to deutzia/cft/nested-stacks
  • athena-stack.yaml
  • cloudwatch-log-stack.yaml
  • glue-crawler-stack.yaml
  • kinesis-stack.yaml
  • s3-stack.yaml
  • cloudtrail-stack.yaml
  • dynamodb-stack.yaml
  • iam-role-stack.yaml
  • lambda-function-stack.yaml
  • sqs-stack.yaml
• Upload the following YAML templates to deutzia/cft/cross-stacks
  • custom-resource-lambda-stack.yaml
• Upload the following YAML templates to deutzia/cft/
  • deutzia-root-stack.yaml
• Zip and Upload the Python file to deutzia/cft/code
• Create the cross-stack using the template custom-resource-lambda-stack.yaml by using the S3 url and pass the appropriate parameters.
• Create the entire using by using the root stack template custom-resource-lambda-stack.yaml by providing the required parameters and the s3 cross stack name created in the previous step.

Executing program

• Create, Delete or Update an item in the DynamoDB table.
• After around 15 mins, query the Athena table to check the audit trail.

Help

Post message in my blog (https://blog.subhamay.com)

Authors

Contributors names and contact info

Subhamay Bhattacharyya - [e-mail]

Version History

• 0.1
  • Initial Release

License

This project is licensed under Subhamay Bhattacharyya. All Rights Reserved.

Acknowledgments

Inspiration, code snippets, etc.

• Stephane Maarek
• Neal Davis
• Adrian Cantrill