leapdao / le-aws-cloudwatch

le-aws-cloudwatch

AWS Lambda function for sending AWS CloudWatch logs to Logentries in near real-time for processing and analysing

Acebusters fork changes

The code fitted to stream logs from AWS Lambda in JSON format.

Install with:

zip -r lambda.zip certifi/ yaml/ le_cloudwatch.py
aws lambda update-function-code --function-name cloudwatch-to-logentries --zip-file fileb://lambda.zip --region eu-west-1

Example use cases:

• Forwarding AWS VPC flow Logs
• Forwarding AWS Lambda function logs
• Forwarding AWS CloudTrail logs
• Forwarding any other AWS CloudWatch logs

Obtain log token(s)

1. Log in to your Logentries account
2. Add a new token based log
   • Optional: repeat to add second log for debugging

Deploy the script on AWS Lambda

1. Create a new Lambda function

2. On the "Select Blueprint" screen, press "Skip"

3. Configure function:

   • Give your function a name
   • Set runtime to Python 2.7

4. Upload function code:

   • Create a .ZIP file, containing le_cloudwatch.py and the folder certifi
     • Make sure the files and certifi folder are in the root of the ZIP archive
   • Choose "Upload a .ZIP file" in "Code entry type" dropdown and upload the archive created in previous step

5. Lambda function handler and role

   • Change the "Handler" value to le_cloudwatch.lambda_handler
   • Create a new basic execution role (your IAM user must have sufficient permissions to create & assign new roles)

6. Set Environment Variables:

   • Token value should match UUID provided by Logentries UI or API
   • Region should be that of your LE account - currently only eu

   Key	Value
   region	eu
   token	token uuid

7. Allocate resources:

   • Set memory to 128 MB
   • Set timeout to ~2 minutes (script only runs for seconds at a time)

8. Enable function:

   • Click "Create function"

Configure CloudWatch Stream

1. Create a new stream:

   • Select CloudWatch log group
   • Navigate to "Actions / Stream to AWS Lambda"

   

2. Choose destination Lambda function:

   • Select the AWS Lambda function deployed earlier from drop down menu
   • Click "Next" at the bottom of the page

   

3. Configure log format:

   • Choose the correct log format from drop down menu
   • Specify subscription filter pattern
     • Please see AWS Documentation for more details
     • If this is blank / incorrect, only raw data will be forwarded to Logentries
     • Amazon provide preconfigured filter patterns for some logs
   • Click "Next" at the bottom of the page

   

4. Review and start log stream

   • Review your configuration and click "Start Streaming" at the bottom of the page

   

5. Watch your logs come in:

   • Navigate to your Logentries account and watch your CloudWatch logs appear