AWS-firehose-s3-producer
AWS-firehose-s3-producer is an AWS Lambda function in Javascript that is designed to ingest files uploaded to a S3 bucket into AWS Kinesis Firehose.
Files are uploaded to a S3 bucket which is configured to publish notifications to SQS. SQS then triggers this lambda, which will read the file and stream it to Firehose.
Features
- streams files rather than load them to memory, therefore large files can be processed with minimal resources
- gzipped files are supported
Configuration
The Lambda requires the following environment variable to be set:
FIREHOSE_STREAM_NAME: the Kinesis firehose datastream to stream to.
Edit, then attach the provided policy to the Role your AWS Lambda is configured to use.