nroldanf / aws-da-specialty-playground

Playground to use AWS Data Analytics - Specialty Services

🚀 AWS Data Analytics Specialty

Prerequisites

• AWS CLI v2
• AWS CDK v2.85.0 (requires Node and npm)

Create and initialize the environment:

conda env create -f environment.yaml && conda activate aws-da

Deploy the infrastructure

cd iac && cdk --profile {} deploy --all

Kinesis Data Stream

You can use the SDK (boto3), AWS CLI, Kinesis KPL (C++ or Java library) to ingest data into kinesis.

When writing to kinesis, you need to specify the stream name and the partition key (defines the shard where the data will be written).

CLI

Put a record in the kinesis data stream stream. It returns the shardId and the SequenceNumber for the record.

aws kinesis put-record --stream-name {} --partition-key {} --data "this is my data" --cli-binary-format raw-in-base64-out

Describe the stream:

aws kinesis describe-stream --stream-name {}

Get the shard iterator:

aws kinesis get-shard-iterator --stream-name {} --shard-id {} --shard-iterator-type TRIM_HORIZON

Read the data:

aws kinesis get-records --shard-iterator {}