sam-goodwin / punchcard

Type-safe AWS infrastructure.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

collect IEnumerable to S3, Glue and DynamoDB

sam-goodwin opened this issue · comments

commented

We want to be able to collect stream transformations to:

  • S3
  • Glue Tables
  • DynamoDB Tables etc.

Some sort of Collector interface could work, or individual toX methods:

// attach a firehose delivery stream 
stream.collect(scope, id, Collectors.toS3());
// or something like
stream.toS3(scope, id, {
  encryption: BucketEncryption.KMS
});

const ddbTable = stream.collect(scope, id, Collectors.toDynamoDB());
const ddbTable = stream.toDynamoDB(scope, id);

const database = new Database(scope, 'database');
stream
  .map(async event => ({
    id: event.id,
    timestamp: new Date()
  }))
  .collect(scope, id, Collectors.toGlue({
    database,
    schema: new Schema({
      schemaName: 'table-name',
      shape: {
        id: string(),
        timestamp,
      }
      timestampField: 'timestamp'
    })));
// or ..
stream.toGlue(..);