openownership / register-transformer-dk

Register Transformer DK ingests records from a Kinesis stream (published by register_ingester_dk) and transforms them into BODS v0.2 records

Register Transformer DK

Register Transformer DK is a data transformer for the OpenOwnership Register project. It processes bulk data published to AWS S3, such as emitted from AWS Kinesis Data Firehose, converts them into the Beneficial Ownership Data Standard (BODS) format, and stores records in Elasticsearch. Optionally, it can also use AWS Kinesis for processing streamed data (rather than bulk data published to AWS S3), or for publishing newly-transformed records to a different stream.

The transformation schema is BODS 0.2.

Installation

Install and boot Register.

Configure your environment using the example file:

cp .env.example .env

Create the Elasticsearch indexes:

docker compose run transformer-dk create-indexes

Testing

Run the tests:

docker compose run transformer-dk test

Usage

To transform the bulk data from a prefix in AWS S3:

docker compose run transformer-dk transform-bulk raw_data/source=DK/year=2023/month=10/