Dataflow BigQuery Dynamic Destinations
Occasionally you may have data of different domains being ingested into a single topic which
requires dynamically routing to the proper output table in BigQuery. One way to accomplish this is
to use a message attribute on the header of the Pub/Sub message to indicate the table which the
message should be routed to. The routing of the messages to the correct table can be accomplished in
a single pipeline by using the BigQueryIO transform's dynamic destination capabilities. In this
pipeline, a SerializableFunction is used to extract the table attribute from the Pub/Sub message
and then subsequently route to the proper table destination.
Pipeline
PubsubToBigQueryDynamicDestinations - A pipeline which consumes JSON messages from Pub/Sub and outputs records to BigQuery tables using a Pub/Sub message attribute to determine the proper table to route the message to.
Getting Started
Requirements
- Java 8
- Maven 3
Building the Project
Build the entire project using the maven compile command.
mvn clean && mvn compile