GoogleCloudPlatform / data-lineage-producer-java-library

Data Lineage Producer Java Library

Java producer library for Data Lineage.

The library provides Synchronous and Asynchronous clients for interacting with Data Lineage API.

Building

To build the library, run

./gradlew build

Or, for a shadow jar:

./gradlew shadowJar

The output will be located at lib/build/libs.

Performing calls

To perform a call, you need to:

1. Create a client
2. Create a request object
3. Call a corresponding method

Sample code:

SyncLineageProducerClient client = SyncLineageProducerClient.create();
ProcessOpenLineageRunEventRequest request =
        ProcessOpenLineageRunEventRequest.newBuilder()
            .setParent(parent)
            .setOpenLineage(openLineageMessage)
            .build();
client.processOpenLineageRunEvent(request);

Sample code to compose an OpenLineage message:

Struct job = Struct.newBuilder()
    .putFields("namespace", Value.newBuilder().setStringValue("namespace-1").build())
    .putFields("name", Value.newBuilder().setStringValue("job-1").build()).build();
Struct run = Struct.newBuilder()
    .putFields("runId", Value.newBuilder().setStringValue("1234-5678").build()).build();
Struct openLineageMessage = Struct.newBuilder()
    .putFields("eventType", Value.newBuilder().setStringValue("COMPLETE").build())
    .putFields("eventTime",
        Value.newBuilder().setStringValue("2023-04-04T13:21:16.098Z").build())
    .putFields("producer", Value.newBuilder().setStringValue("producer-1").build())
    .putFields("schemaURL", Value.newBuilder()
        .setStringValue("https://openlineage.io/spec/1-0-5/OpenLineage.json#/$defs/RunEvent")
        .build())
    .putFields("job", Value.newBuilder().setStructValue(job).build())
    .putFields("run", Value.newBuilder().setStructValue(run).build())
    .putFields("inputs",
        Value.newBuilder().setListValue(ListValue.newBuilder().addValues(Value.newBuilder()
            .setStructValue(Struct.newBuilder()
                .putFields("namespace", Value.newBuilder().setStringValue("bigquery").build())
                .putFields("name",
                    Value.newBuilder().setStringValue("input_project.input_dataset.input_table")
                        .build()).build()).build()).build()).build())
    .putFields("outputs",
        Value.newBuilder().setListValue(ListValue.newBuilder().addValues(Value.newBuilder()
            .setStructValue(Struct.newBuilder()
                .putFields("namespace", Value.newBuilder().setStringValue("bigquery").build())
                .putFields("name", Value.newBuilder()
                    .setStringValue("output_project.output_dataset.output_table").build())
                .build()).build()).build()).build()).build();

or, to convert it from an existing string:

Struct openLineageStruct = OpenLineageHelper.jsonToStruct(jsonString);