- No separate types for sources and destinations
- It's common to write to and read from the same resource.
- When reading, you must specify a "collection".
- A chaining API is used.
- Visualizes the pipeline better.
- Similar style used by some streaming frameworks: Java streams, RxJava, Flink, Kafka Streams.
To see how the API is used in different use cases, the following examples are available in the examples directory:
- Simple (read records from a source, write them to a destination)
- Simple with processing
Multiple sources to multiple destinations-- not really a thing- One source, many destination, with record processing
- Explore joins (merge payloads of records with same keys, assuming they are structured)
- How to configure the app?
- Provide Maven archetype
- Provide guidelines for logging
- Decide which Java versions should be supported, likely Java 8 or 11, see: this and this.
Why does process() need to be an exported/public method?- Should we allow reading all collections from a resource? We're considering that an anti-pattern and discourage that,
so should we make it impossible?
- Answer: no, we shouldn't be allowing that, for two reasons:
- Implicitly, tables are mapped to topics, but the issue is that we cannot predict what the topic name will be.
- Different collections (tables, buckets, etc.) mean different types, so a
- Is there any metadata we want to automatically add to all records (e.g. name of the turbine app and the version)?
- Sensible idea, but not a must-have for now.