CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

CoxAutomotiveDataSolutions/waimak Issues

Track lineage in Waimak
Updated 4 years ago1
Generic sql deequ checks with commas in do not work
Closed 5 years ago
writeAsNamedFiles should create the output directory if it does not exist
Closed 5 years ago
Make Storage Layer implementation consistent on S3
Updated 5 years ago
Retry on property value getters
Closed 5 years ago
Only cache labels if they are actually used by more than one action further down the flow
Closed 5 years ago1
Add execute action/function to DataFlow/SparkDataFlow
Closed 5 years ago
Make the spark prefix configurable for Case Class config parser
Closed 5 years ago1
Allow the fail behaviour of the Parallel Scheduler to be configurable
Updated 5 years ago1
Waimak tmp cleanup flag
Closed 5 years ago
Simplify compaction stages
Closed 5 years ago
Create a generic Main method for Waimak
Closed 6 years ago
Auto-detect schema change a recreate tables for HadoopConnectors
Updated 6 years ago1
Allow tables in the storage layer to be marked in such a way that they will be deduplicated on compaction with no history retained
Closed 6 years ago
Force recreate tables should be a Waimak flag rather than connector flag
Closed 6 years ago
Clean-up strategy should error if no folders exist after committing
Updated 6 years ago
Compaction should take into account width of data
Closed 6 years ago1
Rework storage API
Closed 6 years ago
Test sources are missing from Maven
Closed 6 years ago
Allow repartition by non-named partition for ParquetDataCommiter
Closed 6 years ago
ParquetDataCommitter fails when performing listStatus on the output directory if it does not exist
Closed 6 years ago1
Expose URI used when creating the SparkFlowContext
Closed 6 years ago
Implement an approach to filter existing flows so a subflow could be executed
Updated 6 years ago
Fix logging levels throughout Waimak
Updated 6 years ago
Documentation
Closed 6 years ago3
Actions that do not run should produce an error at the end of a flow by default instead of silently not running
Closed 6 years ago1
Stage and commit post actions do not run if there was an unrelated non-present input label
Closed 6 years ago1
Various API cleanup changes
Closed 6 years ago
Exception when using register as temp view if label has hyphens
Closed 6 years ago
Make region cache append-only apart from during compaction
Closed 6 years ago1
Add trash purge capability to storage layer
Closed 6 years ago
Azure table uploaded spawning too many threads
Closed 6 years ago1
Issue using ADLS for Waimak tmp directory
Closed 6 years ago1
Post action interceptor include intercepted action details in the description
Closed 6 years ago1
Impala submit opens a JDBC connection for every statement, which is slow
Closed 6 years ago
Fix build for Spark 2.0, 2.1 and 2.3
Closed 6 years ago