ray-project / deltacat

A portable Pythonic Data Catalog API powered by Ray that brings exabyte-level scalability and fast, ACID-compliant, change-data-capture to your big data workloads.

ray-project/deltacat Issues

Use FilenameProvider in ray dataset instead of BlockWritePathProvider
Updated 21 days ago
Docs + Examples
Updated 2 months ago3
Daft reader for DeltaCAT catalogs in general
Updated 3 months ago
Extending the delete formats supported by DeltaCAT
Updated 4 months ago
Deprecate sliced_string_cast in favor of library utils
Updated 6 months ago
Update input_size_bytes, input_records, and input_records
Updated 7 months ago
Integration tests for materialize: skip untouched files improvement
Closed 7 months ago1
[EPIC] Implement a version of compactor that writes files partitioned by pk hash
Closed 7 months ago
Split object before persisting to memcached object store if exceeds max size
Closed 7 months ago
Persist audit information regardless of compaction failure
Updated 7 months ago
[Compaction] Integrate with Daft parquet reader to efficiently read subset of columns
Closed 8 months ago1
Log the memory consumption to calculate correctness of worker estimation
Closed 9 months ago1
Audit relevant execution info
Closed 9 months ago1
Refactor the existing discover_delta to handle different cases
Closed 9 months ago1
Test compaction session v1 and v2 using local deltacat storage
Closed 9 months ago2
Implement merge v2
Closed 9 months ago1
Gracefully handle empty CSV
Closed 9 months ago
Optimize pyarrow uTSV reads
Closed 9 months ago1
Implement compaction session v2
Closed 9 months ago
[Iceberg] Hash Bucket Aware Table Joins
Updated 9 months ago2
[Iceberg] AWS Glue Job Runner
Updated 9 months ago1
[Iceberg] Daft Dataframe Reader
Updated 9 months ago
[Iceberg] Implement compaction for partitioned tables
Updated 9 months ago1
[Iceberg] Implement compaction support for unpartitioned tables
Updated 9 months ago1
[Iceberg] Describe strategy to commit partitions and deltas
Updated 9 months ago
[Iceberg] Support commit operation in pyiceberg
Updated 9 months ago
[Iceberg] Describe how Iceberg table maps to DeltaCAT Delta concept
Updated 9 months ago2
Add local deltacat storage that mimics the Catalog behavior
Closed 10 months ago
Implement hash bucket v2
Closed 10 months ago
Investigation: Discover and document all DeltaCAT storage APIs used by compaction
Closed 10 months ago1
Inject dynamic object store implementations to share objects across cluster
Closed 10 months ago
Support reading Parquet meta using ParquetFile class in pyarrow
Closed 10 months ago
Cleanup rehash and other dead primary key index code
Closed 10 months ago1
Support pyarrow 12.0.1 for repartitioning
Closed 10 months ago
Support read Iceberg dataset in Ray in distributed mode
Closed 10 months ago8
Move stage delta implementation into internal implementation
Updated a year ago
Always persist a high watermark for source table
Updated a year ago2
Set appropriate hash bucket count and distribute deltas uniformly across cluster
Closed a year ago
Add unit test for repartition API
Closed a year ago
Replace DistributedDataset with Ray Dataset
Updated a year ago
Use NumPy return types for dedupe tasks
Updated a year ago
Support passing kwargs to the underlying reader/writer in pyarrow via compact_partition
Closed a year ago
fix bug due to assumption that list_deltas returns in ascending order
Closed a year ago1
Support read Iceberg dataset in Ray
Closed a year ago1
raise error when compaction cannot be completed in a single round
Closed a year ago
Allow passing list_deltas kwargs to compact_partition
Closed a year ago
Support different delta locator during dedupe and materialize
Closed a year ago
Create an enum for storing locators
Updated a year ago
Compaction recover from partition commit failures
Updated a year ago
Backfill compaction from source partition only and no previous round completion fails
Closed a year ago