Empty source loading fails with `delta` table format on remote filesystems for not-yet-existing tables
jorritsandbrink opened this issue · comments
dlt version
0.5.3
Describe the problem
Empty source loading (e.g. when using dlt.mark.materialize_table_schema()
) fails with delta
table format on remote filesystems for not-yet-existing tables.
Reason is that storage_options
are not passed to DeltaTable.create
.
Does not fail for local filesystem because it does not need storage_options
.
Expected behavior
No response
Steps to reproduce
Run test_delta_table_empty_source
with remote filesystem (e.g. add AZ_BUCKET
to bucket_subset
).
Operating system
Windows
Runtime environment
Local
Python version
3.8
dlt data source
No response
dlt destination
No response
Other deployment details
No response
Additional information
No response