zillow / intake-dal

Dataset abstraction over disparate storage systems (eg: bulk, streaming, serving, ...).

Welcome to Intake DAL (data access layer) plugin

This Intake plugin helps abstract a dataset over disparate storage systems (eg: bulk, streaming, serving, ...). It also provides an easy way to specialize a hierarchical catalog to a default DAL storage system.

Sample Catalog source entry:

user_events:
  driver: dal
  args:
    default: 'local'
    storage:
      local: 'csv://{{ CATALOG_DIR }}/data/user_events.csv'
      serving: 'in-memory-kv://foo'
      batch: 'parquet://{{ CATALOG_DIR }}/data/user_events.parquet'

Example code using sample catalog:

# Specialize the catalog dal default storge mode datasources
# to be "serving".
cat = DalCatalog(path, storage_mode="serving")

# reads from the serving storage system
# using the in-memory-kv Intake plugin
df = cat.user_events.read()