TobikoData / sqlmesh

Efficient data transformation and modeling framework that is backwards compatible with dbt.

Home Page:https://sqlmesh.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Document the process of moving an existing table to SQLMesh

izeigerman opened this issue · comments

We need a guide that covers available options like:

  1. Figure out which snapshot table the model maps to and just rename your historical table to that snapshot table name. Make sure to make the model forward_only, however, so that the prod table is not rebuilt on every change. https://sqlmesh.readthedocs.io/en/stable/guides/incremental_time/#forward-only-models
  2. Create a staging model that writes into a new table and have an actual model on top of it that UNIONs records from the staging model with the records from your existing historical table. This way your historical table never goes away but all updates go into the new model table

To find out the name of the physical table of the local model version the sqlmesh table_name <model_name> command should be used.

Also, depending on the underlying engine, additional techniques can be used. For example, data files can be copied directly and then added manually by partition with Iceberg / Hive.

Duplicate of #2337