Giters

damavis / airflow-pentaho-plugin

Pentaho plugin for Apache Airflow - Orquestate pentaho transformations and jobs from Airflow

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

usage of this plugin

erwanj opened this issue · comments

commented

Hello,

thank you for your plugin.
I have been trying to use it and have some question.
I am wondering how do you proceed with airflow when you need to run a job/transformation to reload historical datas. I have seen that it is possible to trigger manually the dag giving "Configuration JSON". Are you using this?

Thank you.

Erwan

commented

Hello @erwanj

We use to run all transformations with a date as an input parameter, so we use the command airflow backfill, check it out at official documentation: https://airflow.apache.org/docs/stable/cli-ref#backfill

It's recommended to stop the DAG before start the backfill in order to avoid duplicate tasks by the scheduler.

Thank you.