damavis / airflow-pentaho-plugin

Pentaho plugin for Apache Airflow - Orquestate pentaho transformations and jobs from Airflow

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

CarteTransOperator works incorrect when transformation failed

evgdol opened this issue · comments

commented

Scenario (simplified)

  1. We have the table in the database with constraints
    CREATE TABLE tst_constraints( id int PRIMARY KEY, val text )

  2. We have the simple transformation that inserts the data into this table
    image

  3. Call the transformation in Airflow
    trans = CarteTransOperator( task_id='tst_trans_fail', pdi_conn_id='pentaho', trans='/public/test/tr_tst_failed_tab_constraint' )

Result

  1. If table is empty (no constraints inconsistency) all works as expected
  2. If the table contains inserted data, I have found two possible scenarios (the second is more often):
    2.1. The task finished with SUCCESS flag
    image
    image

2.2. The task freezing, but the transformation already failed.
image

Expected result

CarteTransOperator works properly

Additions

After discussion in Issue #29 :
If we wrap around the transformation in job, it works fine. But I want to know if it is possible to solve this issue without this.

Environment

Airflow 2.5.1 + Python 3.10 + Pentaho server CE 8.0

commented

Not solved!
image

Sorry. Now it should work.

commented

Nothing changed. The result is the same as on previous my screenshot.

commented

I have found the issue.
You should change the condition in WHILE:
image
There are FINISHED_STATUSES used in your code.
I have changed on my side and it works.

You are right @evgdol . I need to add some tests for this. But I'm about to release a version with this fix.