CarteTransOperator works incorrect when transformation failed
evgdol opened this issue · comments
Scenario (simplified)
-
We have the table in the database with constraints
CREATE TABLE tst_constraints( id int PRIMARY KEY, val text )
-
We have the simple transformation that inserts the data into this table
-
Call the transformation in Airflow
trans = CarteTransOperator( task_id='tst_trans_fail', pdi_conn_id='pentaho', trans='/public/test/tr_tst_failed_tab_constraint' )
Result
- If table is empty (no constraints inconsistency) all works as expected
- If the table contains inserted data, I have found two possible scenarios (the second is more often):
2.1. The task finished with SUCCESS flag
2.2. The task freezing, but the transformation already failed.
Expected result
CarteTransOperator works properly
Additions
After discussion in Issue #29 :
If we wrap around the transformation in job, it works fine. But I want to know if it is possible to solve this issue without this.
Environment
Airflow 2.5.1 + Python 3.10 + Pentaho server CE 8.0
Sorry. Now it should work.
Nothing changed. The result is the same as on previous my screenshot.
You are right @evgdol . I need to add some tests for this. But I'm about to release a version with this fix.