damavis / airflow-hop-plugin

Apache Hop plugin for Apache Airflow - Orquestate Apache Hop pipelines and workflows from Airflow

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Tasks

pauloricardoferreira opened this issue · comments

hello... I'll give you some improvements soon... I'm going to fork and then make a PR. thanks

I have evolved the project a lot.

I made some improvements that the project needed.

As Apache Hop was developed to work in a decoupled way, I made some adjustments to make the most of the Plugin.

At the moment I am creating a routine to create a json with the metadata.json that the plugin needs, this way, it will no longer be necessary to export through the graphical interface.

As soon as possible, I will create a PR and hope it is reviewed and approved.

Thanks.

Hello,
I've seen your PRs, I will be checking and testing the changes in the next couple of days. I'll keep you informed.

TKS.

Something interesting is that the Plugin reads the variables used in the project.

And one of the important variables for the Project is PROJECT_HOME, which needs to be defined in the Apache Hop interface for perfect execution.

image

Example project configuration.

It is necessary to create this environment variable, pointing to the exact location of the project.

This way the plugin works in an exemplary way.


You could also find a way to make this variable and add it to the Payload XML.

It is found in the hop configuration file.

"projectConfigurations" : [ {
"projectName" : "hop_repo",
"projectHome" : "/opt/projetos/hop_repo",
"configFilename" : "project-config.json"
}

I didn't implement this part, but I believe it is possible.

This way the plugin is completely independent

TKS

I did a test running only the Pipeline, and the process ran perfectly.

Now, when a Workflow calls a pipeline, it cannot execute it, because it cannot find the PROJECT_HOME path, and therefore cannot find the pipeline path.

I tried to find it in the code, but I couldn't

TKS.

Something interesting is that the Plugin reads the variables used in the project.

And one of the important variables for the Project is PROJECT_HOME, which needs to be defined in the Apache Hop interface for perfect execution.

image

Example project configuration.

It is necessary to create this environment variable, pointing to the exact location of the project.

This way the plugin works in an exemplary way.

You could also find a way to make this variable and add it to the Payload XML.

It is found in the hop configuration file.

"projectConfigurations" : [ { "projectName" : "hop_repo", "projectHome" : "/opt/projetos/hop_repo", "configFilename" : "project-config.json" }

I didn't implement this part, but I believe it is possible.

This way the plugin is completely independent

TKS

I encountered the problem with the PROJECT HOME variable.

I needed to add a variable to root: root.append(project_home).

PR Created.

hello... Did you test the plugin?

hello... Did you test the plugin?

Hello, sorry for taking so long. Life got busy. Could you unify all your PRs into a single PR? It is really bothersome to perform testing this way and I don't like merging partial implementations. Also, I've noticed some breaking changes. Please, try to maintain backwards compatibility. If you need to you can create a new "updated" operator.