Import Pipeline and Job - Inconsistencies and using wrong configuraitons

Question

Import Pipeline and Job - Inconsistencies and using wrong configuraitons

sgokaram-saagie opened this issue 4 years ago · comments

Shivakumar Gokaram commented 4 years ago

When running a integration test suite noticed that the ImportJob was using a wrong configuration and importing old artifacts.

See below output refer to importpython , instead of importing a single python job its importing pipeline. This is a similar issue to caching issue we faced before.

saagies-mbp:Gradle shivakumar$ gradle projectTestAll

Task :projectExportDocker
{"status":"success","exportfile":"./job/docker.zip"}

Task :projectImportDocker
{status=success, job=[{id=22712, name=jupyter/scipy-notebook}], pipeline=[], variable=[]}

Task :projectExportJava
{"status":"success","exportfile":"./job/java.zip"}

Task :projectImportJava
{status=success, job=[{id=21754, name=RetrieveCustomerData}], pipeline=[], variable=[]}

Task :projectExportPipelineWithJobandVersion
{"status":"success","exportfile":"./pipeline/pipelinejobver.zip"}

Task :projectImportPipelineWithJobandVersion
{status=success, job=[{id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=21754, name=RetrieveCustomerData}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=10}], variable=[]}

Task :projectExportPython
{"status":"success","exportfile":"./job/python.zip"}

Task :projectImportPython
{status=success, job=[{id=22702, name=GradlePluginMigraiton-Manual}, {id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportR
{"status":"success","exportfile":"./job/r.zip"}

Task :projectImportR
{status=success, job=[{id=20983, name=Demo Projects 2}, {id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportSpark
{"status":"success","exportfile":"./job/spark.zip"}

Task :projectImportSpark
{status=success, job=[{id=2777, name=Build new Model, versions=2}, {id=2779, name=ERP Data Cleansing, versions=10}, {id=2778, name=Import ERP Data, versions=4}, {id=9726, name=Build model (sales / ERP), versions=3}, {id=4708, name=pm Extraction batch PLCs, versions=7}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportSqoop
{"status":"success","exportfile":"./job/sqoop.zip"}

Task :projectImportSqoop
{status=success, job=[{id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}, {id=22186, name=test_koalas}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportTalend
{"status":"success","exportfile":"./job/talend.zip"}

Task :projectImportTalend
{status=success, job=[{id=21439, name=talend job}, {id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

Task :projectExportV1Var FAILED

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':projectExportV1Var'.

Didn't find variable name: DOESNOTEXIST in the required environment variables list in V1

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org

BUILD FAILED in 7m 19s
17 actionable tasks: 17 executed

mohamed amine ziraoui · Answer 1 · Sat Sep 19 2020 01:59:06 GMT+0800 (China Standard Time)

HOW TO TEST WIP:

The release : 2.1.11

import io.saagie.plugin.dataops.tasks.projects.artifact.ProjectsExportJobV1Task
import io.saagie.plugin.dataops.tasks.projects.artifact.ProjectsImportJobTask
import groovy.json.JsonSlurper;
plugins {
	id 'groovy'
	id 'base'
	id 'io.saagie.gradle-saagie-dataops-plugin' version '2.1.11'
}

dependencies {
	implementation group: 'gradle.plugin.io.saagie', name: 'gradle-saagie-dataops-plugin', version: '2.1.11'
}

def exportLocation = '/test_gradle/' // <=== edit this with the export path
def tmpLocation = './tmp' // <=== edit this with the tmp file path 

task projectExportSqoop(type: ProjectsExportJobV1Task) { // <====== Create the first job
	configuration = saagie {}
	doFirst {
		projectExportSqoop.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			job {
				ids = ['22186']
			}
			exportArtifacts {
				export_file = exportLocation + '/job/sqoop.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportSqoop'
	group = 'Saagie'
	description= 'export v1 sqoop job for saagie'
}

task projectExportSpark(type: ProjectsExportJobV1Task) { // <====== Create the second job
	configuration = saagie {}
	doFirst {
		projectExportSpark.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			job {
				ids = ['4708']
				include_all_versions=true
			}
			exportArtifacts {
				export_file = exportLocation + '/job/spark.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportSpark'
	group = 'Saagie'
	description= 'export v1 spark job for saagie'
}

task projectExportJava(type: ProjectsExportJobV1Task) { // <====== Create the second job
	configuration = saagie {}
	doFirst {
		projectExportJava.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			job {
				ids = ['21754']
			}
			exportArtifacts {
				export_file = exportLocation + '/job/java.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportJava'
	group = 'Saagie'
	description= 'export v1 Java job for saagie'
}

task projectExportR(type: ProjectsExportJobV1Task) { // <====== Create the second job
	configuration = saagie {}
	doFirst {
		projectExportR.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			job {
				ids = ['20983']
			}
			exportArtifacts {
				export_file = exportLocation + '/job/r.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportR'
	group = 'Saagie'
	description= 'export v1 R job for saagie'
}

task projectExportTalend(type: ProjectsExportJobV1Task) { // <====== Create the second job
	configuration = saagie {}
	doFirst {
		projectExportTalend.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			job {
				ids = ['21439']
			}
			exportArtifacts {
				export_file = exportLocation + '/job/talend.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportTalend'
	group = 'Saagie'
	description= 'export v1 talend job for saagie'
}
task projectExportPython(type: ProjectsExportJobV1Task) { // <====== Create the second job
	configuration = saagie {}
	doFirst {
		projectExportPython.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			job {
				ids = ['22702']
			}
			exportArtifacts {
				export_file = exportLocation + '/job/python.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportPython'
	group = 'Saagie'
	description= 'export v1 python job for saagie'
}
task projectExportDocker(type: ProjectsExportJobV1Task) { // <====== Create the second job
	configuration = saagie {}
	doFirst {
		projectExportDocker.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			job {
				ids = ['22712']
			}
			exportArtifacts {
				export_file = exportLocation + '/job/docker.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportDocker'
	group = 'Saagie'
	description= 'export v1 Docker job for saagie'
}
task projectExportPipelineWithJobandVersion(type: ProjectsExportJobV1Task) { // <====== Create the second job
	configuration = saagie {}
	doFirst {
		projectExportPipelineWithJobandVersion.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			pipeline {
				ids = ['484']
				include_job=true
				include_all_versions=true
			}
			exportArtifacts {
				export_file = exportLocation + '/pipeline/pipelinejobver.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportPipelineWithJobandVersion'
	group = 'Saagie'
	description= 'export Pipleline with job and version dependencies'
}
task projectExportPipeline(type: ProjectsExportJobV1Task) { // <====== Create the second job
	configuration = saagie {}
	doFirst {
		projectExportPipeline.configuration = saagie {
			newTaskContext {}
			server {
				url = saagieurl
				login = saagieuserid
				password = saagiepassword
				environment = saagieplatformid
				jwt = true
				acceptSelfSigned = true
			}
			pipeline {
				ids = ['36']
				include_job=false
				include_all_versions=false
			}
			exportArtifacts {
				export_file = exportLocation + '/pipeline/pipeline.zip'
				overwrite=true
				temporary_directory= tmpLocation
			}
		}
	}
	taskName = 'projectExportPipeline'
	group = 'Saagie'
	description= 'export Pipleline with out dependency'
}
task projectImportSqoop( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportSqoop'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportSqoop.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/job/sqoop.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'import sqoop job for saagie'
	taskName = 'projectImportSqoop'
}
task projectImportSpark( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportSpark'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportSpark.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/job/spark.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'import sqoop job for saagie'
	taskName = 'projectImportSpark'
}
task projectImportJava( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportJava'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportJava.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/job/java.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'import java job for saagie'
	taskName = 'projectImportJava'
}
task projectImportR( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportR'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportR.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/job/r.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'import R job for saagie'
	taskName = 'projectImportR'
}
task projectImportTalend( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportTalend'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportTalend.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/job/talend.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'import Talend job for saagie'
	taskName = 'projectImportTalend'
}
task projectImportPython( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportPython'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportPython.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/job/python.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'import Python job for saagie'
	taskName = 'projectImportPython'
}
task projectImportDocker( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportDocker'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportDocker.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/job/docker.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'import Docker job for saagie'
	taskName = 'projectImportDocker'
}
task projectImportPipelineWithJobandVersion( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportPipelineWithJobandVersion'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportPipelineWithJobandVersion.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/pipeline/pipelinejobver.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'Import Pipeline with Dependency Job and all versions'
	taskName = 'projectImportPipelineWithJobandVersion'
}
task projectImportPipeline( // <======== Import artifacts, see that we use the gradle dependOn
		type: ProjectsImportJobTask) {
	dependsOn(['projectExportPipeline'])
	configuration = saagie {
		server {
			url = saagieurl
			login = saagieuserid
			password = saagiepassword
			environment = saagieplatformid
			jwt = true
		}
	}
	doFirst {
		projectImportPipeline.configuration = saagie {
			newTaskContext {}
			project {
				id = saagieprojectid
			}
			importArtifacts {
				import_file = exportLocation + '/pipeline/pipeline.zip'
				temporary_directory= tmpLocation
			}
		}
	}
	description= 'Import Pipeline with Dependency Job and all versions'
	taskName = 'projectImportPipeline'
}
task projectTestAll( // <======== Import artifacts, see that we use the gradle dependOn
		type:Exec) {
	dependsOn(['projectImportSqoop','projectImportSpark','projectImportJava','projectImportR','projectImportTalend','projectImportPython','projectImportDocker', 'projectImportPipeline', 'projectImportPipelineWithJobandVersion' ])
	commandLine 'echo','Testing'
}

If you got this response then It worked fine

❯ gradle -b build.tasks.gradle projectTestAll

> Task :projectExportDocker
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/docker.zip"}

> Task :projectImportDocker
{status=success, job=[{id=22712, name=jupyter/scipy-notebook}], pipeline=[], variable=[]}

> Task :projectExportJava
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/java.zip"}

> Task :projectImportJava
{status=success, job=[{id=21754, name=RetrieveCustomerData}], pipeline=[], variable=[]}

> Task :projectExportPipeline
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/pipeline/pipeline.zip"}

> Task :projectImportPipeline
{status=success, job=[], pipeline=[{id=36, name=Wine Recognition}], variable=[]}

> Task :projectExportPipelineWithJobandVersion
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/pipeline/pipelinejobver.zip"}

> Task :projectImportPipelineWithJobandVersion
{status=success, job=[{id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}, {id=2777, name=Build new Model}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

> Task :projectExportPython
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/python.zip"}

> Task :projectImportPython
{status=success, job=[{id=22702, name=GradlePluginMigraiton-Manual}], pipeline=[], variable=[]}

> Task :projectExportR
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/r.zip"}

> Task :projectImportR
{status=success, job=[{id=20983, name=Demo Projects 2}], pipeline=[], variable=[]}

> Task :projectExportSpark
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/spark.zip"}

> Task :projectImportSpark
{status=success, job=[{id=4708, name=pm Extraction batch PLCs, versions=8}], pipeline=[], variable=[]}

> Task :projectExportSqoop
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/sqoop.zip"}

> Task :projectImportSqoop
{status=success, job=[{id=22186, name=test_koalas}], pipeline=[], variable=[]}

> Task :projectExportTalend
{"status":"success","exportfile":"/home/amine/Desktop/test_gradle/job/talend.zip"}

> Task :projectImportTalend
{status=success, job=[{id=21439, name=talend job}], pipeline=[], variable=[]}

> Task :projectTestAll
Testing

BUILD SUCCESSFUL in 2m 43s
19 actionable tasks: 19 executed

For every export / import tasks with corresponding task
then the build script did succeed.

You then need to check in the saagie platform that the all the job / pipeline that is exported, and the versions for the job or the pipeline is added like it s shown in the job.json/pipeline.json for every job or pipeline in every exported artifact

Tested on version : 2.1.11. Worked fine

Bug Behavior :
Python import with have multiple jobs and pipelines from previous tasks:

Task :projectImportPython
{status=success, job=[{id=22702, name=GradlePluginMigraiton-Manual}, {id=2777, name=Build new Model}, {id=2779, name=ERP Data Cleansing}, {id=2778, name=Import ERP Data}, {id=9726, name=Build model (sales / ERP)}], pipeline=[{id=484, name=Churn Update model, versions=11}], variable=[]}

mohamed amine ziraoui · Answer 2 · Sat Sep 19 2020 02:32:13 GMT+0800 (China Standard Time)

Tested on version : 2.1.11 OK

ZouhairBear · Answer 3 · Tue Sep 22 2020 06:44:25 GMT+0800 (China Standard Time)

saagie / gradle-saagie-dataops-plugin

Import Pipeline and Job - Inconsistencies and using wrong configuraitons

Tested on version : 2.1.11 OK

Tested on version : 2.1.12 OK