Explicit parameter passing between steps

[Jacek Kuśmierczyk]

Summary

Example:

out1 = couler.create_parameter_artifact(path="/mnt/test.txt")
out2 = couler.create_parameter_artifact(path="/mnt/test2.txt")


def producer(name):
    return couler.run_container(
        image="alpine:3.6", command=["sh", "-c", 'echo "test" > /mnt/test.txt']
        , step_name=name, output=[out1, out2]
    )


def consumer(name):
    inputs = couler.get_step_output(step_name="1")
    return couler.run_container(
        image="alpine:3.6", command=["sh", "-c", 'cat /mnt/test.txt']
        , step_name=name, args=[inputs[0]],
    )


couler.set_dependencies(lambda: producer("1"), dependencies=None)
couler.set_dependencies(lambda: consumer("2"), dependencies=["1"])

Now arguments in consumer template look like this:

arguments:
              parameters:
                - name: para-2-0
                  value: "{{tasks.1.outputs.parameters.output-id-15}}"
                - name: para-2-1
                  value: "{{tasks.1.outputs.parameters.output-id-15}}"
                - name: para-2-2
                  value: "{{tasks.1.outputs.parameters.output-id-16}}"

It would be useful if there was a way for setting dependency between steps without implicit parameter passing.

For myself I just added a flag to run_container that just turns off this behavior.

Use Cases

I have one parent step that generates couple outputs, and I have multiple children steps, each one of them only needs proper subset of parent outputs, the rest of information would be redundant.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[Yuan Tang]

I agree an option to turn off these automatic param name generation might be useful. Would you like to help look into it?