Question: file format for saving pipelines?

Question

Question: file format for saving pipelines?

COLABORATI opened this issue 8 years ago · comments

Hi, interesting project!

When a web gui would like to enable save / edit / of pipelines, it should have some way to save the definition of that pipeline. Do you plan on implementing such a thing? I do not know if there is some common format for these kind of things, but it seems necessary to have something to be able to implement a gui. What are your ideas about this?

Thanks for your attention!

COLABORATI · Answer 1 · Wed Aug 03 2016 01:16:20 GMT+0800 (China Standard Time)

I just learned that airflow uses dill for this.

Reuben Cummings · Answer 2 · Wed Aug 03 2016 04:44:57 GMT+0800 (China Standard Time)

There is already a json serialization format in the original implementation. I removed it in this version since I broke a few Yahoo! Pipes conventions. I do plan to reimplement a serialized pipe format in either json, yaml, or xml. But I'd rather work from a sample GUI output than try to come up with a format on my own. See #4 for more info.