Is it possible to load data from Source to Dest between servers?
shut000 opened this issue · comments
Is it possible to load data from Source to Dest between servers? Like what ETL usually does. Thanks!
And I don't know where to add questions, just put it here...
hmm, you'd probably need to expand the ETLs class's ability to hold more than one connection (source_connection / destination_connection) so you could pull from one server and send to another. Let me know if that makes sense and answers your question.
thank you! I tried another one (activewarehouse-etl), it's more complex and support multiple servers. I'm new in Ruby and trying to break it down...
interesyed in this functionality as well. Wondering what the API might look like.
@Emerson @shut000 was just giving this a look over and have an idea on what could work here: right now we have ETL.connection that's used to speak with a single db via #query. This works fine when doing data processing within the same server / db, but totally see your use case. My thought: add a ETL.read_connection and ETL.write_connectionthat you can set and then interact with through two new methods#readand#writewhich would use either{read,write}_connection#query`. One requirement is we'd need to hold the read data in memory and then prepare it to be written, but that shouldn't been too bad!
Happy to take a pass at doing the above, let me know if it's still something you're interested in!