Giters

zerodha / dungbeetle

A highly opinionated, distributed job-queue built specifically for queuing and executing heavy SQL read jobs asynchronously. Supports MySQL, Postgres, ClickHouse.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Ideas for an architectural change to add new plugins (db)

josejuanmontiel opened this issue · comments

commented

As comented in this issue here we can discuss options to implements new drivers (or plugins) in a fancy way that could select and reduce (in the build) dependencies, mantaining clear an easy compile/build the code for the standard cases.

I think those are the principal points for this: maintain easy (build and code) but reduce the size (if that it's important in some scenarios)

I read something like this in some place under the machinery proyect (maybe the unexplore for me now to the v2 experiment https://github.com/RichardKnop/machinery#v2-experiment) but backing to this proyect... i would like to explore two options:

  1. The standard way in golang (i thought)
  2. The wasm aproach e.g this one... that seems an aproach that some project took https://github.com/proxy-wasm/spec/blob/master/docs/WebAssembly-in-Envoy.md

Let's talk

commented

I read something like this in some place under the machinery proyect (maybe the unexplore for me now to the v2 experiment https://github.com/RichardKnop/machinery#v2-experiment) but backing to this proyect... i would like to explore two options:

Machinery is nice. We've used it at work and ended up writing a light weight alternative, https://github.com/kalbhor/Tasqueue which specifically addresses some pitfalls in Machinery. However, it may not be ideal to have to write plugins/servers that depend on a distributed queue. Maybe a simple HTTP webhook system will do? POST from sql-jobber to arbitrary-HTTP-driver-webook endpoint to get results.

  1. Standard Go plugin is a really poor system and is only meaningful in a closed environment. All "plugins" have to always be recompiled against master any time the master changes. I'd tried this approach in OTPGateway, but abandoned it for a simple HTTP webhook system because of how impractical and annoying this was: knadh/otpgateway@97baef7

  2. WASM is interesting. Haven't toyed with Go/WASM yet.

Thinking about it, a simple HTTP webhook system maybe ideal, just like what I ended up with in OTPGateway. Users can write an HTTP server in any language they prefer and needn't know Go.