Using native APIs instead of using command line interfaces of Terraform, Helm, Kubectl, and Docker [feature request]
d4h0 opened this issue · comments
Hi,
I know, it's a goal of the project to get rid of using command line interfaces.
After reading issue #25, I did some research, and I thought it would be useful to start this issue to collect relevant information, and to track progress.
For kubectl u/garypen proposed kube.rs and k8s-openapi.
For the other tools: Bindings most likely also would be useful to the ecosystems of other programming languages as well. Maybe a solution that's based on gRPC, WASM, or similar would make it easier to secure support from the maintainers of these tools?
Basically, there would need to be a definition of the API (i.e. in JSON) and a gRPC server (or similar). New clients then could automatically generate the native API from the protocol definition (which would be almost trivial in Rust with serde).
Go has experimental WASM support.
Terraform already has a gRPC server for plugins, but I'm not sure if it could be used to implement a native API for what the command line tool does.
Another approach would be to use different tools, that have better APIs.
Here is a list of alternatives to Terraform, for example:
https://en.wikipedia.org/wiki/Infrastructure_as_code#Tools
Pulumi looks promising. They have already an SDK for TypeScript, JavaScript, Python, and Go, which they seem to have implemented via a gRPC server.
Currently, no support for Rust is planned, but community contributed support would be welcome.
I found the following resources that show how to implement an SDK for a new language:
What do you guys think?
I'd love to use Rust for my Infrastructure as Code needs, but using command line interfaces internally is a dealbreaker for me (and most likely for many other people, as well).
Hi @d4h0 ,
Thanks a lot for all of your propositions. They all look interesting. To get a context of history, when we started Qovery, we first did the engine in Python and we were using as much as we could dedicate libraries to avoid shell out. The drawback for example with kubectl was the lack of strong support. We encountered unattended behavior with several of those libs.
While rewriting it into Rust, we didn't want to make the same mistake. We want to take the required time to test and ensure it works as expected. What we want is to get strong support for the lib we're using. We're a small team yet and we're trying to be pragmatic. Using shell out with json export works far better than the previous Engine version. And the reason is simple, CLIs support for this kind of tool is generally very well supported, and from what we see since for the last months, we confirm it.
So even if it's not what we want in the long term, and will want to switch to dedicated Rust libs as you suggested, we've got other priorities at the moment and we're confident it will be ok for the next coming months. Indeed we're really interested in feedback on production usage for the proposed libs you've made. So if you have some inputs, feel free to talk about it :).
@deimosfr: Thanks for the feedback!
Using shell out with json export works far better than the previous Engine version.
Does this mean, the command line tools print JSON to stdout, which then is interpreted by Qovery engine? I haven't looked at the actual source code (or used any of the command line tools). I think, this approach would have comparable reliability to a gRPC-based API, as described above.
Indeed we're really interested in feedback on production usage for the proposed libs you've made. So if you have some inputs, feel free to talk about it :).
Unfortunately, I'm very inexperienced with anything related to cloud computing, so I don't have anything to say at the moment. But I'm pretty interested in Pulumi (after researching it a bit more), and maybe will give it a try. So maybe in the future I have something to report.
I stumbled upon a few things that are probably relevant to this discussion:
The "Pulumi vs. Terraform" page looks interesting. It says that Pulumi supports Terraform providers, so Pulumi is a superset of Terraform.
"From Kubernetes or Helm YAML" also could be interesting. Basically, with Pulumi it's possible to reuse existing Kubernetes and Helm YAML configuration files.
It seems, Helm charts can be used with Pulumi.
I also found pulumi/pulumi-kubernetes, which might be relevant.
As mentioned above, I'm basically clueless when it comes to cloud computing, so the above might not be useful for Qovery engine.
Does this mean, the command line tools print JSON to stdout, which then is interpreted by Qovery engine? I haven't looked at the actual source code (or used any of the command line tools).
Exactly. We do it as much as we can (if the CLI support it, for example on Terraform, it's supported on the plan command but not apply)
The "Pulumi vs. Terraform" page looks interesting. It says that Pulumi supports Terraform providers, so Pulumi is a superset of Terraform.
There is one important point I forgot to talk about. Most of our clients want to easily read the configuration we provide, so the used tools should remain "classic". Most of Ops/DevOps/SREs doing Cloud and Kubernetes are using helm and Terraform. In nowadays, asking to read a Terrafom config is more straightforward than anything else for this kind of population.
We want to keep it really simple for developers, allowing them to easily deploy quickly and efficiently their app with dependencies. But for Production usage, companies have Ops/DevOps/SREs, to control and customize based on what the company needs.
I'll be interested to see as well, the adoption of Pulumi solution. To me, Terraform is super as soon as we do simple things because it's using a DSL behind the scene (HCL). When we start having more and more stuffs with dependencies, options etc...we sruggle with the DSL and need a programmatic language. This is why we made the choice to use Rust to prepare the maximum of the job, then use a templating solution on top of it (Jinja). This allows us to make complex choices and keep Terraform config simple and readable.
To me Pulumi idea is super when it comes to perform complex stuffs and when Terraform DSL is starting to show its limits. As exemplained above, we need to show something standard in the ecosystem, so at the moment Terraform is the only option we can consider to be honest :/