Giters

xtruder / kubenix

Replaced by https://github.com/hall/kubenix

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Kubenix refactoring

offlinehacker opened this issue · comments

commented

Description

Kubenix is in stage of refactoring due all the things we discovered at @GateHubNet while we were using kubenix for complex production deployments. We made kubenix public even before, so someone could reuse ideas, but it was still highly specialized tool for our needs. This refactoring will make kubenix generally available, but will also introduce breaking changes.

For work in progress branch please look into: https://github.com/xtruder/kubenix/tree/kubenix-2.0

New features

  • Improved and reimplemented submodule system with versioning support
  • Pre-generated kubernetes module definitions with explicit API versioning support
  • Pre-generated istio module definitions
  • Support for k8s 1.11,1.12,1.13
  • Validation for io.k8s.apimachinery.pkg.api.resource.Quantity and similar types
  • Helm integration
  • Initial testing framework
  • Improved unit tests
  • Improved e2e tests
  • CI integration (travisci and hercules-ci)
  • More examples
  • Usage documentation
  • Submodule documentation
  • Generated documentation for all kubernetes options
  • Legacy kubenix 1.0 support
commented

Today i added initial istio pre-generated modules. Generator is really ugly, but gets a job done.

commented

Implemented a simple testing framework and initial tests.

commented

Initial support for e2e testing has been implemented.

commented

Helm has been integrated in ba1144a thanks to @matejc who implemented: https://github.com/matejc/helm2json example usage is here: ba1144a#diff-b025248827de015784990d5577d75c8eR32

commented

I'm trying to the 2.0 branch for a little test ride but i'm having some difficulties figuring out what the API has changed to now. In my code I currently have something like the following to generate different configurations:

{
  buildConfig = t: kubenix.buildResources { configuration = import ./configuration.nix { type = t; }; };
}

Looking at the code I tried something like..

  buildConfig = t: builtins.toJSON (kubenix.buildResources { 
    module = import ./configuration.nix { type = t; }; 
    modules = [ kubenix.k8s ];
  });

but that doesn't quite do the right thing and I have the feeling i am doing something very wrong here 😅

commented

@offlinehacker thank you so much :-)

commented

I did some refactoring on codebase, fixed a bug and added e2e test for nginx-deployment example (test.nix file). Will add more tests tomorrow, including for helm and kubenix submodules, stay tuned.

commented
  • CI integration

@offlinehacker What's the plan with regards to CI integration? I'm very much interested in replacing Helm in a GitOps managed environment with KubeNix. Is that feasible?

cc @lzaldivarkt

commented

@kalbasit You are probably taking about e2e tests for kubenix 2.0, it's already implemented. It's spawns kubernetes cluster in qemu using nixos testing framework and nixos kubernetes module. Take a look here for example: https://github.com/xtruder/kubenix/blob/kubenix-2.0/examples/nginx-deployment/test.nix#L8 and README in same folder to see how you can run test. Also you can find bunch of unit and e2e tests for kubenix 2.0 here: https://github.com/xtruder/kubenix/tree/kubenix-2.0/tests

commented

@kalbasit also implementation of testing is defined in this kubenix module: https://github.com/xtruder/kubenix/blob/kubenix-2.0/modules/testing.nix

commented

As from implementation point of view kubenix 2.0 is getting more stable and i have implemented most if not all features we need, i will work now on documentation, more examples, and on refactoring of https://github.com/xtruder/kubenix-modules for kubenix 2.0

commented

@kalbasit You are probably taking about e2e tests for kubenix 2.0, it's already implemented. It's spawns kubernetes cluster in qemu using nixos testing framework and nixos kubernetes module. Take a look here for example: /examples/nginx-deployment/test.nix@kubenix-2.0#L8 and README in same folder to see how you can run test. Also you can find bunch of unit and e2e tests for kubenix 2.0 here: /tests@kubenix-2.0

This is truly awesome!!

However, I was actually talking about JenkinsX and Prow for managing and testing pull requests against an application project on GitHub. JenkinX offers deployment of applications to multiple environments (dev, staging, prod) via GitOps, which are really just Git repositories where each repositories describe the state of the world on k8s using Helm. See https://jenkins-x.io/about/concepts/#architecture and https://www.weave.works/technologies/gitops/ for more information on how this works.

commented

This is work in progress refactoring of kubenix-submodules, i'm currently working on to make reusable set of submodules for kubenix 2.0: xtruder/kubenix-modules#88

commented

@kalbasit We have similar setup, i will let you know how we integrate kubenix 2.0 and make an example of such setup

commented

I am running kubenix tests on hercules-ci: https://hercules-ci.com/github/xtruder/kubenix. I still did not manage to run e2e tests due some issues, but hopefully to resolve these soon :)

commented

You are probably taking about e2e tests for kubenix 2.0, it's already implemented. It's spawns kubernetes cluster in qemu using nixos testing framework and nixos kubernetes module.

EPIC

commented

Is the 2.0 base stable now? It seems to me like it could be merged into master. There are always more things to do.

commented

ping :)

commented

KubeNix 2.0 branch was just merged into master. Documentation and tooling around still needs to be implemented and kubenix 2.0 submodule system might be refactored.

commented
commented

KubeNix 2.0 branch was just merged into master. Documentation and tooling around still needs to be implemented and kubenix 2.0 submodule system might be refactored.

If I started using kubenix now, what parts should I expect to break soon.

commented

Since I stopped working for company where we used kubenix, I also halted development of it, and am unsure if they will use it in the future.

The last thing I was working on was better e2e tests using kubetest python library, which I got it somehow finished, but I did not push it yet.

@kolloch What I was asking myself is if some parts were overengeneered, especially submodule system. What I was thinking if it is really needed or could use functional approach instead of declarative to define/import submodules.

commented

@offlinehacker Would you be willing to cede ownership/maintenance so somebody else can take this over and develop it into the future. The future is bright (nickel!)

commented

@blaggacao looking at forks seems like https://github.com/GTrunSec/kubenix is the most active currently. I wish @offlinehacker would just officially transfer ownership. When this project was being developed I didn't have a need for this project and now while I do the project seems to be dead :(

commented

@takeda my conclusion is: don't use the module system as config validator. Use jsonschema validation. Much more interoperable.

Unfortunately, the upstream nix patch to add jsonschema support hasn't been accepted.

commented

@takeda my conclusion is: don't use the module system as config validator. Use jsonschema validation. Much more interoperable.

Unfortunately, the upstream nix patch to add jsonschema support hasn't been accepted.

Are you saying you gave up on kubenix? Or are you suggesting it should be designed differently? Could you elaborate? Thanks!

commented

This repo has been deprecated, since I stopped maintaining it some time ago. There is a fork maintained by @hall available at https://github.com/hall/kubenix, that has better documentation and looks like a way further.