Proposal: CRD Synchronization in Multi-tenancy Syncer

Question

Proposal: CRD Synchronization in Multi-tenancy Syncer

weiling61 opened this issue 3 years ago · comments

Wei Ling commented 3 years ago

CRD Synchronization in Multi-tenancy Syncer

Objective

The key objectives of this document are:

Install annotated super cluster CRDs in all tenants virtual clusters.
Dynamically build CR scheme and CR clients.
Construct an example CR syncer using current syncer infrastructure, and keep the CR syncer code separate.
CR Syncer cannot be registered dynamically. It should be built together with rest of multi-tenancy syncer.

Overview

In this design, a customized syncer that supports both custom resources (CRs) and standard K8S resources will be built based on multi-tenant syncer. Proprietary CR syncers, main and resource type selection logics are kept in a private repo. Current Multi-tenant Syncer code used as libraries.

The CR syncer code will be located at “multi-tenancy/incubator/virtualcluster/example” ?

A new config option will be introduced to enable CRD Syncer in multi-tenancy syncer code.

CRD Installation

CRDs with annotation: tenancy.x-k8s.io/super.public will be installed into tenant’s virtual cluster as soon as the virtual cluster is ready.
VC-syncer will keep monitoring new CRD in super cluster and deploy a copy to all tenant virtual clusters.

CR Scheme Building

Controller runtime scheme builder is used to add CR scheme into existing scheme singleton.
Dynamic Scheme modification is allowed.

CR Syncer Construction and Registration

Current vc-syncer requires a dedicated resource syncer component to be registered at startup.

CR specific syncer components, including CR controller, CR DWS, and CR patroller, need to be compiled together with existing multi-tenant syncer code. No dynamic CR syncer generation and registeration will be allowed.

CR Client Construction

CR client is constructed during CR resource controller creation using Controller runtime client.

Resource Type Selection

Resource Syncer initialization module is responsible to initialize (register) some or all existing resource synchronization features for a customized syncer.

CR Fake Client

CR relies on controller runtime fake client to perform tests.

Current multi-tenant syncer test framework does not support cross resource type testing, e.g. test interaction among multiple runtime objects of different resource types. In the future, a list of virtual and super cluster runtime objects will be supported to handles theses scenarios.

Christopher Hein · Answer 1 · Wed Feb 17 2021 02:38:32 GMT+0800 (China Standard Time)

/cc @Fei-Guo

Fei Guo · Answer 2 · Wed Feb 17 2021 04:02:53 GMT+0800 (China Standard Time)

For CRD installation, will only the CRD that has the compiled resource syncer be synced to virtualcluster?

Christopher Hein · Answer 3 · Wed Feb 17 2021 04:40:03 GMT+0800 (China Standard Time)

The idea is like storageclasses & priorityclasses this would include doing the work to add a new resource syncer to the vc-syncer that allows it to auto-sync CRDs that have the tenancy.x-k8s.io/super.public: true label. THis should be built as an optional like priorityclasses too so it can default to off.

Fei Guo · Answer 4 · Wed Feb 17 2021 05:29:37 GMT+0800 (China Standard Time)

The idea is like storageclasses & priorityclasses this would include doing the work to add a new resource syncer to the vc-syncer that allows it to auto-sync CRDs that have the tenancy.x-k8s.io/super.public: true label. THis should be built as an optional like priorityclasses too so it can default to off.

Yes. I was thinking the same. This sounds like a separate issue for "CustomResourceDefinition" kind resource population rather than the state synchronization for CR instances.

For the repo, I think we can create a directory called /virtualclusteer/crdextension to hold all source codes, example crd, example clients, tutorial etc.

Christopher Hein · Answer 5 · Wed Feb 17 2021 05:50:20 GMT+0800 (China Standard Time)

Yeah, @weiling61 maybe we should make two issues and track the CRD resource syncer separate from the example codes and do them in separate PRs.

Wei Ling · Answer 6 · Thu Feb 18 2021 02:09:52 GMT+0800 (China Standard Time)

@Fei-Guo @christopherhein Agree to make 2 PR from it. 1) the CRD syncing is in multi-tenancy code, more specifically will be in listener/ package, in AddCluster(), WatchCluster() .. An option config will be used to active this feature 2) Will create a new repo to perform CR synchronization, for now, only compiled CR Syncer will be supported. No dynamic CR syncer registration.

jerryzhuang · Answer 7 · Thu Feb 18 2021 11:32:20 GMT+0800 (China Standard Time)

oh i got you. it is much easier to add custom CR syncer after recent refactoring. feel free to add an example CR synchronization code

fejta-bot · Answer 8 · Wed May 19 2021 12:16:15 GMT+0800 (China Standard Time)

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

Christopher Hein · Answer 9 · Tue May 25 2021 06:47:38 GMT+0800 (China Standard Time)

/close

@weiling61 correct me if I'm wrong this was accomplished when you added https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/customresource-syncer.md

Kubernetes Prow Robot · Answer 10 · Tue May 25 2021 06:47:44 GMT+0800 (China Standard Time)

@christopherhein: Closing this issue.

In response to this:

/close

@weiling61 correct me if I'm wrong this was accomplished when you added https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/customresource-syncer.md

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.