openyurtio / openyurt

What happened:
I got a question about etcd storage of yurthub. When replacing, we can look up the keys in etcd based on the prefix, and then perform batch operations. We just need to ensure that the batch operations are atomic. It doesn't matter if the getting and subsequent writing aren't atomic, right?

openyurt/pkg/yurthub/storage/etcd/storage.go

Lines 76 to 80 in 21d7f68

    
           // localComponentKeyCache persistently records keys owned by different components 
        
           // It's useful to recover previous state when yurthub restarts. 
        
           // We need this cache at local host instead of in etcd, because we need to ensure each 
        
           // operation on etcd is atomic. If we store it in etcd, we have to get it first and then 
        
           // do the action, such as ReplaceComponentList, which makes it non-atomic.

Anything else we need to know?:

Environment:

OpenYurt version:
Kubernetes version (use kubectl version):
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

others
/kind question

@Congrool Hi, I noticed you're the author of this commit, could you explain it to me?

By the way, when would scheme not be able to recognize gvk?

openyurt/pkg/yurthub/cachemanager/storage_wrapper.go

Lines 116 to 121 in 3713163

    
           var UnstructuredObj runtime.Object 
        
           if scheme.Scheme.Recognizes(*gvk) { 
        
           	UnstructuredObj = nil 
        
           } else { 
        
           	UnstructuredObj = new(unstructured.Unstructured) 
        
           }

@vie-serendipity

The localComponentKeyCache works like a shim to fill the gap between local cache and etcd cache. We know that the original local cache (which caches resources files under /etc/kubernetes/cache) can distinguish resources by component, whose key is /component/gvr/namespace/name. However, the cache in etcd is stored in different format like /gvr/namespace/name, which does not know anything about component.

Because of the legacy problem, the yurthub cache is tightly coupled with component. Some functions of storage semantically use component as input, for example ListResourceKeysOfComponent, ReplaceComponentList. As a solution, we provide the localComponentKeyCache which record the component info for the etcd cache.

We store the localComponentKeyCache at disk persistently for yurthub to remember resources it owns in etcd cache, it's useful when deleting resources. For example, when the node calling ReplaceComponentList, we should know which resources should be added, updated and deleted, expecially the deletion. Assuming that if the yurthub restart without knowing which resources it owned in etcd cache, which were A, B, C, after relisting, it gets the latest A, B from APIServer, and C is deleted which yurthub does not know. Thus the yurthub will leave C retain in the etcd cache and possibly never be deleted.

BTW, the atomicity of operations is totaly depends on Txn capability provided by etcd. You can find that each operation provided by the etcd storage only call s.client.Txn()...Commit() only once. That's the key to keep them atomic.

However, the cache in etcd is stored in different format like /gvr/namespace/name, which does not know anything about component.

Why not using /component/gvr/namespace/name as key when caching objects to etcd? Is it possible to remove localComponentKeyCache by this way?

Unfortunately, we can't, because the APIServer only recognizes format /gvr/namespace/name. If we save keys in other format, we cannot use kubectl to get resources through yurt-coordinator. If we save both formats in etcd cache, for example two records in format of /gvr/namespace/name and /component/gvr/namespace/name for one pod, it also does not work. As I said in comment, it will break the atomicity of etcd storage operations, because we need commit twice, one for get and one for other operation(like deletion).

// We need this cache at local host instead of in etcd, because we need to ensure each
// operation on etcd is atomic. If we store it in etcd, we have to get it first and then
// do the action, such as ReplaceComponentList, which makes it non-atomic.

@Congrool It's clear. Thanks!

	// localComponentKeyCache persistently records keys owned by different components
	// It's useful to recover previous state when yurthub restarts.
	// We need this cache at local host instead of in etcd, because we need to ensure each
	// operation on etcd is atomic. If we store it in etcd, we have to get it first and then
	// do the action, such as ReplaceComponentList, which makes it non-atomic.

	var UnstructuredObj runtime.Object
	if scheme.Scheme.Recognizes(*gvk) {
	UnstructuredObj = nil
	} else {
	UnstructuredObj = new(unstructured.Unstructured)
	}

[Question] what's the necessity of localComponentKeyCache in etcd storage of yurthub?