When client request put operation and requested key is already exists, then the values will be overwritten or save the delta ?
kimsehwan96 opened this issue · comments
Hello, I have a question about etcd put operation with data_model document (https://etcd.io/docs/v3.5/learning/data_model/)
etcd stores the physical data as key-value pairs in a persistent b+tree. Each revision of the store’s state only contains the delta from its previous revision to be efficient. A single revision may correspond to multiple keys in the tree.
The document describes each revision of the store's state only contains the delta from its previous revision to be efficient.
So I thought if client requests put operation with existing key, then the etcd backend storage(bolt) should save the difference from previous revision's value and current requested value.
But in bbolt doucment about put (https://pkg.go.dev/go.etcd.io/bbolt#Bucket.Put)
If the key exist then its previous value will be overwritten
The document says its previous value will be overwritten.
I know that etcd client's put operation(e.g etcdctl put) and the bbolt go package's put operation is not same thing.
However, when I tested with etcdctl(put) to save same value in same key, the etcd's db files size increased linearly.
Is there anything I missed? Thank you!
- Maybe question like this should be opened in
etcd-io/etcd.. If this place is not for this, I will close this and move this question!
Hey @kimsehwan96 - Thanks for your question. I think this just comes down to the phrase:
"Each revision of the store’s state only contains the delta from its previous revision to be efficient."
Let's use a simplified example to illustrate the concept.
Let's say you have a key-value store managed by etcd with the following data:
Revision 1:
Key1: Value1
Key2: Value2
If we were to make two changes, an update to Key1 and inserting a new Key3 value the next revision might look like:
Revision 2 (after changes):
Key1: Value1_updated
Key3: Value3_added
Instead of storing the entire dataset again for Revision 2, etcd would only store the changes (deltas). This is based on my understanding, happy for another maintainer to jump in with another example if I am missing some nuance here.
Let me know if that helps @kimsehwan96?
@jmhbnz Thank you for your answer!
I have one more question.
Revision 1:
Key1: Value1
Key2: Value2
If I request "put" command to etcd (e.g with etcdctl) with same value Value1 to same key Key1 in above state, then Revision 2 data should be like this?
Revision 2 (after changes. The value was not really changed however put request was received):
Key1: Value1
I wonder that if the value was not really changed (simplifying, put Key1:Value1 twice) then, the newer revision has its data as shown above or just Revision 1 is there?
Thank you!
Hey @kimsehwan96 - We can understand this by using more detailed output from etcdctl as follows:
Start etcd server and confirm revision state
james ~ Documents etcd bin main 10:11:20
➜ ./etcd
james ~ Documents etcd bin main 10:11:21
➜ ./etcdctl get --prefix "" -w fields
"ClusterID" : 14841639068965178418
"MemberID" : 10276657743932975437
"Revision" : 1
"RaftTerm" : 2
"More" : false
"Count" : 0Add first key and confirm revision state
james ~ Documents etcd bin main 10:11:46
➜ ./etcdctl put Key1 Value1
OK
james ~ Documents etcd bin main 10:14:23
➜ ./etcdctl get --prefix "" -w fields
"ClusterID" : 14841639068965178418
"MemberID" : 10276657743932975437
"Revision" : 2
"RaftTerm" : 2
"Key" : "Key1"
"CreateRevision" : 2
"ModRevision" : 2
"Version" : 1
"Value" : "Value1"
"Lease" : 0
"More" : false
"Count" : 1
Repeat adding the key and confirm revision status
james ~ Documents etcd bin main 10:14:27
➜ ./etcdctl put Key1 Value1
OK
james ~ Documents etcd bin main 10:15:00
➜ ./etcdctl get --prefix "" -w fields
"ClusterID" : 14841639068965178418
"MemberID" : 10276657743932975437
"Revision" : 3
"RaftTerm" : 2
"Key" : "Key1"
"CreateRevision" : 2
"ModRevision" : 3
"Version" : 2
"Value" : "Value1"
"Lease" : 0
"More" : false
"Count" : 1
Notice that in the above example, revision does increase even though the data is the same. While the actual key values are the same, the delta between the revisions is still a put operation for that key which is recorded as it's own revision of said key. Long story short etcd will store both revisions, even though the key's value may be the same.