[Bug]: Node Storage pruning may not be working as expected (DEVNET,TESTNET,MAINNET)
frankf1957 opened this issue · comments
Contact Details
No response
Description
Node Storage pruning may not be working as expected (DEVNET,TESTNET,MAINNET)
Node version
Elrond Node CLI App version v1.3.42.0-0-47b8513d6a/go1.17.6/linux-amd64/b3f0d25a83
Host machine
Hetzner - dedicated
Steps to reproduce
I received a disk space utilization alert on my DEVNET system over the weekend. Investigation showed that Epoch_nnnnn directories we not being cleaned (removed) on my DEVNET nodes. There were 98 Epochs for each DEVNET validator node. I checked my TESTNET system and noticed the same behavior.
I just looked on Mainnet this morning and see that Mainnet seems to be affected the same. (although the count of Epoch_nnn directories is not so high yet).
All nodes are validators, I have no observers.
On which network is the bug manifesting itself?
Mainnet
Relevant log output
mainnet
=======
Elrond Node CLI App version v1.3.42.0-0-47b8513d6a/go1.17.6/linux-amd64/b3f0d25a83
$ elrond-nodes/node-0$ cat config/config.toml | egrep -v '^$|#' | sed -n '/\[StoragePruning\]/,/^\[/p'
[StoragePruning]
Enabled = true
ValidatorCleanOldEpochsData = true
ObserverCleanOldEpochsData = false
AccountsTrieCleanOldEpochsData = true
AccountsTrieSkipRemovalCustomPattern = "%50"
NumEpochsToKeep = 4
NumActivePersisters = 3
FullArchiveNumActivePersisters = 10
[MiniBlocksStorage]
$ elrond-nodes/node-0$ ls -ld db/1/Epoch_*
drwx------ 3 elrond elrond 4096 Sep 18 15:18 db/1/Epoch_780
drwx------ 3 elrond elrond 4096 Sep 19 15:18 db/1/Epoch_781
drwx------ 3 elrond elrond 4096 Sep 20 15:18 db/1/Epoch_782
drwx------ 3 elrond elrond 4096 Sep 21 15:18 db/1/Epoch_783
drwx------ 3 elrond elrond 4096 Sep 22 15:18 db/1/Epoch_784
drwx------ 4 elrond elrond 4096 Sep 26 16:07 db/1/Epoch_785
drwx------ 4 elrond elrond 4096 Sep 26 16:07 db/1/Epoch_786
drwx------ 4 elrond elrond 4096 Sep 26 16:07 db/1/Epoch_787
drwx------ 4 elrond elrond 4096 Sep 26 16:07 db/1/Epoch_788
drwx------ 3 elrond elrond 4096 Sep 27 15:19 db/1/Epoch_789
drwx------ 3 elrond elrond 4096 Sep 28 15:19 db/1/Epoch_790
drwx------ 3 elrond elrond 4096 Sep 29 15:19 db/1/Epoch_791
drwx------ 3 elrond elrond 4096 Sep 30 15:19 db/1/Epoch_792
drwx------ 3 elrond elrond 4096 Oct 1 15:19 db/1/Epoch_793
drwx------ 3 elrond elrond 4096 Oct 2 15:19 db/1/Epoch_794
ls -ld db/1/Epoch_* | wc -l
15
devnet
======
Elrond Node CLI App version D1.3.43.0-0-3b2f905ab2/go1.17.6/linux-amd64/be36dd5835
elrond-nodes/node-0$ cat config/config.toml | egrep -v '^$|#' | sed -n '/\[StoragePruning\]/,/^\[/p'
[StoragePruning]
Enabled = true
ValidatorCleanOldEpochsData = true
ObserverCleanOldEpochsData = false
AccountsTrieCleanOldEpochsData = true
AccountsTrieSkipRemovalCustomPattern = "%50"
NumEpochsToKeep = 4
NumActivePersisters = 3
FullArchiveNumActivePersisters = 10
[MiniBlocksStorage]
elrond-nodes/node-0$ ls -ld db/D/Epoch_*
drwx------ 3 elrond elrond 4096 Oct 1 12:43 db/D/Epoch_2231
drwx------ 4 elrond elrond 4096 Oct 1 21:08 db/D/Epoch_2232
drwx------ 4 elrond elrond 4096 Oct 1 21:08 db/D/Epoch_2233
drwx------ 4 elrond elrond 4096 Oct 1 21:08 db/D/Epoch_2234
drwx------ 4 elrond elrond 4096 Oct 1 21:08 db/D/Epoch_2235
drwx------ 3 elrond elrond 4096 Oct 1 22:43 db/D/Epoch_2236
drwx------ 3 elrond elrond 4096 Oct 2 00:43 db/D/Epoch_2237
drwx------ 4 elrond elrond 4096 Oct 2 09:00 db/D/Epoch_2238
drwx------ 4 elrond elrond 4096 Oct 2 09:00 db/D/Epoch_2239
drwx------ 4 elrond elrond 4096 Oct 2 08:59 db/D/Epoch_2240
drwx------ 4 elrond elrond 4096 Oct 2 09:00 db/D/Epoch_2241
drwx------ 3 elrond elrond 4096 Oct 2 10:44 db/D/Epoch_2242
drwx------ 4 elrond elrond 4096 Oct 2 19:02 db/D/Epoch_2243
drwx------ 4 elrond elrond 4096 Oct 2 19:02 db/D/Epoch_2244
drwx------ 4 elrond elrond 4096 Oct 2 19:02 db/D/Epoch_2245
drwx------ 4 elrond elrond 4096 Oct 2 19:02 db/D/Epoch_2246
drwx------ 3 elrond elrond 4096 Oct 2 20:45 db/D/Epoch_2247
drwx------ 3 elrond elrond 4096 Oct 2 22:45 db/D/Epoch_2248
drwx------ 3 elrond elrond 4096 Oct 3 00:45 db/D/Epoch_2249
drwx------ 3 elrond elrond 4096 Oct 3 02:45 db/D/Epoch_2250
drwx------ 3 elrond elrond 4096 Oct 3 04:45 db/D/Epoch_2251
drwx------ 3 elrond elrond 4096 Oct 3 06:45 db/D/Epoch_2252
drwx------ 3 elrond elrond 4096 Oct 3 08:45 db/D/Epoch_2253
drwx------ 3 elrond elrond 4096 Oct 3 10:45 db/D/Epoch_2254
drwx------ 3 elrond elrond 4096 Oct 3 12:45 db/D/Epoch_2255
elrond-nodes/node-0$ ls -ld db/D/Epoch_* | wc -l
25
Code of Conduct
- I agree to follow this project's Code of Conduct
Hello! Thanks for reporting this. Can you also attach logs with log level *:DEBUG
?
catched and fixed in #4549
@frankf1957 can you confirm that the issue is fixed and proceed with closing this issue?
Checked and fixed in all environments.
No further action is required.
Closing issue.