Bug in the Working space calculation
andrevinette opened this issue · comments
There seems to be a bug in the working space calculation.
Calculation seems ok on first 2 tests but when numbers get bigger it fails to return a proper result.
See results table attached.
calculator test.pdf
This is actually not a bug but a feature. The idea is that workspace is related to a full backup (you should always have space to execute an active full). However the bigger the subset of data, the more jobs you will have, thus the less need to provision an active full on all of them at the same time
Thus workspace does not grow linear with the ingest data. Imagine you need to backup 500TB, you would need to keep more than 200TB of workspace which is a lot.
This is why an easing function is built in starting from 0.2.8. So working space will not grow linear with the source configuration but will grow slower if the source size becomes bigger.
The idea is built with a bucket system. Once the bucket overflows, the next bucket is used.
0-10 TB = source data will be compressed and then multiplied with a factor of 1.05
10-20 TB = source data will be compressed and then multiplied with a factor of 0.66
20 - 100 TB = source data will be compressed and then multiplied with a factor of 0.4
100 - 500 TB = source data will be compressed and then multiplied with a factor of 0.25
500 TB+ = source data will be compressed and then multiplied with a factor of 0.10
If you have for example 5 TB of source data and 50% compression configured, this would result in:
5TB x 50/100 x 1.05 =~ 2.6 TB Workspace
If you have for example 50 TB of source data and 50% compression configured, this would result in:
10 TB x 50/100 x 1.05 + 10 TB x 50/100 x 0.66 + 30 TB x 50/100 x 0.4 =~ 5 + 3 + 6 = 14TB Workspace
If you have for example 500 TB of source data and 50% compression configured, this would result in:
10 TB x 50/100 x 1.05 + 10 TB x 50/100 x 0.66 + 80 TB x 50/100 x 0.4 + 400 TB x 50/100 x 0.25 =~ 5 + 3 + 16 + 50 = 74TB Workspace
You can see that even though 500TB is 100x 5TB, the work space does not grow linearly ( 74TB =~ 28x 2.6 )