[out-of-space-prevention] stop writes to the replica at configurable disk usage threshold
denesb opened this issue · comments
Simple hit-the-wall behaviour. When disk usage goes above the configured threshold, stop all user writes to the replica. Allow writes to system tables so tablet state machine can be advanced and tablets can be migrated away.
We have to consider all possible source of disk writes:
- regular mutation writes
- streaming
- repair
- sstable upload
- hints
- snapshots
- compactions
We will probably make some exceptions for system tables too, e.g. don't accept writes to system_tracing
keyspace.
This feature will only work in tablet-only cluster. With vnodes, there are too many things to consider, especially around bootstrapping new nodes with RBNO.
With tablet-only cluster, only a few system tables use vnodes and these have small amount of data, so allowing RBNO (which may bring in data to the local node too) is safe.
Ref #14067