Trying to delete a table while a disk is in flood stage fails

[Georg Traar]

CrateDB version

5.6.2

CrateDB setup information

Single node
8 GiB disk

Problem description

When a nodes reaches the disk flood stage water mark tables get "blocks.read_only_allow_delete" set to true.
Trying to delete the table with drop table fails with:

ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

Note

one can manually remove the block with

ALTER TABLE my_import SET ("blocks.read_only_allow_delete" = 'false');

then the DROP TABLE works as expected.

Steps to Reproduce

1. Fill up a node to flood stage water mark.
2. Try to drop the table

   DROP TABLE my_import;
   ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

Note

It seems like this cannot be reproduced by setting the "blocks.read_only_allow_delete" manually to false;

Actual Result

DROP TABLE my_import;
--ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

Expected Result

DROP TABLE my_import;
--DROP OK, 1 row affected (0.053 sec)

[Marios Trivyzas]

doesn't this relate to: #14963 ?

[Georg Traar]

doesn't this relate to: #14963 ?

How can I free up space, if I can't delete a table :)

[Marios Trivyzas]

As we state now in the docs, if the table is read_only, you need to reset this flag, or set write to true to be able to delete. the blocks.read_only_allow_delete is a flag that is switched back and forth automatically if the watermark is reached/unershot.

[Marios Trivyzas]

Currently, blocks.read_only_allow_delete flag is switched automatically to true when the watermark is reached. At this point, user can only delete rows from tables, and nothing more. To drop a table you need to change Metadata, and currently blocks.read_only_allow_delete blocks such operations, probably because we assume that if there is no disk space left, changing the metadata could potentially cause corruptions.

Allowing only delete of rows, if space is exhausted, won't help, as deleting rows, means adding data to translog and then new segments, which will require more space (until afterwards, segments are merged and space is indeed freed up).

We should probably allow to drop tables when blocks.read_only_allow_delete is automatically set to true, to free up space, BUT we need to investigate and make sure that changing cluster state metadata in such conditions (very low, or even no space left on device situation) is safe.

Thank you @seut for the discussion on this topic.

[Mathias Fußenegger]

Relates a bit to #12710. I extended the info there to close this as duplicate