Metadata is not deleted if a non-standard path is used.
silentsokolov opened this issue · comments
Metadata is not deleted if a non-standard path is used.
We dont use /var/lib/clickhouse/ instead use custom path (/mnt/hot-1/clickhouse/lib/)
Version 2.3.2
$ clickhouse-backup delete local test1
2024/04/05 11:50:03.910719 info SELECT path, any(name) AS name, any(type) AS type FROM system.disks GROUP BY path logger=clickhouse
2024/04/05 11:50:03.914149 debug remove '/mnt/hot-1/clickhouse/lib/backup/test1' logger=RemoveBackupLocal
2024/04/05 11:50:03.933050 debug remove '/mnt/cold-1/clickhouse/data/backup/test1' logger=RemoveBackupLocal
2024/04/05 11:50:03.933112 debug remove '/mnt/hot-1/clickhouse/data/backup/test1' logger=RemoveBackupLocal
2024/04/05 11:50:03.933144 debug remove '/mnt/hot-2/clickhouse/data/backup/test1' logger=RemoveBackupLocal
2024/04/05 11:50:03.933160 info done backup=test1 duration=26ms location=local logger=RemoveBackupLocal operation=delete
2024/04/05 11:50:03.933206 info clickhouse connection closed logger=clickhouse
Version 2.4.35
$ clickhouse-backup delete local test1
2024/04/05 11:47:02.850957 debug b.skipIfTheSameRemoteBackupPresent return skip=true logger=RemoveBackupLocal
2024/04/05 11:47:02.850972 debug remove '/mnt/cold-1/clickhouse/data/backup/test1' logger=RemoveBackupLocal
2024/04/05 11:47:02.850995 debug remove '/mnt/hot-1/clickhouse/data/backup/test1' logger=RemoveBackupLocal
2024/04/05 11:47:02.851030 debug remove '/mnt/hot-2/clickhouse/data/backup/test1' logger=RemoveBackupLocal
2024/04/05 11:47:02.851054 info done backup=test1 duration=145ms location=local logger=RemoveBackupLocal operation=delete
2024/04/05 11:47:02.851132 info clickhouse connection closed logger=clickhouse
/mnt/hot-1/clickhouse/lib/ didn't cleanup
AND
metadata stored in /var/lib/clickhouse/ :(
could you share
SELECT * FROM system.disks FORMAT Vertical
Row 1:
──────
name: default
path: /mnt/hot-1/clickhouse/lib/
free_space: 22345788416
total_space: 31023119360
unreserved_space: 22345788416
keep_free_space: 1024
type: Local
object_storage_type: None
metadata_type: None
is_encrypted: 0
is_read_only: 0
is_write_once: 0
is_remote: 0
is_broken: 0
cache_path:
Row 2:
──────
name: fast_1
path: /mnt/hot-1/clickhouse/data/
free_space: 22345788416
total_space: 31023119360
unreserved_space: 22345788416
keep_free_space: 1024
type: Local
object_storage_type: None
metadata_type: None
is_encrypted: 0
is_read_only: 0
is_write_once: 0
is_remote: 0
is_broken: 0
cache_path:
Row 3:
──────
name: fast_2
path: /mnt/hot-2/clickhouse/data/
free_space: 27915500544
total_space: 31023119360
unreserved_space: 27915500544
keep_free_space: 1024
type: Local
object_storage_type: None
metadata_type: None
is_encrypted: 0
is_read_only: 0
is_write_once: 0
is_remote: 0
is_broken: 0
cache_path:
Row 4:
──────
name: slow_1
path: /mnt/cold-1/clickhouse/data/
free_space: 57556446208
total_space: 82882247680
unreserved_space: 57556446208
keep_free_space: 1024
type: Local
object_storage_type: None
metadata_type: None
is_encrypted: 0
is_read_only: 0
is_write_once: 0
is_remote: 0
is_broken: 0
cache_path:
4 rows in set. Elapsed: 0.002 sec.
SELECT * FROM system.storage_policies
Row 1:
──────
policy_name: default
volume_name: default
volume_priority: 1
disks: ['fast_1','fast_2']
volume_type: JBOD
max_data_part_size: 0
move_factor: 0
prefer_not_to_merge: 0
perform_ttl_move_on_insert: 1
load_balancing: ROUND_ROBIN
Row 2:
──────
policy_name: fast
volume_name: fast
volume_priority: 1
disks: ['fast_1','fast_2']
volume_type: JBOD
max_data_part_size: 0
move_factor: 0
prefer_not_to_merge: 0
perform_ttl_move_on_insert: 1
load_balancing: ROUND_ROBIN
Row 3:
──────
policy_name: fast_slow
volume_name: fast
volume_priority: 1
disks: ['fast_1','fast_2']
volume_type: JBOD
max_data_part_size: 0
move_factor: 0
prefer_not_to_merge: 0
perform_ttl_move_on_insert: 1
load_balancing: ROUND_ROBIN
Row 4:
──────
policy_name: fast_slow
volume_name: slow
volume_priority: 2
disks: ['slow_1']
volume_type: JBOD
max_data_part_size: 0
move_factor: 0
prefer_not_to_merge: 0
perform_ttl_move_on_insert: 1
load_balancing: ROUND_ROBIN
Row 5:
──────
policy_name: slow
volume_name: slow
volume_priority: 1
disks: ['slow_1']
volume_type: JBOD
max_data_part_size: 0
move_factor: 0
prefer_not_to_merge: 0
perform_ttl_move_on_insert: 1
load_balancing: ROUND_ROBIN
SELECT metadata_path FROM system.tables WHERE database = 'system' AND metadata_path!='' LIMIT 1;
SELECT
d.path, any(d.name) AS name, any(d.type) AS type,
min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies
FROM system.disks AS d
LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s
ON s.disk = d.name GROUP BY d.path;
SELECT metadata_path
FROM system.tables
WHERE (database = 'system') AND (metadata_path != '')
LIMIT 1
Query id: c6dd91f5-484c-4014-aa82-3e3966ccdc3d
┌─metadata_path───────────────────────────────────────────────────────────────────────────────────────────────┐
1. │ /mnt/hot-1/clickhouse/lib/store/486/4862b2bd-4d72-452c-b2b9-91d72f79114d/aggregate_function_combinators.sql │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
SELECT
d.path,
any(d.name) AS name,
any(d.type) AS type,
min(d.free_space) AS free_space,
groupUniqArray(s.policy_name) AS storage_policies
FROM system.disks AS d
LEFT JOIN
(
SELECT
policy_name,
arrayJoin(disks) AS disk
FROM system.storage_policies
) AS s ON s.disk = d.name
GROUP BY d.path
Query id: 2d8aaa71-acda-4a12-bc5b-f4830370fe30
┌─path───────────────────────────────┬─name────┬─type──────────┬───────────free_space─┬─storage_policies───────────────┐
1. │ /mnt/hot-1/clickhouse/lib/ │ default │ Local │ 16183053312 │ [''] │
2. │ /mnt/cold-1/clickhouse/data/ │ slow_1 │ Local │ 36136311808 │ ['fast_slow','slow'] │
3. │ /mnt/hot-1/clickhouse/data/ │ fast_1 │ Local │ 16183053312 │ ['fast_slow','default','fast'] │
4. │ /mnt/hot-2/clickhouse/data/ │ fast_2 │ Local │ 26824367104 │ ['fast_slow','default','fast'] │
└────────────────────────────────────┴─────────┴───────────────┴──────────────────────┴────────────────────────────────┘
5 rows in set. Elapsed: 0.002 sec.
general:
remote_storage: s3
allow_empty_backups: true
backups_to_keep_local: -1
backups_to_keep_remote: 50
download_concurrency: 8
upload_concurrency: 8
use_embedded_backup_restore: false
root reason is default disk is not in any storage policy
duplicated with #845
It shall be fixed in v2.5 check
$docker_id=$(docker create altinity/clickhouse-backup:merge)
docker cp $docker_id:/bin/clickhouse-backup /tmp/clickhouse-backup
docker rm -f $docker_id
LOG_LEVEL=debug /tmp/clickhouse-backup delete local test1