Directory for table data store/d51/d51243d3-2961-40bd-8f99-d0338ad96769/ already exists after 44 times, please check your schema dependencies
dxygit1 opened this issue · comments
When I deleted the database and restarted ClickHouse, I encountered an error during rollback, indicating that the path exists.
curl "chi-clickhouse-cluster-dtstack-0-0:7171/backup/restore/shard0-increment-20240408044511?rm" -X POST
{"status":"acknowledged","operation":"restore","backup_name":"shard0-increment-20240408044511"}
curl "chi-clickhouse-cluster-dtstack-0-0:7171/backup/status"
{"command":"restore --rm shard0-increment-20240408044511","status":"error","start":"2024-04-09 05:38:56","finish":"2024-04-09 05:38:57","error":"can't create table `distributed_1`.`easydo_agent_monitor`: code: 57, message: Directory for table data store/d51/d51243d3-2961-40bd-8f99-d0338ad96769/ already exists after 44 times, please check your schema dependencies"}
I deleted the storage directory and restarted ClickHouse, but I still encountered an error.
curl "chi-clickhouse-cluster-dtstack-1-0:7171/backup/status"
{"command":"restore --rm shard1-increment-20240408050855","status":"error","start":"2024-04-09 06:36:04","finish":"2024-04-09 06:36:10","error":"can't create table `local_1`.`proc_stat_metrics`: code: 57, message: Directory for table data store/72b/72b5f6fe-0721-4c58-aac0-e5598e194e56/ already exists after 44 times, please check your schema dependencies"}
2024/04/09 06:40:20.991523 info CREATE TABLE local_1.proc_stat_metrics UUID '72b5f6fe-0721-4c58-aac0-e5598e194e56' (`_tenant_id_` Float64, `pid` Float64, `__instance_id__` LowCardinality(String), `_mode_` String, `mem_percent` Float64, `_at_date_` Date, `cmdline` String, `iowrite_bytes` Float64, `_receive_timestamp_` DateTime64(3, 'UTC'), `_custom_tags_` String, `page_faults_children_minor_faults` Float64, `ioread_bytes` Float64, `cpu_percent` Float64, `page_faults_children_major_faults` Float64, `open_file_descriptors` Float64, `run_time` Float64, `__appname__` LowCardinality(String), `mem_rss` Float64, `__source__` LowCardinality(String), `cpu_normalized_pct` Float64, `threads` Float64, `involuntary_ctx_switches` Float64, `__ci_resource__` LowCardinality(String), `mem_real` Float64, `__process_name__` LowCardinality(String), `__ip__` LowCardinality(String), `page_faults_major_faults` Float64, `ioread_count` Float64, `iowrite_count` Float64, `__instance_name__` LowCardinality(String), `page_faults_minor_faults` Float64, `_alias_` String, `name` String, `_at_timestamp_` DateTime64(3, 'UTC'), `voluntary_ctx_switches` Float64, `_tag_set_` Array(String), `__catalog__` LowCardinality(String), `_field_set_` Array(String), `status` String, `username` String, `mem_vms` Float64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/proc_stat_metrics/2024-04-08/{shard}/hits', '{replica}') PARTITION BY toString(_at_date_) ORDER BY (_at_date_, _at_timestamp_, intHash64(toInt64(toDateTime(_at_timestamp_)))) SAMPLE BY intHash64(toInt64(toDateTime(_at_timestamp_))) SETTINGS index_granularity = 8192 logger=clickhouse
2024/04/09 06:40:20.993446 info clickhouse connection closed logger=clickhouse
2024/04/09 06:40:20.993472 error metrics.ExecuteWithMetrics(restore) return error: can't create table `local_1`.`proc_stat_metrics`: code: 57, message: Directory for table data store/72b/72b5f6fe-0721-4c58-aac0-e5598e194e56/ already exists after 44 times, please check your schema dependencies logger=metrics
2024/04/09 06:40:20.993508 debug api.status.stop -> status.commands[5] == {ActionRowStatus:{Command:restore --rm shard1-increment-20240408050855 Status:error Start:2024-04-09 06:40:20 Finish:2024-04-09 06:40:20 Error:can't create table `local_1`.`proc_stat_metrics`: code: 57, message: Directory for table data store/72b/72b5f6fe-0721-4c58-aac0-e5598e194e56/ already exists after 44 times, please check your schema dependencies} Ctx:<nil> Cancel:<nil>} logger=status
2024/04/09 06:40:20.993540 error API /backup/restore error: can't create table `local_1`.`proc_stat_metrics`: code: 57, message: Directory for table data store/72b/72b5f6fe-0721-4c58-aac0-e5598e194e56/ already exists after 44 times, please check your schema dependencies logger=server
grep -m 1 -C 50 -i "code: 57" /var/log/clickhouse-server/clickhouse-server.err.log
clickhouse@chi-clickhouse-cluster-dtstack-1-0-0:/$ grep -m 1 -C 50 -i "code: 57" /var/log/clickhouse-server/clickhouse-server.err.log
11. DB::create(DB::StorageFactory::Arguments const&) @ 0x0000000012f55958 in /usr/bin/clickhouse
12. DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x000000001268513b in /usr/bin/clickhouse
13. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x0000000011d1b91e in /usr/bin/clickhouse
14. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
15. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
16. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122e2d15 in /usr/bin/clickhouse
17. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122de475 in /usr/bin/clickhouse
18. DB::TCPHandler::runImpl() @ 0x0000000013155799 in /usr/bin/clickhouse
19. DB::TCPHandler::run() @ 0x0000000013167b79 in /usr/bin/clickhouse
20. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5e154 in /usr/bin/clickhouse
21. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b5f351 in /usr/bin/clickhouse
22. Poco::PooledThread::run() @ 0x0000000015c95b87 in /usr/bin/clickhouse
23. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c93e5c in /usr/bin/clickhouse
24. ? @ 0x00007f41f0328609 in ?
25. ? @ 0x00007f41f024d353 in ?
2024.04.09 06:36:09.239587 [ 304 ] {d244a429-507e-4c0b-bc14-e7224485c684} <Warning> StorageDistributed (metrics_summary): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored.
2024.04.09 06:36:09.307556 [ 280 ] {} <Error> void DB::SystemLog<DB::AsynchronousMetricLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) [LogElement = DB::AsynchronousMetricLogElement]: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in create_directories: Permission denied ["/var/lib/clickhouse/cold/store"], Stack trace (when copying this message, always include the lines below):
0. std::system_error::system_error(std::error_code, String const&) @ 0x0000000018069955 in /usr/bin/clickhouse
1. std::__fs::filesystem::filesystem_error::filesystem_error[abi:v15000](String const&, std::__fs::filesystem::path const&, std::error_code) @ 0x000000000c68d121 in /usr/bin/clickhouse
2. void std::__fs::filesystem::__throw_filesystem_error[abi:v15000]<String&, std::__fs::filesystem::path const&, std::error_code const&>(String&, std::__fs::filesystem::path const&, std::error_code const&) @ 0x000000001801f17a in /usr/bin/clickhouse
3. std::__fs::filesystem::detail::(anonymous namespace)::ErrorHandler<bool>::report(std::error_code const&) const (.llvm.17456556289392299875) @ 0x0000000018022636 in /usr/bin/clickhouse
4. std::__fs::filesystem::__create_directories(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000018022e39 in /usr/bin/clickhouse
5. std::__fs::filesystem::__create_directories(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000018022ed5 in /usr/bin/clickhouse
6. std::__fs::filesystem::__create_directories(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000018022ed5 in /usr/bin/clickhouse
7. std::__fs::filesystem::__create_directories(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000018022ed5 in /usr/bin/clickhouse
8. DB::DiskLocal::createDirectories(String const&) @ 0x00000000114e1d19 in /usr/bin/clickhouse
9. DB::MergeTreeData::initializeDirectoriesAndFormatVersion(String const&, bool, String const&, bool) @ 0x0000000012bece0d in /usr/bin/clickhouse
10. DB::StorageMergeTree::StorageMergeTree(DB::StorageID const&, String const&, DB::StorageInMemoryMetadata const&, bool, std::shared_ptr<DB::Context>, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr<DB::MergeTreeSettings, std::default_delete<DB::MergeTreeSettings>>, bool) @ 0x0000000012f58fd6 in /usr/bin/clickhouse
11. DB::create(DB::StorageFactory::Arguments const&) @ 0x0000000012f557cb in /usr/bin/clickhouse
12. DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr<DB::Context>, std::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0x000000001268513b in /usr/bin/clickhouse
13. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x0000000011d1b91e in /usr/bin/clickhouse
14. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
15. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
16. DB::SystemLog<DB::AsynchronousMetricLogElement>::prepareTable() @ 0x00000000121005a0 in /usr/bin/clickhouse
17. DB::SystemLog<DB::AsynchronousMetricLogElement>::savingThreadFunction() @ 0x0000000012101b8a in /usr/bin/clickhouse
18. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::SystemLogBase<DB::AsynchronousMetricLogElement>::startup()::'lambda'()>(DB::SystemLogBase<DB::AsynchronousMetricLogElement>::startup()::'lambda'()&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c7609b2 in /usr/bin/clickhouse
19. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c725fa4 in /usr/bin/clickhouse
20. ? @ 0x00007f41f0328609 in ?
21. ? @ 0x00007f41f024d353 in ?
Cannot print extra info for Poco::Exception (version 23.8.8.21.altinitystable (altinity build))
2024.04.09 06:36:09.307656 [ 304 ] {6f12932c-35cf-470b-9e0a-328b42d7c91a} <Warning> StorageDistributed (system_metrics): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored.
2024.04.09 06:36:09.409480 [ 304 ] {ded26fd4-38f6-4ef9-8120-b04c28e0928b} <Warning> StorageDistributed (arms_frontend_log): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored.
2024.04.09 06:36:09.428884 [ 304 ] {55dc6cf7-b37f-4530-adb0-efe818032324} <Warning> StorageDistributed (easydo_agent_monitor): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored.
2024.04.09 06:36:09.728726 [ 304 ] {db11e494-903f-4f17-bd11-7d95d2048793} <Warning> StorageDistributed (easydo_collect_task_monitor): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored.
2024.04.09 06:36:09.853968 [ 304 ] {a5ed609b-f284-4987-a6a0-df96c6f943d9} <Warning> StorageDistributed (proc_stat_metrics): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored.
2024.04.09 06:36:09.865944 [ 304 ] {0d1cacc1-0939-4aa8-b5b0-b3e9e0e0f9d2} <Warning> StorageDistributed (test): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored.
2024.04.09 06:36:10.165324 [ 304 ] {8b79bb80-4a1d-412e-8ae5-de47eff7c39b} <Warning> StorageDistributed (system_attribute_meta): Storage policy for Distributed table has multiple volumes. Only default volume will be used to store data. Other will be ignored.
2024.04.09 06:36:10.176574 [ 304 ] {c27d069e-e20f-46e7-ac79-c1a386f9fa68} <Error> executeQuery: Code: 57. DB::Exception: Directory for table data store/287/2875f3f9-466f-43ac-938a-62fe69210777/ already exists. (TABLE_ALREADY_EXISTS) (version 23.8.8.21.altinitystable (altinity build)) (from [::1]:54944) (in query: CREATE TABLE local_1.test UUID '2875f3f9-466f-43ac-938a-62fe69210777' (`__appname__` LowCardinality(String), `__source__` LowCardinality(String), `_tenant_id_` Float64, `_message_` String, `_mode_` String, `__path__` LowCardinality(String), `__ip__` LowCardinality(String), `_collect_time_` Float64, `_at_date_` Date, `__hostname__` LowCardinality(String), `_receive_timestamp_` DateTime64(3, 'UTC'), `_custom_tags_` String, `_at_timestamp_` DateTime64(3, 'UTC'), `_tag_set_` Array(String), `_context_seq_` Float64, `_field_set_` Array(String)) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/test/2024-04-08/{shard}/hits', '{replica}') PARTITION BY toString(_at_date_) ORDER BY (_at_date_, _at_timestamp_, intHash64(toInt64(toDateTime(_at_timestamp_)))) SAMPLE BY intHash64(toInt64(toDateTime(_at_timestamp_))) SETTINGS index_granularity = 8192), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c644ed7 in /usr/bin/clickhouse
1. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x0000000007157e87 in /usr/bin/clickhouse
2. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x0000000011d1e14c in /usr/bin/clickhouse
3. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
4. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
5. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122e2d15 in /usr/bin/clickhouse
6. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122de475 in /usr/bin/clickhouse
7. DB::TCPHandler::runImpl() @ 0x0000000013155799 in /usr/bin/clickhouse
8. DB::TCPHandler::run() @ 0x0000000013167b79 in /usr/bin/clickhouse
9. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5e154 in /usr/bin/clickhouse
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b5f351 in /usr/bin/clickhouse
11. Poco::PooledThread::run() @ 0x0000000015c95b87 in /usr/bin/clickhouse
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c93e5c in /usr/bin/clickhouse
13. ? @ 0x00007f41f0328609 in ?
14. ? @ 0x00007f41f024d353 in ?
2024.04.09 06:36:10.176780 [ 304 ] {c27d069e-e20f-46e7-ac79-c1a386f9fa68} <Error> TCPHandler: Code: 57. DB::Exception: Directory for table data store/287/2875f3f9-466f-43ac-938a-62fe69210777/ already exists. (TABLE_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c644ed7 in /usr/bin/clickhouse
1. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x0000000007157e87 in /usr/bin/clickhouse
2. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x0000000011d1e14c in /usr/bin/clickhouse
3. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
4. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
5. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122e2d15 in /usr/bin/clickhouse
6. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122de475 in /usr/bin/clickhouse
7. DB::TCPHandler::runImpl() @ 0x0000000013155799 in /usr/bin/clickhouse
8. DB::TCPHandler::run() @ 0x0000000013167b79 in /usr/bin/clickhouse
9. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5e154 in /usr/bin/clickhouse
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b5f351 in /usr/bin/clickhouse
11. Poco::PooledThread::run() @ 0x0000000015c95b87 in /usr/bin/clickhouse
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c93e5c in /usr/bin/clickhouse
13. ? @ 0x00007f41f0328609 in ?
14. ? @ 0x00007f41f024d353 in ?
2024.04.09 06:36:10.182710 [ 304 ] {8e74c6fa-8e93-4c7b-921c-2b6d485f09d7} <Error> executeQuery: Code: 57. DB::Exception: Directory for table data store/0ad/0add270e-2cdc-4e78-aadb-11c0bf77df7b/ already exists. (TABLE_ALREADY_EXISTS) (version 23.8.8.21.altinitystable (altinity build)) (from [::1]:54968) (in query: CREATE TABLE local_1.system_metrics UUID '0add270e-2cdc-4e78-aadb-11c0bf77df7b' (`kernel_vmstat__thp_zero_page_alloc` Float64, `netstat__tcp_closing` Float64, `disk__used_percent` Float64, `net__interface` String, `processes__paging` Float64, `processes__total` Float64, `kernel_vmstat__zone_reclaim_failed` Float64, `swap__in` Float64, `procstat__mem_vms` Float64, `kernel_vmstat__numa_foreign` Float64, `kernel_vmstat__numa_huge_pte_updates` Float64, `kernel_vmstat__pgpgin` Float64, `net__bytes_sent` Float64, `procstat__pid` Float64, `cpu__usage_guest` Float64, `cpu__time_guest_nice` Float64, `mem__cached` Float64, `system__n_users` Float64, `cpu__usage_guest_nice` Float64, `diskio__name` String, `linux_sysctl_fs__aio_nr` Float64, `net__bytes_recv_per_sec` Float64, `kernel_vmstat__nr_page_table_pages` Float64, `kernel_vmstat__unevictable_pgs_stranded` Float64, `system__n_cpus` Float64, `__hostname__` String, `linux_sysctl_fs__inode_nr` Float64, `kernel_vmstat__compact_fail` Float64, `cpu__time_iowait` Float64, `kernel_vmstat__thp_split` Float64, `processes__unknown` Float64, `kernel_vmstat__balloon_deflate` Float64, `system__n_net_cards` Float64, `cpu__usage_iowait` Float64, `diskio__read_await` Float64, `processes__zombies` Float64, `system__load5` Float64, `_tenant_id_` Float64, `_mode_` String, `kernel_vmstat__compact_stall` Float64, `kernel_vmstat__nr_dirty` Float64, `kernel_vmstat__nr_kernel_stack` Float64, `kernel_vmstat__nr_anon_pages` Float64, `kernel_vmstat__nr_free_pages` Float64, `disk__total` Float64, `kernel_vmstat__allocstall` Float64, `diskio__iops_in_progress` Float64, `kernel_vmstat__nr_vmscan_write` Float64, `procstat__mem_stack` Float64, `procstat__cpu_usage` Float64, `kernel_vmstat__pgrefill_normal` Float64, `kernel_vmstat__unevictable_pgs_mlocked` Float64, `timestamp` String, `kernel_vmstat__nr_file_pages` Float64, `netstat__tcp_fin_wait1` Float64, `procstat__mem_rss` Float64, `netstat__tcp_fin_wait2` Float64, `kernel_vmstat__pgsteal_direct_dma32` Float64, `kernel_vmstat__nr_inactive_file` Float64, `kernel__processes_forked` Float64, `kernel_vmstat__swap_ra_hit` Float64, `kernel_vmstat__nr_anon_transparent_hugepages` Float64, `netstat__tcp_listen` Float64, `kernel_vmstat__htlb_buddy_alloc_fail` Float64, `kernel_vmstat__pgrefill_movable` Float64, `system__load1` Float64, `processes__running` Float64, `kernel_vmstat__pgscan_kswapd_dma32` Float64, `mem__buffered` Float64, `kernel_vmstat__compact_isolated` Float64, `disk__free` Float64, `kernel_vmstat__pageoutrun` Float64, `kernel_vmstat__pgalloc_movable` Float64, `mem__available_percent` Float64, `kernel_vmstat__pgscan_direct_normal` Float64, `procstat__mem_swap` Float64, `kernel_vmstat__nr_shmem` Float64, `kernel_vmstat__pgfault` Float64, `__instance_id__` LowCardinality(String), `kernel_vmstat__pgscan_direct_dma` Float64, `_custom_tags` String, `kernel_vmstat__kswapd_inodesteal` Float64, `cpu__time_softirq` Float64, `disk__inodes_total` Float64, `kernel_vmstat__pgscan_kswapd_movable` Float64, `kernel_vmstat__workingset_refault` Float64, `procstat__mem_data` Float64, `kernel_vmstat__nr_isolated_file` Float64, `linux_sysctl_fs__aio_max_nr` Float64, `diskio__write_time` Float64, `linux_sysctl_fs__inode_free_nr` Float64, `swap__used_percent` Float64, `cpu__time_steal` Float64, `netstat__tcp_syn_sent` Float64, `kernel_vmstat__kswapd_high_wmark_hit_quickly` Float64, `kernel_vmstat__pgmigrate_success` Float64, `mem__total` Float64, `kernel_vmstat__pglazyfreed` Float64, `linux_sysctl_fs__dentry_want_pages` Float64, `cpu__usage_idle` Float64, `cpu__usage_user` Float64, `__ip__` String, `kernel_vmstat__pgscan_direct_dma32` Float64, `kernel_vmstat__unevictable_pgs_culled` Float64, `kernel_vmstat__thp_fault_fallback` Float64, `kernel_vmstat__pgsteal_kswapd_dma32` Float64, `kernel_vmstat__workingset_activate` Float64, `kernel_vmstat__thp_collapse_alloc` Float64, `cpu__time_irq` Float64, `kernel_vmstat__drop_slab` Float64, `kernel_vmstat__balloon_migrate` Float64, `system__n_disks` Float64, `kernel_vmstat__nr_written` Float64, `kernel_vmstat__nr_dirty_threshold` Float64, `kernel_vmstat__pgsteal_kswapd_dma` Float64, `kernel_vmstat__pgsteal_kswapd_normal` Float64, `kernel_vmstat__slabs_scanned` Float64, `procstat__mem_locked` Float64, `diskio__writes` Float64, `diskio__write_bytes_per_sec` Float64, `netstat__tcp_time_wait` Float64, `cpu__usage_irq` Float64, `kernel_vmstat__nr_slab_reclaimable` Float64, `disk__inodes_used` Float64, `kernel_vmstat__numa_interleave` Float64, `_receive_timestamp_` DateTime64(3, 'UTC'), `kernel_vmstat__pgmigrate_fail` Float64, `_custom_tags_` String, `kernel_vmstat__unevictable_pgs_scanned` Float64, `diskio__await` Float64, `net__drop_out` Float64, `kernel_vmstat__nr_unstable` Float64, `kernel_vmstat__nr_vmscan_immediate_reclaim` Float64, `kernel__boot_time` Float64, `kernel_vmstat__pgrotated` Float64, `mem__free` Float64, `netstat__tcp_last_ack` Float64, `kernel_vmstat__pgmajfault` Float64, `diskio__io_time` Float64, `processes__blocked` Float64, `cpu__time_guest` Float64, `procstat__write_count` Float64, `kernel_vmstat__nr_dirty_background_threshold` Float64, `kernel_vmstat__htlb_buddy_alloc_success` Float64, `kernel_vmstat__pgalloc_normal` Float64, `procstat__process_name` String, `kernel_vmstat__swap_ra` Float64, `_tag_set_` Array(String), `swap__free` Float64, `kernel_vmstat__pgfree` Float64, `swap__total` Float64, `kernel_vmstat__numa_pte_updates` Float64, `kernel_vmstat__nr_writeback` Float64, `net__err_out` Float64, `net__drop_in` Float64, `kernel_vmstat__compact_success` Float64, `kernel_vmstat__pgdeactivate` Float64, `_at_date_` Date, `procstat__status` Float64, `kernel_vmstat__nr_unevictable` Float64, `cpu__usage_cpu` String, `netstat__tcp_established` Float64, `kernel_vmstat__pgscan_kswapd_normal` Float64, `kernel_vmstat__nr_active_file` Float64, `mem__active` Float64, `kernel_vmstat__pgscan_direct_throttle` Float64, `kernel_vmstat__pgsteal_direct_movable` Float64, `_metrics_type_` String, `netstat__udp_socket` Float64, `diskio__merged_read_count_per_sec` Float64, `diskio__reads_per_sec` Float64, `kernel__context_switches` Float64, `kernel_vmstat__nr_inactive_anon` Float64, `net__packets_sent` Float64, `netstat__tcp_close_wait` Float64, `system__load15` Float64, `kernel_vmstat__numa_miss` Float64, `kernel__interrupts` Float64, `diskio__read_bytes_per_sec` Float64, `kernel_vmstat__nr_alloc_batch` Float64, `kernel_vmstat__pgalloc_dma32` Float64, `mem__used` Float64, `_field_set_` Array(String), `kernel_vmstat__pgrefill_dma` Float64, `diskio__read_bytes` Float64, `kernel_vmstat__kswapd_low_wmark_hit_quickly` Float64, `procstat__write_bytes` Float64, `mem__used_percent` Float64, `disk__path` String, `linux_sysctl_fs__dentry_age_limit` Float64, `netstat__tcp_syn_recv` Float64, `kernel_vmstat__compact_migrate_scanned` Float64, `kernel_vmstat__pgsteal_kswapd_movable` Float64, `kernel_vmstat__nr_bounce` Float64, `cpu__usage_nice` Float64, `kernel_vmstat__nr_free_cma` Float64, `kernel_vmstat__thp_zero_page_alloc_failed` Float64, `disk__inodes_fress` Float64, `kernel_vmstat__pgalloc_dma` Float64, `disk__fstype` String, `kernel_vmstat__pgscan_kswapd_dma` Float64, `linux_sysctl_fs__file_max` Float64, `kernel_vmstat__pgscan_direct_movable` Float64, `diskio__read_time` Float64, `net__packets_recv` Float64, `diskio__writes_per_sec` Float64, `processes__total_threads` Float64, `cpu__usage_system` Float64, `procstat__read_bytes` Float64, `kernel_vmstat__pgrefill_dma32` Float64, `kernel_vmstat__thp_fault_alloc` Float64, `cpu__time_cpu` String, `cpu__usage_steal` Float64, `kernel_vmstat__nr_mapped` Float64, `kernel_vmstat__unevictable_pgs_rescued` Float64, `linux_sysctl_fs__file_nr` Float64, `kernel_vmstat__nr_mlock` Float64, `diskio__write_bytes` Float64, `net__hw_addr` String, `kernel_vmstat__nr_dirtied` Float64, `kernel_vmstat__unevictable_pgs_munlocked` Float64, `diskio__write_await` Float64, `cpu__time_nice` Float64, `swap__out` Float64, `swap__used` Float64, `diskio__merged_write_count_per_sec` Float64, `netstat__tcp_close` Float64, `linux_sysctl_fs__dentry_unused_nr` Float64, `system__n_services` Float64, `kernel_vmstat__numa_hint_faults_local` Float64, `__appname__` String, `diskio__merged_read_count` Float64, `__source__` String, `processes__stopped` Float64, `linux_sysctl_fs__dentry_nr` Float64, `net__err_in` Float64, `kernel_vmstat__pswpin` Float64, `kernel_vmstat__drop_pagecache` Float64, `__instance_name__` LowCardinality(String), `diskio__merged_write_count` Float64, `_at_timestamp_` DateTime64(3, 'UTC'), `mem__inactive` Float64, `net__bytes_sent_per_sec` Float64, `kernel_vmstat__pgsteal_direct_dma` Float64, `kernel_vmstat__pswpout` Float64, `net__bytes_recv` Float64, `kernel_vmstat__numa_pages_migrated` Float64, `disk__used` Float64, `netstat__tcp_none` Float64, `kernel_vmstat__workingset_nodereclaim` Float64, `kernel_vmstat__balloon_inflate` Float64, `cpu__time_system` Float64, `procstat__read_count` Float64, `mem__available` Float64, `kernel_vmstat__numa_other` Float64, `diskio__io_util` Float64, `kernel_vmstat__pgpgout` Float64, `system__uptime` Float64, `processes__sleeping` Float64, `kernel_vmstat__nr_active_anon` Float64, `kernel_vmstat__numa_hint_faults` Float64, `cpu__time_user` Float64, `kernel_vmstat__pginodesteal` Float64, `cpu__usage_softirq` Float64, `processes__dead` Float64, `disk__device` String, `kernel_vmstat__nr_writeback_temp` Float64, `kernel_vmstat__nr_isolated_anon` Float64, `system__uptime_format` String, `diskio__reads` Float64, `kernel_vmstat__nr_slab_unreclaimable` Float64, `kernel_vmstat__pgsteal_direct_normal` Float64, `kernel_vmstat__numa_local` Float64, `cpu__time_idle` Float64, `kernel_vmstat__numa_hit` Float64, `kernel_vmstat__compact_free_scanned` Float64, `net__packets_sent_per_sec` Float64, `kernel_vmstat__pgactivate` Float64, `kernel_vmstat__thp_collapse_alloc_failed` Float64, `linux_sysctl_fs__inode_preshrink_nr` Float64, `kernel_vmstat__unevictable_pgs_cleared` Float64, `net__packets_recv_per_sec` Float64, `__path__` LowCardinality(String), `__device__` LowCardinality(String), `__fstype__` LowCardinality(String), `__name__` LowCardinality(String), `__interface__` LowCardinality(String), `system__platform` String, `system__platform_family` String, `system__os` String, `system__platform_version` String, `system__kernel_arch` String, `system__kernel_version` String, `cpu__usage_busy` Float64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/system_metrics/2024-04-08/{shard}/hits', '{replica}') PARTITION BY toString(_at_date_) ORDER BY (_at_date_, _at_timestamp_, intHash64(toInt64(toDateTime(_at_timestamp_)))) SAMPLE BY intHash64(toInt64(toDateTime(_at_timestamp_))) SETTINGS index_granularity = 8192), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c644ed7 in /usr/bin/clickhouse
1. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&) @ 0x0000000007157e87 in /usr/bin/clickhouse
2. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_delete<DB::DDLGuard>>&) @ 0x0000000011d1e14c in /usr/bin/clickhouse
3. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x0000000011d14d5d in /usr/bin/clickhouse
4. DB::InterpreterCreateQuery::execute() @ 0x0000000011d221c0 in /usr/bin/clickhouse
5. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122e2d15 in /usr/bin/clickhouse
6. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122de475 in /usr/bin/clickhouse
7. DB::TCPHandler::runImpl() @ 0x0000000013155799 in /usr/bin/clickhouse
8. DB::TCPHandler::run() @ 0x0000000013167b79 in /usr/bin/clickhouse
9. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5e154 in /usr/bin/clickhouse
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b5f351 in /usr/bin/clickhouse
11. Poco::PooledThread::run() @ 0x0000000015c95b87 in /usr/bin/clickhouse
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c93e5c in /usr/bin/clickhouse
Could it be because I restored a distributed table referencing a local table first, causing the local table to be affected during the restoration?
could you share
grep "72b5f6fe-0721-4c58-aac0-e5598e194e56" -r /var/lib/clickhouse/backup/shard1-increment-20240408050855/metadata/
Could it be because I restored a distributed table referencing a local table first, causing the local table to be affected during the restoration?
no,
it looks like data was not dropped as expected.
but you shared screenshot where executed DROP DATABASE local_1 SYNC
this operation shall delete physical all /var/lib/clikhouse/store/XXX/XXXXX directories which relaed to tables which present in local_1 database
according to shared screenshots did you deleted database on -0-0-0 but try to restore on -0-1-0 node?
Yesterday, I tried restoring local_1 first, and then restoring others had no issues. I'll retest it again. Thanks.
ok. close issue, feel free reopen if reproduce again