use CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE start error

Question

use CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE start error

dxygit1 opened this issue 2 months ago · comments

error info

error clickhouse `timeout: 5m`, not enough for `use_embedded_backup_restore: true` logger=server.Run

deploy

            env:
              - name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
                value: "false"

dxygit1 commented 2 months ago

1.txt

dxygit1 commented 2 months ago

2.txt

Eugene Klimov · Answer 1 · Tue Mar 26 2024 20:45:03 GMT+0800 (China Standard Time)

provide more context
which container you setup env section?

dxygit1 · Answer 2 · Tue Mar 26 2024 21:22:23 GMT+0800 (China Standard Time)

When adding this parameter during deployment, the following error occurs

    - name: CLICKHOUSE_PASSWORD
        value: abc123
      - name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
        value: "true"
      - name: LOG_LEVEL
        value: debug
      - name: ALLOW_EMPTY_BACKUPS
        value: "true"
      - name: API_LISTEN
        value: 0.0.0.0:7171
      - name: API_CREATE_INTEGRATION_TABLES
        value: "true"
      - name: BACKUPS_TO_KEEP_REMOTE
        value: "3"
      - name: REMOTE_STORAGE
        value: s3
      - name: S3_ACL
        value: private
      - name: S3_ENDPOINT
        value: https://oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup
      - name: S3_ACCESS_KEY
        value: xxxx
      - name: S3_SECRET_KEY
        value: xxx
      - name: S3_FORCE_PATH_STYLE
        value: "true"
      - name: S3_DISABLE_SSL
        value: "true"
      - name: S3_DEBUG
        value: "true"

kubectl logs pod error

error clickhouse `timeout: 5m`, not enough for `use_embedded_backup_restore: true` logger=server.Run

Eugene Klimov · Answer 3 · Tue Mar 26 2024 22:15:52 GMT+0800 (China Standard Time)

Eugene Klimov commented 2 months ago

Eugene Klimov · Answer 4 · Tue Mar 26 2024 22:16:11 GMT+0800 (China Standard Time)

apply true instead of ture

dxygit1 · Answer 5 · Wed Mar 27 2024 09:48:55 GMT+0800 (China Standard Time)

Can you take a look at my configuration

      - name: CLICKHOUSE_PASSWORD
        value: abc123
      - name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
        value: "true"
      - name: LOG_LEVEL
        value: debug
      - name: ALLOW_EMPTY_BACKUPS
        value: "true"
      - name: API_LISTEN
        value: 0.0.0.0:7171
      - name: API_CREATE_INTEGRATION_TABLES
        value: "true"
      - name: BACKUPS_TO_KEEP_REMOTE
        value: "3"
      - name: REMOTE_STORAGE
        value: s3
      - name: S3_ACL
        value: private
      - name: S3_ENDPOINT
        value: https://oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup
      - name: S3_ACCESS_KEY
        value: xxxx
      - name: S3_SECRET_KEY
        value: xxx
      - name: S3_FORCE_PATH_STYLE
        value: "true"
      - name: S3_DISABLE_SSL
        value: "true"
      - name: S3_DEBUG
        value: "true"

The image was added later for demonstration purposes. Changing it to true will result in an error.

Eugene Klimov · Answer 6 · Wed Mar 27 2024 12:46:33 GMT+0800 (China Standard Time)

Changing it to true will result in an error.

@dxygit1
Do you mean error which described in #877 ?

dxygit1 · Answer 7 · Wed Mar 27 2024 12:56:19 GMT+0800 (China Standard Time)

The error is this. It has nothing to do with either.

error clickhouse `timeout: 5m`, not enough for `use_embedded_backup_restore: true` logger=server.Run

Eugene Klimov · Answer 8 · Wed Mar 27 2024 13:13:51 GMT+0800 (China Standard Time)

just add in your configuration

- name: CLICKHOUSE_TIMEOUT
  value: 4h

cause BACKUP / RESTORE SQL command could take a lot of time during execution

use_backup_embedded_restore: true
require define separate backup disk

look example in
https://github.com/Altinity/clickhouse-backup/blob/master/test/integration/config-s3-embedded.yml#L20-L23

and
https://github.com/Altinity/clickhouse-backup/blob/master/test/integration/dynamic_settings.sh#L214-L238

dxygit1 · Answer 9 · Wed Mar 27 2024 13:32:04 GMT+0800 (China Standard Time)

Thank you for your response. Now I have three questions:

1、I didn't find the corresponding environment variable for embedded_backup_disk. For example, timeout corresponds to CLICKHOUSE_TIMEOUT. Is embedded_backup_disk corresponding to CLICKHOUSE_EMBEDDED_BACKUP_DISK?

2、Do these need to be added as configurations within ClickHouse?

3、How to backup and restore? Is it done through API requests? Is there any documentation explaining the backup and restore principles?

dxygit1 · Answer 10 · Wed Mar 27 2024 15:16:49 GMT+0800 (China Standard Time)

I haven't found any API for performing both backup and upload simultaneously. Currently, it seems that the APIs for backup and upload are separate. Is that correct? And for incremental backup, is the only option the /backup/watch endpoint?

Eugene Klimov · Answer 11 · Wed Mar 27 2024 15:33:00 GMT+0800 (China Standard Time)

I didn't find the corresponding environment variable for embedded_backup_disk

Look to
https://github.com/Altinity/clickhouse-backup/blob/master/ReadMe.md#default-config

there is the list of all available variable names

Do these need to be added as configurations within ClickHouse?

backup disk requires to be created explicitly for latest released clickhouse-backup version
use spec.configuration.files in your kind: ClickHouseInstallation manifest if you use clickhouse-operator

spec:
  configuration:
    files:
      config.d/backup_disk.xml: |
        <clickhouse>
        ...
        </clickhouse>

Eugene Klimov · Answer 12 · Wed Mar 27 2024 15:34:13 GMT+0800 (China Standard Time)

How to backup and restore? Is it done through API requests? Is there any documentation explaining the backup and restore principles?

Please read
https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md

Eugene Klimov · Answer 13 · Wed Mar 27 2024 15:38:33 GMT+0800 (China Standard Time)

I haven't found any API for performing both backup and upload simultaneously.

look to
https://github.com/Altinity/clickhouse-backup/?tab=readme-ov-file#default-config

api config section

env:
- name: API_ALLOW_PARALLEL
  value: "true"

be careful because this use-case is rare, and you can make high workload for your system and degrade performance if you don't understand how backup works and how it affects

Eugene Klimov · Answer 14 · Wed Mar 27 2024 15:39:51 GMT+0800 (China Standard Time)

And for incremental backup, is the only option the /backup/watch endpoint?

please read
https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#how-to-back-up--restore-a-sharded-cluster

and
https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#how-to-use-clickhouse-backup-in-kubernetes

dxygit1 · Answer 15 · Wed Mar 27 2024 16:41:51 GMT+0800 (China Standard Time)

Can this parameter be added in this way? Currently, it seems to be causing errors.

curl chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=1h -X POST                      
{"status":"acknowledged","operation":"watch","command":"watch --watch-interval=\"1h\""}

I saw the following error in the logs:

 'error Watch error: fullInterval 24h is too long to keep 3 remote backups with watchInterval 1h logger=server.

I don't want to do a full backup every day. Can I remove this requirement?

What is the relationship between them? After performing an incremental backup, is a full backup still necessary? Will the previous incremental backups be deleted after a full backup?

Eugene Klimov · Answer 16 · Wed Mar 27 2024 22:53:16 GMT+0800 (China Standard Time)

Yes you can
https://github.com/Altinity/clickhouse-backup/blob/master/ReadMe.md?plain=1#L677-L690

but you can't change exists running watch command
first of all you need to POST /backup/kill
look details in
https://github.com/Altinity/clickhouse-backup/blob/master/ReadMe.md?plain=1#L644-L648

dxygit1 · Answer 17 · Thu Mar 28 2024 09:52:21 GMT+0800 (China Standard Time)

Can't I change the parameters? For example, how often to perform backups, how often to perform full backups? If I don't specify tables, does it mean it's a full database backup?

I've reviewed this example. Am I unable to change the parameters? Aren't they optional parameters?

curl -s 'localhost:7171/backup/watch?table=default.billing&watch_interval=1h&full_interval=24h' -X POST

but you can't change exists running watch command

How should I interpret this sentence? I didn't modify it, did I? I only changed the parameters.

dxygit1 · Answer 18 · Thu Mar 28 2024 09:57:27 GMT+0800 (China Standard Time)

Additionally, what is the reason for this log error? Is it because I allocated too little disk space? What is its calculation logic?

 'error Watch error: fullInterval 24h is too long to keep 3 remote backups with watchInterval 1h logger=server.

Eugene Klimov · Answer 19 · Thu Mar 28 2024 11:30:14 GMT+0800 (China Standard Time)

but you can't change exists running watch command
How should I interpret this sentence? I didn't modify it, did I? I only changed the parameters.

It means
Please avoid multiple parallel execution if you use API_ALLOW_PARALLEL: true``

look to SELECT * FROM system.backup_actions or GET /backup/actions
try to avoid multiple parallel execution of watch command

When you run watch first time, then it just run infinite internal commands to create full+incremental backups sequence

create+upload+delete local - full backup
wait watch_interval
create+upload+delete local - increment1' diff from remote full`
wait watch_interval
create+upload+delete local - increment2' diff from remote increment1`
check if full_interval achieved be
look details in
https://github.com/Altinity/clickhouse-backup/?tab=readme-ov-file#cli-command---watch
and
https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#how-to-watch-backups-work

when you call second API call with different parameters and API_ALLOW_PARALLEL is true then you will just execute two backup creation sequences in parallel, it will produce unnecessary workload to your system and degrade performance of clickhouse-server if you have enough data

Additionally, what is the reason for this log error? Is it because I allocated too little disk space? What is its calculation logic?
'error Watch error: fullInterval 24h is too long to keep 3 remote backups with watchInterval 1h logger=server.

No, this is not related to disk size, you defined KEEP_BACKUPS_REMOTE: 3 but try to create full every 24h and increment every 1h, it requires KEEP_BACKUPS_REMOTE: 24 in your env

Don't be shy to make Pull Request with better error message

Please try to read and figure out with links which I shared

so, please kill exists watch command if it's running after that you can success execute

curl -X POST "chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=1h&full_interval=24h"

if you already applied
KEEP_BACKUPS_REMOTE: 24 in your env section of kubernetes manifest

dxygit1 · Answer 20 · Mon Apr 01 2024 14:54:05 GMT+0800 (China Standard Time)

When I execute
curl -X POST "chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=8h&full_interval=24h"
it returns an error
"status":"acknowledged","operation":"watch","command":"watch --watch-interval=\"8h\" --full-interval=\"24h\""
Why is this happening? I don't want backups to be taken every hour; I want them to be taken every 8 hours.

config

<storage_configuration>
              <disks>
                <backups_s3>
                  <type>s3</type>
                  <endpoint>http://minio:9000/clickhouse/{cluster}/{shard}/</endpoint>
                  <!-- https://github.com/Altinity/clickhouse-backup/issues/691
                  <access_key_id>xxx</access_key_id>
                  <secret_access_key>xxxx</secret_access_key>
                  -->
                  <use_environment_credentials>1</use_environment_credentials>
                  <cache_enabled>false</cache_enabled>
                  <!-- to avoid slow startup -->
                  <send_metadata>false</send_metadata>
                </backups_s3>
              </disks>
            </storage_configuration>
            <backups>
              <allowed_disk>backups_s3</allowed_disk>
              <allowed_path>/var/lib/clickhouse/backups_embedded/</allowed_path>
            </backups>

              - name: API_ALLOW_PARALLEL
                value: "true"
              - name: CLICKHOUSE_EMBEDDED_BACKUP_DISK
                value: "backups_s3"
              - name: CLICKHOUSE_TIMEOUT
                value: 4h

logs error info

2024/04/01 07:25:15.202484  info API call POST /backup/watch logger=server
2024/04/01 07:25:15.204612 debug api.status.Start -> status.commands[4] == {ActionRowStatus:{Command:watch --watch-interval="8h" --full-interval="24h" Status:in progress Start:2024-04-01 07:25:15 Finish: Error:} Ctx:context.Background.WithCancel Cancel:0x4a7ba0} logger=status
2024/04/01 07:25:15.205085  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:15.208751  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:15.208807  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.213018  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:15.214155  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.216460  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:15.219344  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 4218d1e2-3bfa-4936-8d4f-d4ee6cf6bad0
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=cbb3a6200b36fa32d62bd5d200342818868cfca1b3f584f89c4e696e45f98aff
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072515Z


2024/04/01 07:25:15.222684  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:44 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C21739887BE7AF
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 07:25:15.223232 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 0 elements logger=s3
2024/04/01 07:25:15.223950  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 38e08fb4-fa0b-4688-8835-733a3e778e46
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=7883b42cbe4c6313353b17c6e1e34a324830a3aa3705f150bc8cb5bbbce9745f
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072515Z


2024/04/01 07:25:15.225501  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 282
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:44 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173988A1BFFB
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 07:25:15.225849 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/04/01 07:25:15.226100  info clickhouse connection closed logger=clickhouse
2024/04/01 07:25:15.226130  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.229716  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:15.231026  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:15.232102  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:15.232779  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.235794  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:15.237472  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:15.238425  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:15.238468  info SELECT metadata_path FROM system.tables WHERE database = 'system' AND metadata_path!='' LIMIT 1; logger=clickhouse
2024/04/01 07:25:15.240480  info SELECT name, engine FROM system.databases WHERE NOT match(name,'^(system|INFORMATION_SCHEMA|information_schema|_temporary_and_external_tables)$') logger=clickhouse
2024/04/01 07:25:15.242805  info SHOW CREATE DATABASE `_internal_ddl_health_check_` logger=clickhouse
2024/04/01 07:25:15.243757  info SHOW CREATE DATABASE `alert` logger=clickhouse
2024/04/01 07:25:15.244580  info SHOW CREATE DATABASE `default` logger=clickhouse
2024/04/01 07:25:15.245376  info SHOW CREATE DATABASE `distributed_1` logger=clickhouse
2024/04/01 07:25:15.246125  info SHOW CREATE DATABASE `dtsmart` logger=clickhouse
2024/04/01 07:25:15.246955  info SHOW CREATE DATABASE `local_1` logger=clickhouse
2024/04/01 07:25:15.248182  info SELECT name, count(*) as is_present FROM system.settings WHERE name IN (?, ?) GROUP BY name with args [show_table_uuid_in_table_create_query_if_not_nil display_secrets_in_show_and_select] logger=clickhouse
2024/04/01 07:25:15.254069  info SELECT name FROM system.databases WHERE engine IN ('MySQL','PostgreSQL','MaterializedPostgreSQL') logger=clickhouse
2024/04/01 07:25:15.255386  info    SELECT     countIf(name='data_path') is_data_path_present,     countIf(name='data_paths') is_data_paths_present,     countIf(name='uuid') is_uuid_present,     countIf(name='create_table_query') is_create_table_query_present,     countIf(name='total_bytes') is_total_bytes_present    FROM system.columns WHERE database='system' AND table='tables'   logger=clickhouse
2024/04/01 07:25:15.258524  info SELECT database, name, engine , data_paths , uuid , create_table_query , coalesce(total_bytes, 0) AS total_bytes   FROM system.tables WHERE is_temporary = 0 ORDER BY total_bytes DESC SETTINGS show_table_uuid_in_table_create_query_if_not_nil=1 logger=clickhouse
2024/04/01 07:25:15.283079  info SELECT metadata_path FROM system.tables WHERE database = 'system' AND metadata_path!='' LIMIT 1; logger=clickhouse
2024/04/01 07:25:15.285256  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_alert.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_alert.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285313  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_arms_app.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_arms_app.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285333  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_arms_service_group.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_arms_service_group.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285357  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_arms_service_group_app.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_arms_service_group_app.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285380  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_chart.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_chart.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285405  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_collect_model.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_collect_model.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285430  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_collect_model_task.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_collect_model_task.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285448  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_dashboard.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_dashboard.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285478  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_data_index.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_data_index.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285501  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_host.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_host.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285539  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_inspect_record.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_inspect_record.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285565  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_inspect_template.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_inspect_template.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285592  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_resource.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_resource.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285621  warn can't read /var/lib/clickhouse/metadata/dtsmart/dt_resource_optimization_suggest.sql: open /var/lib/clickhouse/metadata/dtsmart/dt_resource_optimization_suggest.sql: no such file or directory logger=clickhouse
2024/04/01 07:25:15.285706  info SELECT count() as cnt FROM system.columns WHERE database='system' AND table='functions' AND name='create_query' SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:15.288068  info SELECT name, create_query FROM system.functions WHERE create_query!='' logger=clickhouse
2024/04/01 07:25:15.291454  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/04/01 07:25:15.292687  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/04/01 07:25:15.295271  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/04/01 07:25:15.299456  info BACKUP TABLE `local_1`.`system_metrics`, TABLE `local_1`.`ddd`, TABLE `local_1`.`metrics_summary`, TABLE `local_1`.`arms_frontend_log`, TABLE `local_1`.`system_attribute_meta`, TABLE `local_1`.`easydo_agent_monitor`, TABLE `local_1`.`easydo_collect_task_monitor`, TABLE `distributed_1`.`dt_chart`, TABLE `distributed_1`.`dt_collect_model`, TABLE `distributed_1`.`dt_collect_model_task`, TABLE `distributed_1`.`dt_dashboard`, TABLE `distributed_1`.`dt_data_index`, TABLE `distributed_1`.`dt_host`, TABLE `distributed_1`.`dt_inspect_record`, TABLE `distributed_1`.`dt_inspect_template`, TABLE `distributed_1`.`dt_resource`, TABLE `distributed_1`.`dt_resource_optimization_suggest`, TABLE `distributed_1`.`easydo_agent_monitor`, TABLE `distributed_1`.`easydo_collect_task_monitor`, TABLE `distributed_1`.`metrics_summary`, TABLE `distributed_1`.`system_attribute_meta`, TABLE `distributed_1`.`system_metrics`, TABLE `dtsmart`.`dt_alert`, TABLE `dtsmart`.`dt_arms_app`, TABLE `dtsmart`.`dt_arms_service_group`, TABLE `dtsmart`.`dt_arms_service_group_app`, TABLE `dtsmart`.`dt_chart`, TABLE `dtsmart`.`dt_collect_model`, TABLE `dtsmart`.`dt_collect_model_task`, TABLE `dtsmart`.`dt_dashboard`, TABLE `dtsmart`.`dt_data_index`, TABLE `dtsmart`.`dt_host`, TABLE `dtsmart`.`dt_inspect_record`, TABLE `dtsmart`.`dt_inspect_template`, TABLE `dtsmart`.`dt_resource`, TABLE `dtsmart`.`dt_resource_optimization_suggest`, TABLE `distributed_1`.`dt_arms_app`, TABLE `distributed_1`.`dt_arms_service_group`, TABLE `distributed_1`.`dt_arms_service_group_app`, TABLE `distributed_1`.`dt_alert`, TABLE `distributed_1`.`ddd`, TABLE `distributed_1`.`arms_frontend_log` TO Disk(?,?) with args [backups_s3 shard0-full-20240401072515] logger=clickhouse
2024/04/01 07:25:17.316699  info SELECT sum(total_bytes) AS backup_data_size FROM system.tables WHERE concat(database,'.',name) IN ('local_1.system_metrics', 'local_1.ddd', 'local_1.metrics_summary', 'local_1.arms_frontend_log', 'local_1.system_attribute_meta', 'local_1.easydo_agent_monitor', 'local_1.easydo_collect_task_monitor', 'distributed_1.dt_chart', 'distributed_1.dt_collect_model', 'distributed_1.dt_collect_model_task', 'distributed_1.dt_dashboard', 'distributed_1.dt_data_index', 'distributed_1.dt_host', 'distributed_1.dt_inspect_record', 'distributed_1.dt_inspect_template', 'distributed_1.dt_resource', 'distributed_1.dt_resource_optimization_suggest', 'distributed_1.easydo_agent_monitor', 'distributed_1.easydo_collect_task_monitor', 'distributed_1.metrics_summary', 'distributed_1.system_attribute_meta', 'distributed_1.system_metrics', 'dtsmart.dt_alert', 'dtsmart.dt_arms_app', 'dtsmart.dt_arms_service_group', 'dtsmart.dt_arms_service_group_app', 'dtsmart.dt_chart', 'dtsmart.dt_collect_model', 'dtsmart.dt_collect_model_task', 'dtsmart.dt_dashboard', 'dtsmart.dt_data_index', 'dtsmart.dt_host', 'dtsmart.dt_inspect_record', 'dtsmart.dt_inspect_template', 'dtsmart.dt_resource', 'dtsmart.dt_resource_optimization_suggest', 'distributed_1.dt_arms_app', 'distributed_1.dt_arms_service_group', 'distributed_1.dt_arms_service_group_app', 'distributed_1.dt_alert', 'distributed_1.ddd', 'distributed_1.arms_frontend_log') logger=clickhouse
2024/04/01 07:25:17.320401 debug calculate parts list from embedded backup disk backup=shard0-full-20240401072515 logger=backuper operation=create
2024/04/01 07:25:17.328883  info SELECT value FROM `system`.`build_options` where name='VERSION_DESCRIBE' logger=clickhouse
2024/04/01 07:25:17.330708  info done                      backup=shard0-full-20240401072515 duration=2.093s logger=backuper operation=create_embedded
2024/04/01 07:25:17.330800  info clickhouse connection closed logger=clickhouse
2024/04/01 07:25:17.330883  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:17.331790  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:17.331855  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/04/01 07:25:17.335408  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/04/01 07:25:17.339800  info SELECT max(toInt64(bytes_on_disk * 1.02)) AS max_file_size FROM system.parts logger=clickhouse
2024/04/01 07:25:17.342822  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:17.344884  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:17.345962  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:17.348153  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:17.350285  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 226eae91-ce6b-4f02-9b7d-46655f01aa51
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=edb3e2aa8ffb905ad8f8e15183ad09fc6b91fd7197c57e0b881a39395f8cf0a3
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072517Z


2024/04/01 07:25:17.353222  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173A07785D3A
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 07:25:17.353540 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 0 elements logger=s3
2024/04/01 07:25:17.354045  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 932ec2d5-4139-4679-a3b5-2447606498c1
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=ae8be870680c18d3bfc6f0cd27c46c53d2e527bd65a6f6f934f679f187fa5996
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072517Z


2024/04/01 07:25:17.355857  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 282
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173A07993794
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 07:25:17.356260 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/04/01 07:25:17.360278  warn /var/lib/clickhouse/backup/shard0-full-20240401072515/upload.state empty, will continue from scratch error: open /var/lib/clickhouse/backup/shard0-full-20240401072515/upload.state: no such file or directory logger=resumable
2024/04/01 07:25:17.360332 debug prepare table concurrent semaphore with concurrency=2 len(tablesForUpload)=42 backup=shard0-full-20240401072515 operation=upload
2024/04/01 07:25:17.360499 debug start dtsmart.dt_inspect_template with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360564 debug finish dtsmart.dt_inspect_template with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360501 debug start local_1.easydo_collect_task_monitor with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360656 debug start dtsmart.dt_alert with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360664 debug finish local_1.easydo_collect_task_monitor with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360668 debug finish dtsmart.dt_alert with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360745 debug start dtsmart.dt_host with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360757 debug finish dtsmart.dt_host with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360834 debug start local_1.easydo_agent_monitor with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360865 debug finish local_1.easydo_agent_monitor with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360927 debug start local_1.ddd with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360946 debug finish local_1.ddd with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.360838 debug start dtsmart.dt_inspect_record with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.360985 debug finish dtsmart.dt_inspect_record with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361013 debug start local_1.arms_frontend_log with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361052 debug finish local_1.arms_frontend_log with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361113 debug start dtsmart.dt_resource_optimization_suggest with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361156 debug finish dtsmart.dt_resource_optimization_suggest with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361116 debug start dtsmart.dt_resource with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361237 debug finish dtsmart.dt_resource with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361263 debug start local_1.system_attribute_meta with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361280 debug finish local_1.system_attribute_meta with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361321 debug start local_1.system_metrics with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361337 debug finish local_1.system_metrics with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361394 debug start local_1.metrics_summary with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361416 debug start dtsmart.dt_data_index with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361430 debug finish dtsmart.dt_data_index with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361474 debug start dtsmart.dt_dashboard with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361490 debug finish dtsmart.dt_dashboard with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361418 debug finish local_1.metrics_summary with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361539 debug start dtsmart.dt_collect_model_task with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361550 debug finish dtsmart.dt_collect_model_task with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361593 debug start dtsmart.dt_chart with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361602 debug start dtsmart.dt_collect_model with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361630 debug finish dtsmart.dt_collect_model with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361603 debug finish dtsmart.dt_chart with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361713 debug start dtsmart.dt_arms_service_group_app with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361727 debug finish dtsmart.dt_arms_service_group_app with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361778 debug start dtsmart.dt_arms_service_group with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361794 debug finish dtsmart.dt_arms_service_group with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361801 debug start dtsmart.dt_arms_app with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361817 debug finish dtsmart.dt_arms_app with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361874 debug start distributed_1.dt_data_index with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361890 debug finish distributed_1.dt_data_index with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361943 debug start distributed_1.dt_alert with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361895 debug start distributed_1.dt_arms_service_group_app with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.361981 debug finish distributed_1.dt_arms_service_group_app with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.361952 debug finish distributed_1.dt_alert with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362157 debug start distributed_1.dt_arms_app with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362170 debug start distributed_1.dt_arms_service_group with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362188 debug finish distributed_1.dt_arms_service_group with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362252 debug start distributed_1.dt_chart with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362176 debug finish distributed_1.dt_arms_app with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362429 debug start distributed_1.dt_resource_optimization_suggest with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362452 debug finish distributed_1.dt_resource_optimization_suggest with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362514 debug start distributed_1.dt_resource with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362530 debug finish distributed_1.dt_resource with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362609 debug start distributed_1.dt_inspect_template with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362624 debug finish distributed_1.dt_inspect_template with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362689 debug start distributed_1.dt_inspect_record with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362703 debug finish distributed_1.dt_inspect_record with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362753 debug start distributed_1.dt_host with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362764 debug finish distributed_1.dt_host with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362810 debug start distributed_1.dt_collect_model with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362821 debug finish distributed_1.dt_collect_model with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362869 debug start distributed_1.dt_dashboard with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362883 debug finish distributed_1.dt_dashboard with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362937 debug start distributed_1.dt_collect_model_task with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.362951 debug finish distributed_1.dt_collect_model_task with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363023 debug start distributed_1.arms_frontend_log with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363052 debug finish distributed_1.arms_frontend_log with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363117 debug start distributed_1.easydo_agent_monitor with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363143 debug finish distributed_1.easydo_agent_monitor with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363212 debug start distributed_1.system_metrics with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363227 debug finish distributed_1.system_metrics with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363305 debug start distributed_1.easydo_collect_task_monitor with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363320 debug finish distributed_1.easydo_collect_task_monitor with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363392 debug start distributed_1.metrics_summary with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363407 debug finish distributed_1.metrics_summary with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363507 debug start distributed_1.system_attribute_meta with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363522 debug finish distributed_1.system_attribute_meta with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363623 debug start distributed_1.ddd with concurrency=2 len(table.Parts[...])=0 logger=uploadTableData
2024/04/01 07:25:17.363641 debug finish distributed_1.ddd with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.362270 debug finish distributed_1.dt_chart with concurrency=2 len(table.Parts[...])=0 uploadedFiles=map[], uploadedBytes=0 logger=uploadTableData
2024/04/01 07:25:17.363803  info clickhouse connection closed logger=clickhouse
2024/04/01 07:25:17.363825 error metrics.ExecuteWithMetrics(create_remote) return error: one of upload table go-routine return error: can't open /var/lib/clickhouse/disks/backups_s3/shard0-full-20240401072515/metadata/dtsmart/dt_inspect_template.sql: open /var/lib/clickhouse/disks/backups_s3/shard0-full-20240401072515/metadata/dtsmart/dt_inspect_template.sql: no such file or directory logger=metrics
2024/04/01 07:25:17.363860  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/04/01 07:25:17.365424  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/04/01 07:25:17.365453  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/04/01 07:25:17.369019  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  LEFT JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/04/01 07:25:17.373231  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:17.375608  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:17.376869  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/04/01 07:25:17.379352  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/04/01 07:25:17.381145  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 00f06d8e-d121-467d-801f-7f8459ce753c
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=2d93df77131521eb68188fd28a02563c929b601efea0e9984b5e0de27be927c6
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072517Z


2024/04/01 07:25:17.384393  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 99
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173A0952385E
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 07:25:17.384548 debug /tmp/.clickhouse-backup-metadata.cache.S3 load 0 elements logger=s3
2024/04/01 07:25:17.384987  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 621568fe-08bf-4780-9087-1f2b4f49cfad
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=cad5c2a336642a3c46c8f1376d83855e1cbbc165c915499f4296a787e485544c
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T072517Z


2024/04/01 07:25:17.386492  info [s3:DEBUG] Response
HTTP/1.1 200 OK
Content-Length: 282
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 01 Apr 2024 07:25:46 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2173A09702440
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 07:25:17.386726 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/04/01 07:25:17.386817 debug b.skipIfTheSameRemoteBackupPresent return skip=false logger=RemoveBackupLocal
2024/04/01 07:25:17.386850  info SELECT metadata_path FROM system.tables WHERE database = 'system' AND metadata_path!='' LIMIT 1; logger=clickhouse
2024/04/01 07:25:17.389653  warn b.cleanLocalEmbedded return error: open /var/lib/clickhouse/preprocessed_configs/config.xml: no such file or directory logger=RemoveBackupLocal
2024/04/01 07:25:17.389720  info clickhouse connection closed logger=clickhouse
2024/04/01 07:25:17.389743 error metrics.ExecuteWithMetrics(delete) return error: open /var/lib/clickhouse/preprocessed_configs/config.xml: no such file or directory logger=metrics
2024/04/01 07:25:17.389766 error Watch error: too many errors create_remote: 1, delete local: 1, during watch full_interval: 24h, abort watching logger=server
2024/04/01 07:25:17.389794 debug api.status.stop -> status.commands[4] == {ActionRowStatus:{Command:watch --watch-interval="8h" --full-interval="24h" Status:error Start:2024-04-01 07:25:15 Finish:2024-04-01 07:25:17 Error:too many errors create_remote: 1, delete local: 1, during watch full_interval: 24h, abort watching} Ctx:<nil> Cancel:<nil>} logger=status
2024/04/01 07:25:17.857006 debug API call GET /metrics     logger=server

Eugene Klimov · Answer 21 · Mon Apr 01 2024 15:33:31 GMT+0800 (China Standard Time)

/var/lib/clickhouse/preprocessed_configs/config.xml: no such file or directory

/var/lib/clickhouse
from container where run clickhouse-server shall be available in the clickhouse-backup container

Could you share without sensitive information?

kubectl get chi -n <your-namespace> clickhouse -o yaml

dxygit1 · Answer 22 · Mon Apr 01 2024 15:38:51 GMT+0800 (China Standard Time)

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  annotations:
    meta.helm.sh/release-name: acos
    meta.helm.sh/release-namespace: acos
  creationTimestamp: "2024-04-01T06:07:44Z"
  finalizers:
  - finalizer.clickhouseinstallation.altinity.com
  generation: 6
  labels:
    app.kubernetes.io/managed-by: Helm
  name: clickhouse-cluster
  namespace: acos
  resourceVersion: "44148484"
  uid: 445ab6c3-287b-4545-a9e0-85ea8ef1bae3
spec:
  configuration:
    clusters:
    - layout:
        replicasCount: 1
        shardsCount: 1
      name: xxxxx
      secret:
        value: xxxxx
    users:
      default/networks/ip: ::/0
      default/password: xxxxx
    zookeeper:
      nodes:
      - host: clickhouse-keeper
        port: 2181
  defaults:
    templates:
      podTemplate: clickhouse
      serviceTemplate: clickhouse-clickhouse-cluster
  templates:
    podTemplates:
    - metadata:
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/tcp-probe: "true"
          prometheus.io/tcp-probe-port: 8123,9000
        labels:
          app: clickhouse
      name: clickhouse
      spec:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - clickhouse
              topologyKey: kubernetes.io/hostname
        containers:
        - image: altinity/clickhouse-server:23.8.8.21.altinitystable
          name: clickhouse-cluster
          resources:
            limits:
              cpu: 2
              memory: 4G
            requests:
              cpu: 2
              memory: 4G
          volumeMounts:
          - mountPath: /etc/clickhouse-server/config.xml
            name: chi-clickhouse-cluster-config-file
            subPath: config.xml
          - mountPath: /var/lib/clickhouse/backups_embedded
            name: backups-s3
          - mountPath: /var/lib/clickhouse
            name: ck-data-dir
        - command:
          - bash
          - -xc
          - /bin/clickhouse-backup server
          env:
          - name: S3_COMPRESSION_FORMAT
            value: none
          - name: API_ALLOW_PARALLEL
            value: "true"
          - name: CLICKHOUSE_EMBEDDED_BACKUP_DISK
            value: backups_s3
          - name: CLICKHOUSE_TIMEOUT
            value: 4h
          - name: CLICKHOUSE_PASSWORD
            value: xxxxx
          - name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
            value: "true"
          - name: LOG_LEVEL
            value: debug
          - name: ALLOW_EMPTY_BACKUPS
            value: "true"
          - name: API_LISTEN
            value: 0.0.0.0:7171
          - name: API_CREATE_INTEGRATION_TABLES
            value: "true"
          - name: BACKUPS_TO_KEEP_REMOTE
            value: "24"
          - name: REMOTE_STORAGE
            value: s3
          - name: S3_ACL
            value: private
          - name: S3_ENDPOINT
            value: http://minio:9000
          - name: S3_BUCKET
            value: acos-service
          - name: S3_PATH
            value: backup/shard-{shard}
          - name: S3_ACCESS_KEY
            value: acos
          - name: S3_SECRET_KEY
            value: xxxxx
          - name: S3_FORCE_PATH_STYLE
            value: "false"
          - name: S3_DEBUG
            value: "true"
          image: altinity/clickhouse-backup:master
          imagePullPolicy: IfNotPresent
          name: clickhouse-backup
          ports:
          - containerPort: 7171
            name: backup-rest
        securityContext:
          fsGroup: 101
          runAsGroup: 101
          runAsUser: 101
        volumes:
        - configMap:
            name: chi-clickhouse-cluster-config-file
          name: chi-clickhouse-cluster-config-file
    serviceTemplates:
    - generateName: clickhouse-cluster
      name: clickhouse-clickhouse-cluster
      spec:
        ports:
        - name: native
          port: 9000
          targetPort: 9000
        - name: http
          port: 8123
          targetPort: 8123
        - name: prometheus
          port: 8001
          targetPort: 8001
        type: ClusterIP
    volumeClaimTemplates:
    - name: ck-data-dir
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: open-local-lvm-xfs
    - name: backups-s3
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: open-local-lvm-xfs

Eugene Klimov · Answer 23 · Mon Apr 01 2024 16:01:21 GMT+0800 (China Standard Time)

you don't need separate volume for
/var/lib/clickhouse/backups_embedded
remove

        - mountPath: /var/lib/clickhouse/backups_embedded
          name: backups-s3

and second volumeClaimTemplates item

    - name: backups-s3
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: open-local-lvm-xfs

and add

          - mountPath: /var/lib/clickhouse
            name: ck-data-dir

to second container's item with name: clichouse-backup

dxygit1 · Answer 24 · Mon Apr 01 2024 17:04:59 GMT+0800 (China Standard Time)

deploy config

<storage_configuration>
              <disks>
                <backups_s3>
                  <type>s3</type>
                  <endpoint>http://minio:9000/acos/{cluster}/{shard}/</endpoint>
                  <!-- https://github.com/Altinity/clickhouse-backup/issues/691
                  <access_key_id>acos</access_key_id>
                  <secret_access_key>xxxxxx</secret_access_key>
                  -->
                  <use_environment_credentials>1</use_environment_credentials>
                  <cache_enabled>false</cache_enabled>
                  <!-- to avoid slow startup -->
                  <send_metadata>false</send_metadata>
                </backups_s3>
              </disks>
            </storage_configuration>

If my MinIO bucket is private, I will encounter the following error.

 <Error> Application: DB::Exception: Message: Access Denied., bucket acos, key dtstack/0/yvl/xsfmxmntolofwnmjjwxoqyqliydrf, object size 4: While checking access for disk backups_s3

Switching it to public resolves the issue, but I prefer to use it as private.

Eugene Klimov · Answer 25 · Mon Apr 01 2024 17:30:22 GMT+0800 (China Standard Time)

If my MinIO bucket is private, I will encounter the following error

uncomment credentials in XML config for backups_s3
or add into name: clickhouse-cluster container env section

      - name: AWS_ACCESS_KEY_ID
        value: your_access_key
      - name: AWS_SECRET_ACCESS_KEY
        valoue: your_my_super_secret_key

dxygit1 · Answer 26 · Mon Apr 01 2024 18:16:37 GMT+0800 (China Standard Time)

When I use
curl -X POST "chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=1h&full_interval=24h"
I can see that it has been uploaded to MinIO. However, when I use list to view it, I can only see the local backup and not the remote one.

curl  chi-clickhouse-cluster-dtstack-1-0:7171/backup/list
{"name":"shard1-full-20240401100823","created":"2024-04-01 10:08:24","size":652722,"location":"local","required":"","desc":"embedded"}

What is currently stored in MinIO? Do I need to manually upload local backups to MinIO? I'm not quite understanding the logic here.

dxygit1 · Answer 27 · Mon Apr 01 2024 18:23:48 GMT+0800 (China Standard Time)

Another issue I encountered is that when I use incremental backup, I get an error:

 {"command":"watch --watch-interval=\"8h\" --full-interval=\"24h\"","status":"error","start":"2024-04-01 10:03:43","finish":"2024-04-01 10:09:49","error":"too many errors create_remote: 4, delete local: 0, during watch full_interval: 24h, abort watching"}.

Why is this error happening? Could it be because I have set BACKUPS_TO_KEEP_REMOTE to 3, and it is insufficient?

Eugene Klimov · Answer 28 · Mon Apr 01 2024 18:28:02 GMT+0800 (China Standard Time)

Why is this error happening?

Look to clickhouse-backup container logs

dxygit1 · Answer 29 · Mon Apr 01 2024 18:58:22 GMT+0800 (China Standard Time)


2024/04/01 10:28:40.972282 debug object_disk.ReadMetadataFromFile(/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709/metadata/local_1/easydo_collect_task_monitor.sql)
2024/04/01 10:28:40.972541  info [s3:DEBUG] Request
DELETE /acos/dtstack/0/iqp/wxziuqupsdmtsnomvpbjpurlicbih?x-id=DeleteObject HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: c312dcdc-c381-47a7-824b-4f95b36f610b
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=304779e9f178a9aa556ae0fb108609d50145f671100c8ba4d65159cf02c660e5
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T102840Z


2024/04/01 10:28:40.974231  info [s3:DEBUG] Response
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 01 Apr 2024 10:28:40 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C221353051ED33
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 10:28:40.974281 debug object_disk.ReadMetadataFromFile(/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709/metadata/local_1/metrics_summary.sql)
2024/04/01 10:28:40.974509  info [s3:DEBUG] Request
DELETE /acos/dtstack/0/nxh/wshqwwmtfuwgrcojxamcucvenzznd?x-id=DeleteObject HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 5961e37a-4ce9-4c6e-84f1-469e9f9ee229
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=8a15eff3ad6b098f87e711c98284fdd04157a7adf7c1f056c41158cb009a1064
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T102840Z


2024/04/01 10:28:40.976464  info [s3:DEBUG] Response
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 01 Apr 2024 10:28:40 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2213530707305
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 10:28:40.976513 debug object_disk.ReadMetadataFromFile(/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709/metadata/local_1/system_attribute_meta.sql)
2024/04/01 10:28:40.976743  info [s3:DEBUG] Request
DELETE /acos/dtstack/0/vyt/lxbvewmgrhrcunnnckytkoytispbd?x-id=DeleteObject HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 5cd0e3ac-fe31-47fb-b62a-8af8c32601dc
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=44b403836a7f37a2107392733138ee1c8bcf020fe1a0f021ce1fbc93b0940b17
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T102840Z


2024/04/01 10:28:40.978707  info [s3:DEBUG] Response
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 01 Apr 2024 10:28:40 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C22135309292BA
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 10:28:40.978765 debug object_disk.ReadMetadataFromFile(/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709/metadata/local_1/system_metrics.sql)
2024/04/01 10:28:40.979045  info [s3:DEBUG] Request
DELETE /acos/dtstack/0/txp/kgrfqgtcwpsmrzjzhhihnkqqlvyzp?x-id=DeleteObject HTTP/1.1
Host: minio:9000
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 9adaca0e-987a-4a95-b14e-d26d1ea11649
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=acos/20240401/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=744a67a4774dd105fb54e2f6dd4dc02c7b24a830ad40586c7a656b2d3adde12b
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240401T102840Z


2024/04/01 10:28:40.980732  info [s3:DEBUG] Response
HTTP/1.1 204 No Content
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 01 Apr 2024 10:28:40 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Request-Id: 17C2213530B5B333
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block


2024/04/01 10:28:40.980779 debug remove '/var/lib/clickhouse/backup/shard0-full-20240401102709' logger=RemoveBackupLocal
2024/04/01 10:28:40.980805 debug remove '/var/lib/clickhouse/disks/backups_s3/shard0-full-20240401102709' logger=RemoveBackupLocal
2024/04/01 10:28:40.986507  info done                      backup=shard0-full-20240401102709 duration=443ms location=local logger=RemoveBackupLocal operation=delete
2024/04/01 10:28:40.986607  info clickhouse connection closed logger=clickhouse
2024/04/01 10:28:40.986636 error Watch error: too many errors create_remote: 4, delete local: 0, during watch full_interval: 24h, abort watching logger=server
2024/04/01 10:28:40.986689 debug api.status.stop -> status.commands[9] == {ActionRowStatus:{Command:watch --watch-interval="12h" --full-interval="24h" Status:error Start:2024-04-01 10:22:35 Finish:2024-04-01 10:28:40 Error:too many errors create_remote: 4, delete local: 0, during watch full_interval: 24h, abort watching} Ctx:<nil> Cancel:<nil>} logger=status

dxygit1 · Answer 30 · Mon Apr 01 2024 19:00:58 GMT+0800 (China Standard Time)

When I use curl -X POST "chi-clickhouse-cluster-dtstack-0-0:7171/backup/watch?watch_interval=1h&full_interval=24h" I can see that it has been uploaded to MinIO. However, when I use list to view it, I can only see the local backup and not the remote one.
curl  chi-clickhouse-cluster-dtstack-1-0:7171/backup/list
{"name":"shard1-full-20240401100823","created":"2024-04-01 10:08:24","size":652722,"location":"local","required":"","desc":"embedded"}
What is currently stored in MinIO? Do I need to manually upload local backups to MinIO? I'm not quite understanding the logic here.

Could you please help take a look at this?

Eugene Klimov · Answer 31 · Mon Apr 01 2024 19:54:00 GMT+0800 (China Standard Time)

Could you please help take a look at this?

You shared not the full log from clickhouse-backup container
disable
S3_DEBUG=true in env
and share the logs

Eugene Klimov · Answer 32 · Mon Apr 01 2024 20:56:53 GMT+0800 (China Standard Time)

2024/04/01 12:19:53.136846 warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: 17C22746AC361F67, HostID: 2fd203aa-24e1-4c1a-b55a-529508ffeeb9, NoSuchBucket: logger=s3

check your current configuration

use for minio

  - name: S3_FORCE_PATH_STYLE 
    value: true

Eugene Klimov · Answer 33 · Mon Apr 01 2024 20:59:06 GMT+0800 (China Standard Time)

2024/04/01 12:19:54.010273 warn open /var/lib/clickhouse/disks/backups_s3/shard0-full-20240401121953/data/distributed_1/arms_frontend_log: no such file or directory logger=getTableListByPatternLocal

you not applied my recommendation from #876 (comment)

dxygit1 · Answer 34 · Mon Apr 01 2024 21:11:56 GMT+0800 (China Standard Time)

2024/04/01 12:19:54.010273 警告打开 /var/lib/clickhouse/disks/backups_s3/shard0-full-20240401121953/data/distributed_1/arms_frontend_log：没有这样的文件或目录 logger=getTableListByPatternLocal

你没有应用我在#876 中的建议（评论）

I have implemented your advice.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 850401bf36cd87eb449a943dde9538bd0690281860ec3455b302dacd492f2367
    cni.projectcalico.org/podIP: 100.107.53.47/32
    cni.projectcalico.org/podIPs: 100.107.53.47/32
    meta.helm.sh/release-name: acos
    meta.helm.sh/release-namespace: acos
    prometheus.io/scrape: "true"
    prometheus.io/tcp-probe: "true"
    prometheus.io/tcp-probe-port: 8123,9000
  creationTimestamp: "2024-04-01T12:16:11Z"
  generateName: chi-clickhouse-cluster-dtstack-0-0-
  labels:
    app: clickhouse
    app.kubernetes.io/managed-by: Helm
    clickhouse.altinity.com/app: chop
    clickhouse.altinity.com/chi: clickhouse-cluster
    clickhouse.altinity.com/cluster: dtstack
    clickhouse.altinity.com/namespace: acos
    clickhouse.altinity.com/ready: "yes"
    clickhouse.altinity.com/replica: "0"
    clickhouse.altinity.com/shard: "0"
    controller-revision-hash: chi-clickhouse-cluster-dtstack-0-0-564cc64f95
    statefulset.kubernetes.io/pod-name: chi-clickhouse-cluster-dtstack-0-0-0
  name: chi-clickhouse-cluster-dtstack-0-0-0
  namespace: acos
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: chi-clickhouse-cluster-dtstack-0-0
    uid: b44d04d7-9eb4-45b9-ad54-891cb4776331
  resourceVersion: "44525886"
  uid: 5c3fe1cb-2849-4f58-b61c-8741b530717c
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - clickhouse
        topologyKey: kubernetes.io/hostname
  containers:
  - image: altinity/clickhouse-server:23.8.8.21.altinitystable
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 10
      httpGet:
        path: /ping
        port: http
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 3
      successThreshold: 1
      timeoutSeconds: 1
    name: clickhouse-cluster
    ports:
    - containerPort: 9000
      name: tcp
      protocol: TCP
    - containerPort: 8123
      name: http
      protocol: TCP
    - containerPort: 9009
      name: interserver
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /ping
        port: http
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 3
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "2"
        memory: 4G
      requests:
        cpu: "2"
        memory: 4G
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/clickhouse-server/config.xml
      name: chi-clickhouse-cluster-config-file
      subPath: config.xml
    - mountPath: /etc/clickhouse-server/config.d/
      name: chi-clickhouse-cluster-common-configd
    - mountPath: /etc/clickhouse-server/users.d/
      name: chi-clickhouse-cluster-common-usersd
    - mountPath: /etc/clickhouse-server/conf.d/
      name: chi-clickhouse-cluster-deploy-confd-dtstack-0-0
    - mountPath: /var/lib/clickhouse
      name: ck-data-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-cp66k
      readOnly: true
  - command:
    - bash
    - -xc
    - /bin/clickhouse-backup server
    env:
    - name: S3_COMPRESSION_FORMAT
      value: none
    - name: API_ALLOW_PARALLEL
      value: "true"
    - name: CLICKHOUSE_EMBEDDED_BACKUP_DISK
      value: backups_s3
    - name: CLICKHOUSE_TIMEOUT
      value: 4h
    - name: CLICKHOUSE_PASSWORD
      value: abc123
    - name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
      value: "true"
    - name: LOG_LEVEL
      value: debug
    - name: ALLOW_EMPTY_BACKUPS
      value: "true"
    - name: API_LISTEN
      value: 0.0.0.0:7171
    - name: API_CREATE_INTEGRATION_TABLES
      value: "true"
    - name: BACKUPS_TO_KEEP_REMOTE
      value: "3"
    - name: REMOTE_STORAGE
      value: s3
    - name: S3_ACL
      value: private
    - name: S3_ENDPOINT
      value: http://minio:9000
    - name: S3_BUCKET
      value: acos-service
    - name: S3_PATH
      value: backup/shard-{shard}
    - name: S3_ACCESS_KEY
      value: acos
    - name: S3_SECRET_KEY
      value: DmPWhh6f
    - name: S3_FORCE_PATH_STYLE
      value: "false"
    - name: S3_DEBUG
      value: "false"
    image: altinity/clickhouse-backup:master
    imagePullPolicy: IfNotPresent
    name: clickhouse-backup
    ports:
    - containerPort: 7171
      name: backup-rest
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/clickhouse
      name: ck-data-dir
    - mountPath: /etc/clickhouse-server/config.d/
      name: chi-clickhouse-cluster-common-configd
    - mountPath: /etc/clickhouse-server/users.d/
      name: chi-clickhouse-cluster-common-usersd
    - mountPath: /etc/clickhouse-server/conf.d/
      name: chi-clickhouse-cluster-deploy-confd-dtstack-0-0
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-cp66k
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostAliases:
  - hostnames:
    - chi-clickhouse-cluster-dtstack-0-0
    ip: 127.0.0.1
  hostname: chi-clickhouse-cluster-dtstack-0-0-0
  nodeName: 172-16-121-243
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 101
    runAsGroup: 101
    runAsUser: 101
  serviceAccount: default
  serviceAccountName: default
  subdomain: chi-clickhouse-cluster-dtstack-0-0
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: ck-data-dir
    persistentVolumeClaim:
      claimName: ck-data-dir-chi-clickhouse-cluster-dtstack-0-0-0
  - configMap:
      defaultMode: 420
      name: chi-clickhouse-cluster-config-file
    name: chi-clickhouse-cluster-config-file
  - configMap:
      defaultMode: 420
      name: chi-clickhouse-cluster-common-configd
    name: chi-clickhouse-cluster-common-configd
  - configMap:
      defaultMode: 420
      name: chi-clickhouse-cluster-common-usersd
    name: chi-clickhouse-cluster-common-usersd
  - configMap:
      defaultMode: 420
      name: chi-clickhouse-cluster-deploy-confd-dtstack-0-0
    name: chi-clickhouse-cluster-deploy-confd-dtstack-0-0
  - name: kube-api-access-cp66k
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-04-01T12:16:07Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-04-01T12:16:22Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-04-01T12:16:22Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-04-01T12:16:11Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://12b94ae358c957219c946341262af0c1dff9663e255d81888f08c91dc2dede84
    image: docker.io/altinity/clickhouse-backup:master
    imageID: docker.io/altinity/clickhouse-backup@sha256:c70ede914745fd01da1de51f2fe52e8d38cb371a2660305da5c4685c25140cd2
    lastState: {}
    name: clickhouse-backup
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-04-01T12:16:12Z"
  - containerID: containerd://95eabb9103b27b1184bc6007835c70a200df4496228bf744ba0ebdfac1b9caef
    image: docker.io/altinity/clickhouse-server:23.8.8.21.altinitystable
    imageID: docker.io/altinity/clickhouse-server@sha256:4a25044e9ceab10a6b7f938f7f7c0ae1e56fd9050e3ae260849287cb4cbfebd8
    lastState: {}
    name: clickhouse-cluster
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-04-01T12:16:11Z"
  hostIP: 172.16.121.243
  phase: Running
  podIP: 100.107.53.47
  podIPs:
  - ip: 100.107.53.47
  qosClass: Burstable
  startTime: "2024-04-01T12:16:07Z"

Eugene Klimov · Answer 35 · Tue Apr 02 2024 02:09:42 GMT+0800 (China Standard Time)

ok. i see

  image: altinity/clickhouse-backup:master

please use altinity/clickhouse-backup:latest instead

and share logs from scracth with

- name: S3_DEBUG
  value: "true"

Eugene Klimov · Answer 36 · Tue Apr 02 2024 10:32:30 GMT+0800 (China Standard Time)

@dxygit1 are you sure you install minio properly?

i see in logs

GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: minio:9000

2024/04/02 01:26:47.921971  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found

it should return 200

do you use minio from
https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#how-to-use-clickhouse-backup-in-kubernetes AS IS?

could you share?

kubectl get svc --all-namespaces | grep minio

dxygit1 · Answer 37 · Tue Apr 02 2024 10:42:18 GMT+0800 (China Standard Time)

I'm really sorry, I have installed MinIO correctly, but I haven't created the 'acos-service' bucket. I'll try it out. Thank you for your assistance.

Eugene Klimov · Answer 38 · Tue Apr 02 2024 13:29:36 GMT+0800 (China Standard Time)

I still have a question. When the content of an incremental backup is uploaded to MinIO, how can we restore the incremental part? I understand that restoring the full backup can be done through the interface, but I haven't found any documentation on how to restore the incremental part. Did I overlook something?

During download incremental backup
all required parts from chain of required backups will downloaded and locally you will have full backup with whole set of data parts

dxygit1 · Answer 39 · Tue Apr 02 2024 13:39:18 GMT+0800 (China Standard Time)

Okay, I've seen it. Thank you very much for your response.

Is the data stored here incremental? What is the purpose of this data? Will it be deleted during the next full backup?

Eugene Klimov · Answer 40 · Tue Apr 02 2024 16:05:02 GMT+0800 (China Standard Time)

acos/dstak/0
this is your backup data

also check
S3_PATH
in your settings and look to minio where stored backup_name/metadata.json

check
SELECT * FROM system.backup_list
inside clickhouse-server container

dxygit1 · Answer 41 · Tue Apr 02 2024 16:28:24 GMT+0800 (China Standard Time)

I understand that this is my backup data, but I would like to know what the contents of this backup data are. Is it a full backup of my current database data, or is it incremental data? What is its purpose? I assume I won't need to use the data in this bucket when restoring.

I have another bucket that has corresponding full and incremental backups

Or is the content of these two buckets the same?

Eugene Klimov · Answer 42 · Tue Apr 02 2024 16:36:39 GMT+0800 (China Standard Time)

so do you have two different buckets?
fist for backups_s3 disk which contains backup data
and second acos-service for clickhouse-backup metadata which also contains references to keys inside
backup_name/shadow/db/table/part_name/*.tar files

dxygit1 · Answer 43 · Tue Apr 02 2024 16:41:05 GMT+0800 (China Standard Time)

One is configured in the ClickHouse configuration file. corresponding data

One is configured with environment variables in ClickHouse Backup. corresponding data

Does this mean that these are two copies of the same data?

Eugene Klimov · Answer 44 · Tue Apr 02 2024 18:47:07 GMT+0800 (China Standard Time)

Does this mean that these are two copies of the same data?

No this is different data

backups_s3 disk contains physical backuped data

clickhouse-backup bucket which you defined in S3 section contains metadata which allow restore backups_s3 properly

v2.5 will support empty value embedded_backup_disk: "" and will store both metadata+data in the same bucket

dxygit1 · Answer 45 · Wed Apr 03 2024 10:32:35 GMT+0800 (China Standard Time)

does clickhouse-backup define that only metadata is stored in S3, and during restoration, it will search for the physical data in the backups_s3 disk?

dxygit1 · Answer 46 · Wed Apr 03 2024 11:22:47 GMT+0800 (China Standard Time)

v2.5 will support empty value embedded_backup_disk: "" and will store both metadata+data in the same bucket

When will v2.5 be released approximately?

Eugene Klimov · Answer 47 · Wed Apr 03 2024 11:46:12 GMT+0800 (China Standard Time)

does clickhouse-backup define that only metadata is stored in S3, and during restoration, it will search for the physical data in the backups_s3 disk?

When create, then clickhouse-backup execute BACKUP SQL command on clickhouse-server which upload physical data or execute CopyObject to s3 endpoint which you defined for backup_s3 disk and create metadata files on /var/lib/clickhouse/disks/backup_s3/backup_name/ folder, many small files which contains references to s3 keys inside s3 endpoint,
then clickhouse-backup create /var/lib/clickhouse/disks/backups_s3/backup_name/metadata.json and /var/lib/clickhouse/disks/backups_s3/backup_name/metadata/db/table_name.json

When upload, then clickhouse-backup upload content of /var/lib/clickhouse/disks/backups_s3/backup_name/ to s3 endpoint which you defined in s3 section of /etc/clickhouse-backup/config.yml or overrides in S3_* environment variables

When download then clickhouse-backup download whole content from s3 endpoint /backup_name/ into /var/lib/clickhouse/disks/backups_s3 or any other disk which you defined as backup in clickhouse-server configuration

When restore then clickhouse-backup execute RESTORE SQL command which read s3 keys from /var/lib/clickhouse/disks/backups_s3 and download data parts into /var/lib/clickhouse/tmp for local disks or execute CopyObject for object disks with <type>s3/azure</type>
After that will execute internal processes like ALTER TABLE ... ATTACH PART but more consistent (embedded backup allow backup *Log table engines)

It is clear?

dxygit1 · Answer 48 · Wed Apr 03 2024 11:59:32 GMT+0800 (China Standard Time)

Okay, I understand now. Thank you very much for your help.

Eugene Klimov · Answer 49 · Wed Apr 03 2024 15:29:35 GMT+0800 (China Standard Time)

@dxygit1

When will v2.5 be released approximately?

subscribe to #843
and watch progress