Upload Aliyun OSS issue

Question

Upload Aliyun OSS issue

dxygit1 opened this issue 2 months ago · comments

config

      - name: S3_ACL
        value: private
      - name: S3_ENDPOINT
        value: https://oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup
      - name: S3_ACCESS_KEY
        value: ak
      - name: S3_SECRET_KEY
        value: ak
      - name: S3_FORCE_PATH_STYLE
        value: "true"
      - name: S3_DISABLE_SSL
        value: "true"
      - name: S3_DEBUG
        value: "true

error info

 info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 403, RequestID: 6602A41480647F39337A1EAC, HostID: oss-cn-hangzhou.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access.

Could you please help me see where I filled in wrong?

Eugene Klimov · Answer 1 · Tue Mar 26 2024 18:58:32 GMT+0800 (China Standard Time)

Try to remove

      - name: S3_DISABLE_SSL
        value: "true"

and according to
#505 (comment)
and your shared error message
try to use following URL

      - name: S3_ENDPOINT
        value: https://acos-service.oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_FORCE_PATH_STYLE
        value: "false"

dxygit1 · Answer 2 · Tue Mar 26 2024 21:29:13 GMT+0800 (China Standard Time)

Following your instructions will result in the following error.

2024/03/26 13:27:51.345553  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Tue, 26 Mar 2024 13:27:37 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 6602CD496F20953034A2F02E
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 2


2024/03/26 13:27:51.345754  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 6602CD496F20953034A2F02E, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/26 13:27:51.345815  warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: 6602CD496F20953034A2F02E, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist. logger=s3
2024/03/26 13:27:51.346549 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/03/26 13:27:51.346659  info clickhouse connection closed logger=clickhouse

Eugene Klimov · Answer 3 · Tue Mar 26 2024 22:02:40 GMT+0800 (China Standard Time)

could you provide

[s3:DEBUG] Request which happends before response?

dxygit1 · Answer 4 · Wed Mar 27 2024 09:55:34 GMT+0800 (China Standard Time)

2024/03/27 01:51:10.263948  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.266304  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.266397  info Create integration tables logger=server
2024/03/27 01:51:10.266420  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.267343  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.267427  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 01:51:10.270410  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 01:51:10.274431  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 01:51:10.279339  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 01:51:10.281289  info DROP TABLE IF EXISTS `system`.`backup_actions` NO DELAY logger=clickhouse
2024/03/27 01:51:10.282350  info CREATE TABLE system.backup_actions (command String, start DateTime, finish DateTime, status String, error String) ENGINE=URL('http://127.0.0.1:7171/backup/actions', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 01:51:10.287803  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 01:51:10.291312  info DROP TABLE IF EXISTS `system`.`backup_list` NO DELAY logger=clickhouse
2024/03/27 01:51:10.292185  info CREATE TABLE system.backup_list (name String, created DateTime, size Int64, location String, required String, desc String) ENGINE=URL('http://127.0.0.1:7171/backup/list', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 01:51:10.297966  info clickhouse connection closed logger=clickhouse
2024/03/27 01:51:10.298356  info Starting API server on 0.0.0.0:7171 logger=server.Run
2024/03/27 01:51:10.300054  info Update backup metrics start (onlyLocal=false) logger=server
2024/03/27 01:51:10.300165  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.300302  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.300956  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.300977  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 01:51:10.301206  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.301242  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 01:51:10.302533  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 01:51:10.302770  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 01:51:10.305608  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 01:51:10.306309  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 01:51:10.309160 error ResumeOperationsAfterRestart return error: open /var/lib/clickhouse/backup: no such file or directory logger=server.Run
2024/03/27 01:51:10.309786  info clickhouse connection closed logger=clickhouse
2024/03/27 01:51:10.309822  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 01:51:10.311819  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 01:51:10.311865  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 01:51:10.314505  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 01:51:10.315640  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 01:51:10.318045  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 01:51:10.322159  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 32533705-7dfd-467b-b395-9564b69a34d5
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=aa717b3683089d17111e98a923659b200d267b12776409dd77a4384e18e021cd
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T015110Z


2024/03/27 01:51:10.718922  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 01:50:56 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 66037B80C505B33730A9B573
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 122


2024/03/27 01:51:10.719793  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 66037B80C505B33730A9B573, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/27 01:51:10.719950 debug /tmp/.clickhouse-backup-metadata.cache.S3 not found, load 0 elements logger=s3
2024/03/27 01:51:10.721128  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2F HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 44d0e9db-684f-45f3-bcdd-e44baf4dd3c0
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=xxxxx
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T015110Z


2024/03/27 01:51:10.734252  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 01:50:56 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 66037B80C505B33730A6B773
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 4


2024/03/27 01:51:10.734472  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 66037B80C505B33730A6B773, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/27 01:51:10.734533  warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: 66037B80C505B33730A6B773, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist. logger=s3
2024/03/27 01:51:10.735120 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/03/27 01:51:10.735264  info clickhouse connection closed logger=clickhouse
2024/03/27 01:51:10.735299  info Update backup metrics finish LastBackupCreateLocal=<nil> LastBackupCreateRemote=<nil> LastBackupSizeLocal=0 LastBackupSizeRemote=0 LastBackupUpload=<nil> NumberBackupsLocal=0 NumberBackupsRemote=0 duration=435ms logger=server

Eugene Klimov · Answer 5 · Wed Mar 27 2024 12:41:20 GMT+0800 (China Standard Time)

GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2F HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com

Bucket in path and custom endpoint which already contains bucket name

@dxygit1
are you sure you applied
force_path_style: false?

Let's remove bucket name from endpoint and could you share logs for the following approach?

      - name: S3_ENDPOINT
        value: https://oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_FORCE_PATH_STYLE
        value: "true"

dxygit1 · Answer 6 · Wed Mar 27 2024 13:13:08 GMT+0800 (China Standard Time)

S3_FORCE_PATH_STYLE has been set to false.

    env:
      - name: CLICKHOUSE_PASSWORD
        value: abc123
      - name: CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE
        value: "false"
      - name: LOG_LEVEL
        value: debug
      - name: ALLOW_EMPTY_BACKUPS
        value: "true"
      - name: API_LISTEN
        value: 0.0.0.0:7171
      - name: API_CREATE_INTEGRATION_TABLES
        value: "true"
      - name: BACKUPS_TO_KEEP_REMOTE
        value: "3"
      - name: REMOTE_STORAGE
        value: s3
      - name: S3_ACL
        value: private
      - name: S3_ENDPOINT
        value: https://oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup/shard-{shard}
      - name: S3_ACCESS_KEY
        value: xxxx
      - name: S3_SECRET_KEY
        value: xxxx
      - name: S3_FORCE_PATH_STYLE
        value: "false"
      - name: S3_DEBUG
        value: "true"

error info

2024/03/27 05:08:27.600610  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:27.601825 error clickhouse connection ping: tcp://localhost:9000 return error: dial tcp [::1]:9000: connect: connection refused logger=clickhouse
2024/03/27 05:08:27.601848 error dial tcp [::1]:9000: connect: connection refused logger=server.Run
2024/03/27 05:08:32.603032  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.604575  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.604695  info Create integration tables logger=server
2024/03/27 05:08:32.604728  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.606200  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.606314  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:08:32.608917  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:08:32.612406  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:08:32.616601  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 05:08:32.618099  info DROP TABLE IF EXISTS `system`.`backup_actions` NO DELAY logger=clickhouse
2024/03/27 05:08:32.619185  info CREATE TABLE system.backup_actions (command String, start DateTime, finish DateTime, status String, error String) ENGINE=URL('http://127.0.0.1:7171/backup/actions', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 05:08:32.775490  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 05:08:32.777477  info DROP TABLE IF EXISTS `system`.`backup_list` NO DELAY logger=clickhouse
2024/03/27 05:08:32.778511  info CREATE TABLE system.backup_list (name String, created DateTime, size Int64, location String, required String, desc String) ENGINE=URL('http://127.0.0.1:7171/backup/list', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 05:08:32.785289  info clickhouse connection closed logger=clickhouse
2024/03/27 05:08:32.785770  info Starting API server on 0.0.0.0:7171 logger=server.Run
2024/03/27 05:08:32.787630  info Update backup metrics start (onlyLocal=false) logger=server
2024/03/27 05:08:32.787655  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.787723  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.789457  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.789483  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:08:32.789561  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.789592  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:08:32.791280  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:08:32.791290  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:08:32.794952  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:08:32.795223  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:08:32.801080  info clickhouse connection closed logger=clickhouse
2024/03/27 05:08:32.801133  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:08:32.801852  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:08:32.801901  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 05:08:32.804261  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 05:08:32.805496  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 05:08:32.807831  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 05:08:32.812084  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: f3ec91f5-af08-4f70-8890-eea9733e9976
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=xxxxx
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T050832Z


2024/03/27 05:08:32.914038  info [s3:DEBUG] Response
HTTP/1.1 403 Forbidden
Content-Length: 368
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 05:08:18 GMT
Server: AliyunOSS
X-Amz-Request-Id: 6603A9C22B654B303133FC64
X-Oss-Ec: 0003-00001401


2024/03/27 05:08:32.914850  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 403, RequestID: 6603A9C22B654B303133FC64, HostID: oss-cn-hangzhou.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access.
2024/03/27 05:08:32.914959 debug /tmp/.clickhouse-backup-metadata.cache.S3 not found, load 0 elements logger=s3
2024/03/27 05:08:32.915631  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 4ba83a92-a939-4fa1-9015-425910836b6b
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=759e7bbee2ebfbba9c07886ddd968570afc75cbd3c263c6039693d86cb04d3ef
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T050832Z


2024/03/27 05:08:32.927442  info [s3:DEBUG] Response
HTTP/1.1 403 Forbidden
Content-Length: 368
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 05:08:18 GMT
Server: AliyunOSS
X-Amz-Request-Id: 6603A9C22B654B3031CDFC64
X-Oss-Ec: 0003-00001401


2024/03/27 05:08:32.927621  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 403, RequestID: 6603A9C22B654B3031CDFC64, HostID: oss-cn-hangzhou.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access.
2024/03/27 05:08:32.927669  warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 6603A9C22B654B3031CDFC64, HostID: oss-cn-hangzhou.aliyuncs.com, api error SecondLevelDomainForbidden: Please use virtual hosted style to access. logger=s3
2024/03/27 05:08:32.927959 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/03/27 05:08:32.928052  info clickhouse connection closed logger=clickhouse
2024/03/27 05:08:32.928082  info Update backup metrics finish LastBackupCreateLocal=2024-03-27 03:38:18.515102036 +0000 UTC LastBackupCreateRemote=<nil> LastBackupSizeLocal=687823 LastBackupSizeRemote=0 LastBackupUpload=<nil> NumberBackupsLocal=2 NumberBackupsRemote=0 duration=141ms logger=server

Eugene Klimov · Answer 7 · Wed Mar 27 2024 13:32:12 GMT+0800 (China Standard Time)

let's apply

      - name: S3_ENDPOINT
        value: https://acos-service.oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup/shard-{shard}
      - name: S3_FORCE_PATH_STYLE
        value: "false"

and please share the logs

dxygit1 · Answer 8 · Wed Mar 27 2024 13:40:55 GMT+0800 (China Standard Time)

      - name: S3_ENDPOINT
        value: https://acos-service.oss-cn-hangzhou.aliyuncs.com
      - name: S3_BUCKET
        value: acos-service
      - name: S3_PATH
        value: backup/shard-{shard}
      - name: S3_ACCESS_KEY
        value: xxx
      - name: S3_SECRET_KEY
        value: xxx
      - name: S3_FORCE_PATH_STYLE
        value: "false"

2024/03/27 05:39:16.312745  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:16.315045  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:16.315134  info Create integration tables logger=server
2024/03/27 05:39:16.315162  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:16.316044  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:16.316135  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:39:16.319549  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:39:16.323973  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:39:16.329320  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 05:39:16.330692  info DROP TABLE IF EXISTS `system`.`backup_actions` NO DELAY logger=clickhouse
2024/03/27 05:39:16.331772  info CREATE TABLE system.backup_actions (command String, start DateTime, finish DateTime, status String, error String) ENGINE=URL('http://127.0.0.1:7171/backup/actions', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 05:39:17.024788  info SELECT engine FROM system.databases WHERE name = 'system' logger=clickhouse
2024/03/27 05:39:17.026997  info DROP TABLE IF EXISTS `system`.`backup_list` NO DELAY logger=clickhouse
2024/03/27 05:39:17.027937  info CREATE TABLE system.backup_list (name String, created DateTime, size Int64, location String, required String, desc String) ENGINE=URL('http://127.0.0.1:7171/backup/list', JSONEachRow) SETTINGS input_format_skip_unknown_fields=1 logger=clickhouse
2024/03/27 05:39:17.032606  info clickhouse connection closed logger=clickhouse
2024/03/27 05:39:17.033108  info Starting API server on 0.0.0.0:7171 logger=server.Run
2024/03/27 05:39:17.035021  info Update backup metrics start (onlyLocal=false) logger=server
2024/03/27 05:39:17.035088  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:17.035109  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:17.036207  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:17.036240  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:39:17.036438  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:17.036460  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/27 05:39:17.038041  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:39:17.038045  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/27 05:39:17.041939  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:39:17.042978  info SELECT d.path, any(d.name) AS name, any(d.type) AS type, min(d.free_space) AS free_space, groupUniqArray(s.policy_name) AS storage_policies FROM system.disks AS d  INNER JOIN (SELECT policy_name, arrayJoin(disks) AS disk FROM system.storage_policies) AS s ON s.disk = d.name GROUP BY d.path logger=clickhouse
2024/03/27 05:39:17.049299  info clickhouse connection closed logger=clickhouse
2024/03/27 05:39:17.049350  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/27 05:39:17.050197  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/27 05:39:17.050251  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 05:39:17.052450  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 05:39:17.053496  info SELECT count() AS is_macros_exists FROM system.tables WHERE database='system' AND name='macros'  SETTINGS empty_result_for_aggregation_by_empty_set=0 logger=clickhouse
2024/03/27 05:39:17.055560  info SELECT macro, substitution FROM system.macros logger=clickhouse
2024/03/27 05:39:17.059445  info [s3:DEBUG] Request
GET /acos-service?versioning= HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 9b075f36-e76e-4bff-b498-f34b5f2767ad
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=d9a40ad221e7c608de65011088d3af0ffd2a31c91b216eb03832c6e11be9a246
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T053917Z


2024/03/27 05:39:17.221982  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 05:39:02 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 6603B0F6AF96A13139D7F1F7
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 66


2024/03/27 05:39:17.222652  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 6603B0F6AF96A13139D7F1F7, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/27 05:39:17.222769 debug /tmp/.clickhouse-backup-metadata.cache.S3 not found, load 0 elements logger=s3
2024/03/27 05:39:17.223542  info [s3:DEBUG] Request
GET /acos-service?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: acos-service.oss-cn-hangzhou.aliyuncs.com
User-Agent: aws-sdk-go-v2/1.24.1 os/linux lang/go#1.22.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.48.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: eff8cdd0-6f43-4ac2-b9bd-46680cc97f8c
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20240327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=xxxxx
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240327T053917Z


2024/03/27 05:39:17.238089  info [s3:DEBUG] Response
HTTP/1.1 404 Not Found
Content-Length: 381
Connection: keep-alive
Content-Type: application/xml
Date: Wed, 27 Mar 2024 05:39:02 GMT
Server: AliyunOSS
Vary: Origin
X-Amz-Request-Id: 6603B0F6AF96A13139E0F2F7
X-Oss-Ec: 0026-00000001
X-Oss-Server-Time: 6


2024/03/27 05:39:17.238263  info [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: 6603B0F6AF96A13139E0F2F7, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist.
2024/03/27 05:39:17.238317  warn BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: 6603B0F6AF96A13139E0F2F7, HostID: acos-service.oss-cn-hangzhou.aliyuncs.com, api error NoSuchKey: The specified key does not exist. logger=s3
2024/03/27 05:39:17.238620 debug /tmp/.clickhouse-backup-metadata.cache.S3 save 0 elements logger=s3
2024/03/27 05:39:17.238727  info clickhouse connection closed logger=clickhouse
2024/03/27 05:39:17.238758  info Update backup metrics finish LastBackupCreateLocal=2024-03-27 03:38:18.515102036 +0000 UTC LastBackupCreateRemote=<nil> LastBackupSizeLocal=687823 LastBackupSizeRemote=0 LastBackupUpload=<nil> NumberBackupsLocal=2 NumberBackupsRemote=0 duration=204ms logger=server

Eugene Klimov · Answer 9 · Wed Mar 27 2024 14:43:57 GMT+0800 (China Standard Time)

ok. it looks like some bug in AWS SDK or iside our confugation of AWS SDK

dxygit1 · Answer 10 · Wed Mar 27 2024 14:47:30 GMT+0800 (China Standard Time)

Okay, let's use MinIO as a replacement first. I've tested MinIO, and it's working fine. Could you please kindly help fix this bug? We really need this functionality

dxygit1 · Answer 11 · Tue Apr 02 2024 11:57:26 GMT+0800 (China Standard Time)

May I ask when it will support Alibaba Cloud OSS?