ScanRow error when running backup list/restore commands

[tman5]

This looks like a permission issue with the backup account, but it does have quote a few permissions

clickhouse-backup list -c backup_config

2024/03/01 09:53:01.486176  info clickhouse connection prepared: tcp://localhost:9000 run ping logger=clickhouse
2024/03/01 09:53:01.488415  info clickhouse connection open: tcp://localhost:9000 logger=clickhouse
2024/03/01 09:53:01.488624  info SELECT value FROM `system`.`build_options` where name='VERSION_INTEGER' logger=clickhouse
2024/03/01 09:53:01.490439  info SELECT countIf(name='type') AS is_disk_type_present, countIf(name='free_space') AS is_free_space_present, countIf(name='disks') AS is_storage_policy_present FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies')  logger=clickhouse
2024/03/01 09:53:01.494544  info SELECT d.path, any(d.name) AS name, 'local' AS type, 0 AS free_space, ['default'] AS storage_policies FROM system.disks AS d  GROUP BY d.path logger=clickhouse
2024/03/01 09:53:01.496587  info clickhouse connection closed logger=clickhouse
2024/03/01 09:53:01.496620 error clickhouse [ScanRow]: (free_space) converting UInt8 to *uint64 is unsupported. try using *uint8

This is the config for the backup user being used:

  backup:
    password_sha256_hex: sha_hex
    networks:
      ip: ::/0
    quota: default
    grants:
      - GRANT ALL ON *.*

Backup config:

general:
  log_level: debug
  remote_storage: s3
clickhouse:
  username: backup
  password: password
s3:
  access_key: clickhouse_backups_rw
  acl: private
  endpoint: https://minio.local
  secret_key: backup
  force_path_style: true
  disable_cert_verification: true
  bucket: clickhouse-backups
  path: backups/

[tman5]

Using the default user we don't get the same error. But we get this error:

2024/03/01 10:40:07.460580 error one of restoreDataRegular go-routine return error: can't restore object_disk server-side copy data parts 'default.infi_clickhouse_orm_migrations': data in objects disks, invalid s3->object_disk_path config section, shall be not empty and shall not be prefix for s3->path

When we run clickhouse-backup restore_remote --rm chi-clickhouse-replicated-0-0-increment-2024-02-29-03-00-08 on the first replica in a shard

[Eugene Klimov]

could you create separate issue for the last message?

[Eugene Klimov]

@tman5 which clickhouse-server version do you use?

[Eugene Klimov]

Could you share under backup user

following SQL query results

SHOW GRANTS FORMAT Vertical;

SELECT countIf(name='type') AS is_disk_type_present, 
countIf(name='free_space') AS is_free_space_present, 
countIf(name='disks') AS is_storage_policy_present 
FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies');

[tman5]

Sure

Clickhouse-server version:

ClickHouse server version 23.12.2.59

clickhouse :) SHOW GRANTS FORMAT Vertical;

SHOW GRANTS
FORMAT Vertical

Query id: 7264bce8-bb21-478d-8406-f8d2efe3880e

Row 1:
──────
GRANTS FORMAT Vertical: GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, UNDROP TABLE, TRUNCATE, OPTIMIZE, BACKUP, KILL QUERY, KILL TRANSACTION, MOVE PARTITION BETWEEN SHARDS, ACCESS MANAGEMENT, SYSTEM, dictGet, displaySecretsInShowAndSelect, INTROSPECTION, SOURCES, CLUSTER ON *.* TO default WITH GRANT OPTION

Row 2:
──────
GRANTS FORMAT Vertical: GRANT ALTER NAMED COLLECTION, CREATE NAMED COLLECTION, DROP NAMED COLLECTION, SHOW NAMED COLLECTIONS, NAMED COLLECTION ON * TO default WITH GRANT OPTION

2 rows in set. Elapsed: 0.001 sec. 

clickhouse :) SELECT countIf(name='type') AS is_disk_type_present, 
countIf(name='free_space') AS is_free_space_present, 
countIf(name='disks') AS is_storage_policy_present 
FROM system.columns WHERE database='system' AND table IN ('disks','storage_policies');

SELECT
    countIf(name = 'type') AS is_disk_type_present,
    countIf(name = 'free_space') AS is_free_space_present,
    countIf(name = 'disks') AS is_storage_policy_present
FROM system.columns
WHERE (database = 'system') AND (table IN ('disks', 'storage_policies'))

Query id: d6037b22-ceaa-4d29-ac56-3ada52c01931

┌─is_disk_type_present─┬─is_free_space_present─┬─is_storage_policy_present─┐
│                    1 │                     1 │                         1 │
└──────────────────────┴───────────────────────┴───────────────────────────┘

1 row in set. Elapsed: 0.003 sec. 

[Eugene Klimov]

Weird
is_free_space_present = 1
shall not generate

SELECT d.path, any(d.name) AS name, 
'local' AS type, 
0 AS free_space, 
['default'] AS storage_policies 
FROM system.disks AS d  
GROUP BY d.path

0 AS free_space add only when is_free_space_present = 0

How do you create backup user via SQL CREATE USER or via XML in users.d?

[tman5]

Via users.d

  backup:
    password_sha256_hex: password
    networks:
      ip: ::/0
    quota: default
    grants:
      query: grant backup

Role

  backup:
    profile: default
    grants:
      query: GRANT CREATE TABLE, CREATE DATABASE, INSERT, BACKUP ON *.*

We did confirm this appears to work when assigning the default profile

[tman5]

FYI we could not get this to work without granting the sysadmin role

  backup:
    profile: default
    password_sha256_hex: password
    networks:
      ip: ::/0
    quota: default
    grants:
      query: grant sysadmin

roles:
  sysadmin:
    profile: default
    grants:
      query: GRANT ALL ON *.*

[Eugene Klimov]

So, this is grants to system.* related issue, thanks for sharing your experience

[tman5]

@Slach so is it expected to have to use `GRANT ALL ON ".*" for the backup account? It just requires that many privileges?

[Eugene Klimov]

if you want to backup *.* tables, you need to ALTER, CREATE, DROP and SELECT from system.*