Restore from backup with tar.gz compression

[olegtap]

I have a backup where all the files in shadow directory look like "default_all_1_1_5_120917.tar.gz", and in metadata/xx.json
loks like:

{
        "files": {
                "default": [
                        "default_all_1_1_5_120917.tar.gz",
                        "default_all_2_2_0_120917.tar.gz"
                ]
        },
        "table": "a__job",
        "database": "sp_data",
        "parts": {
                "default": [
                        {
                                "name": "all_1_1_5_120917"
                        },
                        {
                                "name": "all_2_2_0_120917"
                        }
                ]
        },
        "query": "CREATE TABLE sp_data.a__job ON CLUSTER spearcluster1 (`name` String, `ordering` UInt32, `clazz` String, `parallel_tasks` UInt8 DEFAULT 1, `period_sec` UInt32 DEFAULT 300, `init_delay_sec` UInt32 DEFAULT 15, `active` UInt8 DEFAULT 1, `debug` UInt8 DEFAULT 0, `archived` UInt8 DEFAULT 0, `last_run` DateTime, `processed` DateTime) ENGINE = ReplicatedMergeTree PRIMARY KEY name ORDER BY name SETTINGS index_granularity = 8192",
        "size": {
                "default": 5074
        },
        "total_bytes": 3521,
        "metadata_only": false
}

But when I try to restore I get the error that says:

error one of restoreDataRegular go-routine return error: 
can't copy data to detached 'sp_data.a__job': error during filepath.Walk for part 'all_1_1_5_120917': lstat /var/lib/clickhouse/backup/backup_2024-03-09/shadow/sp_data/a__job/all_1_1_5_120917: no such file or directory 
"
What am I doing wrong?

[Eugene Klimov]

looks like you didn't use clickhouse-backup download command and use something else like just copy uploaded backup with tar.gz files directly into local /var/lib/clickhouse/backup/backup_name folder? am i right?

which kind of remote storage do you use?

[olegtap]

You are right, my bad. I just copied backup from Amazon S3.
It is a pretty large backup around 10 TB.
Is there some kind of workaround?
Thanks

[Eugene Klimov]

Manually unpack <disk_name>_<part_name>.tar.gz files in to
/var/lib/clickhouse/backup/backup_name/shadow/<db>/<table>/<part_name>/

[olegtap]

@Slach Thanks a lot.