Giters

akash-network / cosmos-omnibus

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

snapshots: add Storj DCS support for backup & restore

andy108369 opened this issue · comments

commented

Storj is cheap and reliable!

with enterprise-grade 99.95% data availability https://www.storj.io/solutions-brief/data-sovereignty

image

image

Example config:

      - SNAPSHOT_METADATA_URL=https://link.storjshare.io/s/<REDACTED>/akash-snapshots/rpc.la
      - SNAPSHOT_PATH=akash-snapshots/rpc.la
      - SNAPSHOT_JSON=https://link.storjshare.io/s/<REDACTED>/akash-snapshots/rpc.la/snapshot.json?download=1
      - SNAPSHOT_FORMAT=tar.zst
      - ZSTD_NBTHREADS=0 # use all cores => faster

$ curl -s https://link.storjshare.io/s/<REDACTED>/akash-snapshots/rpc.nl/snapshot.json?download=1 | jq -r .latest
https://link.storjshare.io/s/<REDACTED>/akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst

This already is working with Storj DCS S3 Credentials, I've tested it.
Only the snapshot script needs to append ?download=1 suffix to the fileUrl

When sharing the snapshot

The download speed via link.storjshare.io can be very slow.

Whilst the uploading via S3 gateway https://gateway.storjshare.io is quite fast, the downloading is a bit slow and aws s3 cp does not download in parallel.

Hence, it is suggested to use the uplink tool which connects to the Storj DCS nodes directly for downloading.

Example uplink usage:

curl -L https://github.com/storj/storj/releases/latest/download/uplink_linux_amd64.zip -o uplink_linux_amd64.zip
unzip -o uplink_linux_amd64.zip
sudo install uplink /usr/local/bin/uplink
rm uplink

uplink access import --force --interactive=false akash-snapshots <ACCESS_GRANT_TOKEN/or a file>

uplink cp -p 16 sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst .

uplink can do the stream:

  • upload:
tar c -C $SNAPSHOT_DIR . | ZSTD_NBTHREADS=0 zstd -c | uplink cp -p 4 -t 4 - sj://akash-snapshots/test1/test.tar.zst
  • download:
uplink cp -p 4 -t 4 sj://akash-snapshots/rpc.nl/akashnet-2_2022-12-08T22:32:30.tar.zst - | pv -petrafb | ZSTD_NBTHREADS=0 zstd -cd | tar xf -

can add --parallelism-chunk-size 128M (allow user specify the chunk size)

Todo

  • append ?download=1 suffix when forming snapshot.json;
  • update readme with the uplink example when sharing the snapshot for fast downloads;
  • snapshot restore: add the uplink tool for restoring the snapshot;

Refs