Giters

1834924958 / go-swan-client

A PB level data ooborading tool for Filecoin Network

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Swan Client Tool Guide

Made by FilSwan Chat on discord standard-readme compliant

Table of Contents

Functions

  • Generate Car files from your source files by Lotus, Graph-split or Ipfs.
  • Generate metadata e.g. Car file URI, start epoch, etc. and save them to a metadata JSON file.
  • Propose deals based on the metadata JSON file.
  • Generate a final JSON file containing deal CIDs and storage provider ids for storage providers to import deals.
  • Create tasks and offline deals on Swan Platform.
  • Send deals automatically to auto-bid storage providers.

Concepts

Task

  • A task can contain one or multiple car files
  • Each car file can be sent to one or multiple miners
  • Methods to set miners for each car file in a task
    • Auto-bid: task.bid_mode=1, Market Matcher will automatically allocate miners for each car file based on reputation system and the max copy number the task needs.
    • Manual-bid: task.bid_mode=0, After bidders winning the bid, the task holder needs to propose the task to the winners.
    • None-bid: task.bid_mode=2, It is required to propose each car file of a task to a list of specified miners.
  • Task Status:

    • Created: After a task is created, its initial status is Created regardless of its type.

    • ActionRequired: An autobid task, that is, task.bid_mode=1, has some information missing or invalid:

      • MaxPrice: missing, or is not a valid number
      • FastRetrieval: missing
      • Type: missing, or not have valid value

      🔔You need to solve the above problems and change the task status to Created to participate next run of Market Matcher.

Car File

  • A car file is an independent unit to be sent to miners
  • Each car file can be sent to one or multiple miners
  • A car file is generated from source file(s) by Lotus, Graph-split, or Ipfs
  • The size of a car file can be up to 64GB.
  • Car File Status:
    • Created: After a task is created, all its car files are in this status
    • ActionRequired: An autobid task, that is, task.bid_mode=1, its car file has something missing or invalid:
      • FileSize: missing, or is not a valid number
      • FileUrl: missing
      • StartEpoch: missing, or not have valid value, less than 0, or current epoch
      • PayloadCid: missing
      • PieceCid: missing
    • Assigned: When its task is in auto-bid mode, that is, task.bid_mode=1, a car file has been assigned to a list of miners automatically by Market Matcher.

Offline Deal

  • An offline Deal means the transaction that a car file is sent to a miner
  • Offline Deal Status:
    • Assigned: Only in auto-bid mode, that is task.bid_mode=1, when a miner is assigned to a car file, an offline deal record is created, and its status is Assigned.
    • Created: For all the bid modes, after a car file is sent to a miner, the related deal status is Created.
    • ...: There are several other statuses, which are generated and used by Swan Provider and Swan Platform and they have the same meaning for tasks of all bid modes.
  • Every step of this tool will generate a JSON file which contains file(s) description like one of below:
[
 {
  "Uuid": "",
  "SourceFileName": "srcFiles",
  "SourceFilePath": "[source file path]",
  "SourceFileMd5": "",
  "SourceFileSize": 5231342,
  "CarFileName": "bafybeidezzxpy3lrvzz2py56vasl7modkss4v56qwh67tzhetsn2qh3aem.car",
  "CarFilePath": "[car file path]",
  "CarFileMd5": "30fc76af655688cc6ef49bbb96ce938a",
  "CarFileUrl": "[car file url]",
  "CarFileSize": 5234921,
  "PayloadCid": "bafybeidezzxpy3lrvzz2py56vasl7modkss4v56qwh67tzhetsn2qh3aem",
  "PieceCid": "baga6ea4seaqfbtlhrfnzuhbmwnjw4a7ovtjijae32g25o56jcuidk2fdzrjgmoi",
  "StartEpoch": null,
  "SourceId": null,
  "Deals": null
 }
]
[
 {
  "Uuid": "072f8d4a-b79e-42b7-9452-3b8d1d41c11c",
  "SourceFileName": "",
  "SourceFilePath": "",
  "SourceFileMd5": "",
  "SourceFileSize": 0,
  "CarFileName": "",
  "CarFilePath": "",
  "CarFileMd5": "",
  "CarFileUrl": "[car file url]",
  "CarFileSize": 5234921,
  "PayloadCid": "bafybeidezzxpy3lrvzz2py56vasl7modkss4v56qwh67tzhetsn2qh3aem",
  "PieceCid": "baga6ea4seaqfbtlhrfnzuhbmwnjw4a7ovtjijae32g25o56jcuidk2fdzrjgmoi",
  "StartEpoch": null,
  "SourceId": 2,
  "Deals": [
   {
    "DealCid": "bafyreih2feyqpckrsmjnwgkm44el45obi3em7cjh7udkq6jgp4flkce6ra",
    "MinerFid": "t03354",
    "StartEpoch": 575856
   }
  ]
 }
]
  • This JSON file generated in each step will be used in its next step and can be used to rebuild the graph in the future.
  • Uuid is generated for future index purpose.

Prerequisites

  • Lotus node

Ubuntu

  • install missing packages as required

Mac

  • install missing packages as required
  • hwloc, such as
brew install hwloc
  • set path LIBRARY_PATH to point to hwloc, such as
export LIBRARY_PATH=/opt/homebrew/Cellar/hwloc/2.6.0/lib

Installation

Option1️⃣ Prebuilt package: See release assets

mkdir swan-client
cd swan-client
wget --no-check-certificate https://github.com/filswan/go-swan-client/releases/download/v2.0.0-rc1/install.sh
chmod +x install.sh
./install.sh

Option2️⃣ Source Code

🔔go 1.16+ is required

git clone https://github.com/filswan/go-swan-client.git
cd go-swan-client
git checkout <release_branch>
./build_from_source.sh

After Installation

  • If you install from source code, the binary file swan-client is under ./build directory, you need to switch to it.
cd build
  • Before executing, you should check your configuration in ~/.swan/client/config.toml to ensure it is right.
vi ~/.swan/client/config.toml

Configuration

[lotus]

  • client_api_url: Url of lotus client web api, such as: http://[ip]:[port]/rpc/v0, generally the [port] is 1234. See Lotus API
  • client_access_token: Access token of lotus client web api. It should have admin access right. You can get it from your lotus node machine using command lotus auth create-token --perm admin. See Obtaining Tokens

[main]

  • api_url: Swan API address. For Swan production, it is https://go-swan-server.filswan.com. It can be ignored if [sender].offline_mode=true.
  • ‼️api_key: Your Swan API key. Acquire from Swan Platform -> "My Profile"->"Developer Settings". It can be ignored if [sender].offline_mode=true.
  • ‼️access_token: Your Swan API access token. Acquire from Swan Platform -> "My Profile"->"Developer Settings". It can be ignored if [sender].offline_mode=true.
  • ‼️storage_server_type: ipfs server or web server

[web-server]

  • download_url_prefix: Web server url prefix, such as: https://[ip]:[port]/download. Store car files for downloading by storage provider. Car file url will be [download_url_prefix]/[filename]

[ipfs-server]

  • download_url_prefix: Ipfs server url prefix, such as: http://[ip]:[port]. Store car files for downloading by storage provider. Car file url will be [download_url_prefix]/ipfs/[file_hash]
  • upload_url_prefix: Ipfs server url for uploading files, such as http://[ip]:[port]

[sender]

  • bid_mode: [0/1/2] Default 1, which is auto-bid mod and it means swan will automatically allocate storage provider for it, while 0 is manual-bid mode and it needs to be bidded manually by storage providers, and 2 means not need bid, but set miners when creating a task.
  • offline_mode: [true/false] Default false. When set to true, you will not be able to create a Swan task on filswan.com, but you can still generate Car Files, CSV and JSON files for sending deals.
  • output_dir: When you do not set -out-dir option in your command, it is used as the default output directory for saving generated car files, CSV and JSON files. You need have access right to this folder or to create it.
  • verified_deal: [true/false] Whether deals in this task are going to be sent as verified or not.
  • fast_retrieval: [true/false] Indicates that data should be available for fast retrieval or not.
  • generate_md5: [true/false] Whether to generate md5 for each car file and source file, note: this is a resource consuming action.
  • skip_confirmation: [true/false] Whether to skip manual confirmation of each deal before sending.
  • wallet: Wallet used for sending offline deals
  • max_price: Max price willing to pay per GiB/epoch for offline deals
  • start_epoch_hours: Start_epoch for deals in hours from current time.
  • expired_days: Expected completion days for storage provider sealing data.
  • gocar_file_size_limit: Go car file size limit in bytes
  • gocar_folder_based: Generate car file based on whole folder, or on each file separately
  • duration: Expressed in epochs (30 seconds per epoch). Default value is 1512000, that is 525 days. Valid value range:[518400, 1540000].
  • max_auto_bid_copy_number: When in auto-bid mode, that is bid_mode=1, max number of miners a car file can be allocated by Market Matcher

Flowcharts

Option1️⃣ Manual-bid Mode

  • Only Created status exists for both task and car file

Option2️⃣ Auto-bid Mode

  • If a task does not match auto-bid conditions, its status will be changed from Created to Action Required
  • If a car file does not match auto-bid conditions, its status will be changed from Created to Action Required
  • If both a car file and its task match auto-bid conditions
    • Miners that match the task and car file requirements will be assigned to car files
    • The max number of allocated miners depend on max_auto_bid_copy_number, see Configuration
    • If there are miners allocated to a car file, its status will be changed to Assigned and task's status remains at Created
    • If there is no miner meet the the car file's and its task's requirements, then their status remain at Created

Option3️⃣ None-bid Mode

  • In this option, deal(s) will be sent when creating a task.

  • Only Created status exists for both task and car file

Create Car Files

🔔 The input dir and out dir should only be absolute one.

🔔 This step is necessary for tasks in all of the bid modes. You can choose one of the following 3 options.

Option1️⃣ By Lotus

🔔 This option will generate a car file for each file in source directory.

./swan-client car -input-dir [input_files_dir] -out-dir [car_files_output_dir]

Command parameters used in this step:

  • -input-dir(Required): The directory where the source files reside in.
  • -out-dir(optional): Car files and metadata files will be generated into this directory. When omitted, use [sender].output_dir in Configuration

Configurations used in this step:

  • [lotus].client_api_url, see Configuration
  • [lotus].client_access_token, see Configuration
  • [sender].output_dir, only used when -out-dir is omitted in command, see Configuration
  • [sender].generate_md5, when it is true, then generate md5 for source files and car files, see Configuration

Files generated after this step:

  • car.json: contains information for both source files and car files, see Offline Deal
  • [source-file-name].car: each source file has a related car file

Option2️⃣ By Graphsplit

🔔 This option can split a file under source directory or the files in a whole directory to one or more car file(s) in output directory.

./swan-client gocar -input-dir [input_files_dir] -out-dir [car_files_output_dir]

Command parameters used in this step:

  • -input-dir(Required): The directory where the source files reside in.
  • -out-dir(optional): Car files and metadata files will be generated into this directory. When omitted, use [sender].output_dir in Configuration

Configurations used in this step:

Files generated after this step:

  • manifest.csv: this is generated by graphsplit api
  • car.json: contains information for both source files and car files, see Offline Deal, generated from manifest.csv
  • [hash-value-of-file-part].car: if gocar_folder_based=true, the folder will have one or more car files, otherwize each a source file will have one or more related car file(s), according to its size and [sender].gocar_file_size_limit

Note:

  • If source filesize is greater than [sender].gocar_file_size_limit, the source file information in the metadata files are for temporary source files, which are generated by graphsplit api, and after the car files are generated, those temporary source files will be deleted by graphsplit api. And in this condition, we do not create source file MD5 in the metadata files.

Credits should be given to filedrive-team. More information can be found in https://github.com/filedrive-team/go-graphsplit.

Option3️⃣ By Ipfs Web Api

🔔 This option will merge files under source directory to one car file in output directory using ipfs web api.

./swan-client ipfscar -input-dir [input_files_dir] -out-dir [car_file_output_dir]

Command parameters used in this step:

  • -input-dir(Required): The directory where the source files reside in.
  • -out-dir(optional): Car file and metadata files will be generated into this directory. When omitted, use [sender].output_dir in Configuration

Configurations used in this step:

  • [lotus].client_api_url, see Configuration
  • [lotus].client_access_token, see Configuration
  • [sender].output_dir, only used when -out-dir is omitted in command, see Configuration
  • [sender].generate_md5, when it is true, then generate md5 for source files and car files, see Configuration
  • [ipfs_server].upload_url_prefix, see Configuration

Files generated after this step:

  • car.json: contains information for car file, see Offline Deal
  • [car-file-cid].car: the source file(s) will be merged into this car file

Option4️⃣ By ipfs-car Command

🔔 ipfs-car command should be installed first using

sudo npm install -g ipfs-car

🔔 This option will merge files under source directory to one car file in output directory using ipfs-car command.

./swan-client ipfscmdcar -input-dir [input_files_dir] -out-dir [car_file_output_dir]

Command parameters used in this step:

  • -input-dir(Required): The directory where the source files reside in.
  • -out-dir(optional): Car file and metadata files will be generated into this directory. When omitted, use [sender].output_dir in Configuration

Configurations used in this step:

  • [lotus].client_api_url, see Configuration
  • [lotus].client_access_token, see Configuration
  • [sender].output_dir, only used when -out-dir is omitted in command, see Configuration
  • [sender].generate_md5, when it is true, then generate md5 for source files and car files, see Configuration

Files generated after this step:

  • car.json: contains information for car file, see Offline Deal
  • [source-files-dir-name].car: the source file(s) will be merged into this car file

Upload Car Files

🔔 It is required to upload car files to file server(s), either to web server or to ipfs server.

Option1️⃣ To a web-server manually

no swan-client subcommand should be executed

Configurations used in this step:

  • [main].storage_server_type, it should be set to web server, see Configuration

Option2️⃣ To a local ipfs server

./swan-client upload -input-dir [input_file_dir]

Command parameters used in this step:

  • -input-dir(Required): The directory where the car files and metadata files reside in. Metadata files will be used and updated here after car files uploaded.

Configurations used in this step:

  • [main].storage_server_type, it should be set to ipfs server. See Configuration
  • [ipfs_server].download_url_prefix, see Configuration
  • [ipfs_server].upload_url_prefix, see Configuration
  • [sender].output_dir, only used when -out-dir is omitted in command, see Configuration

Files updated after this step:

  • car.json: car file url will be updated on the original one, see Offline Deal

Create A Task

🔔 This step is necessary for tasks in all bid modes. You can choose one of the following 3 options.

Option1️⃣ None-bid Mode

./swan-client task -input-dir [car_files_dir] -out-dir [output_files_dir] -miner [Storage_provider_ids] -dataset [curated_dataset] -description [description]

Command parameters used in this step:

  • -input-dir(Required): Input directory where the generated car files and metadata files reside in.
  • -out-dir(optional): Metadata files and swan task file will be generated to this directory. When ommitted, use default [send].output_dir, see Configuration
  • -miner(Required): Storage provider Ids you want to send car files to, miners separated by comma if there are more than one, e.g f01276 or t03354,f01276
  • -dataset(optional): The curated dataset from which the car files are generated
  • -description(optional): Details to better describe the data and confine the task or anything the storage provider needs to be informed.

Configurations used in this step:

Files generated after this step:

  • [task-name]-metadata.json: Contains more content for creating proposal in the next step. Uuid will be updated based upon car.json generated in last step. See Offline Deal

Option2️⃣ Manual-bid or Auto-Bid mode

  • Conditions: [sender].public_deal=true and [sender].bid_mode=1, see Configuration
./swan-client task -input-dir [car_files_dir] -out-dir [output_files_dir] -dataset [curated_dataset] -description [description]

Command parameters used in this step:

  • -input-dir(Required): Input directory where the generated car files and metadata files reside in.
  • -out-dir(optional): Metadata files and swan task file will be generated to this directory. When ommitted, use default [send].output_dir, see Configuration
  • -dataset(optional): The curated dataset from which the car files are generated
  • -description(optional): Details to better describe the data and confine the task or anything the storage provider needs to be informed.

Configurations used in this step:

Files generated after this step:

  • [task-name]-metadata.json: Contains more content for creating proposal in the next step. Uuid will be updated based upon car.json generated in last step. See Offline Deal

Send Deals

🔔 The input dir and out dir should only be absolute one.

🔔 This step is only necessary for Manual-bid or Auto-bid tasks, since for None-bid tasks, the step Create A Task includes sending deals. You can choose one of the following 2 options according to your task bid_mode.

Option1️⃣ Manual deal

Conditions:

  • task can be found by uuid in JSON file from swan platform
  • task.bid_mode=0
./swan-client deal -json [path]/[task-name]-metadata.json -out-dir [output_files_dir] -miner [storage_provider_ids]

Command parameters used in this step:

  • -json(Required): Full file path to the metadata JSON file, see Offline Deal
  • -out-dir(optional): Swan deal final metadata files will be generated to the given directory. When ommitted, use default: [sender].output_dir. See Configuration
  • -miner(Required): Storage provider Ids you want to send car files to, miners separated by comma if there are more than one, e.g f01276 or t03354,f01276

Configurations used in this step:

Files generated after this step:

  • [task-name]-deals.json: Deal CID updated based on [task-name]-metadata.json generated on previous step, see Offline Deal

Option2️⃣ Auto-bid deal

  • After a miner is allocated to a car file by Market Matcher, the client needs to send auto-bid deals using the information submitted to swan in step Create A Task.
  • This step is executed in infinite loop mode, it will send auto-bid deals contiuously when there are deals that can meet below conditions.

Conditions:

  • your tasks in swan
  • task.bid_mode=1
  • offline_deals.status=Assigned
./swan-client auto -out-dir [output_files_dir]

Command parameters used in this step:

  • -out-dir(optional): Swan deal final metadata files will be generated to the given directory. When ommitted, use default: [sender].output_dir. See Configuration

Configurations used in this step:

Files generated for each task after this step:

  • [task-name]-auto-deals.json: Deal CID updated based on [task-name]-metadata.json generated on next step. See Offline Deal

Note:

  • Logs are in directory ./logs
  • You can add nohup before ./swan-client to ignore the HUP (hangup) signal and therefore avoid stop when you log out.
  • You can add >> swan-client.log in the command to let all the logs output to swan-client.log.
  • You can add & at the end of the command to let the program run in background.
  • Such as:
nohup ./swan-client auto -out-dir [output_files_dir] >> swan-client.log &

About

A PB level data ooborading tool for Filecoin Network

Languages

Language:Go 96.7%Language:Makefile 1.9%Language:Shell 1.4%