Multi-Chain SubSquid API

This is a project generated by hydra-cli scaffold. It allows for multiple substrate chains to be indexed and queried through a GraphQL endpoint. Currently this only supports the Polkadot and Kusama substrate chains. This project uses packages derived from a fork of the subsquid/hydra which supports a hydra-processor which can pull and process multiple chains. There is a slightly modified schema.graphql which supports querying each Substrate chain.

Prerequisites

Node v14x
Docker

Bootstrap

# The dependencies setup relies on de-duplication, use `ci` to get everything right
npm ci

# Start a postgres instance
docker-compose up db # add optional -d flag to detach from terminal

# Apply migrations related to the processor's state keeping tables
npm run processor:migrate

# Apply the project's migrations
npm run db:migrate

# Now you can start processing chain data
npm run processor:start

# The above command will block
# Open a separate terminal and launch the graphql server to query the processed data
npm run query-node:start

Querying the SubSquid API

Hello

Assuming that you have done the previous bootstrap commands you should then be able to query the locally running graphql endpoint (http://localhost:4000/graphql) with the following query:

query {
  hello {
    greeting
  }
}

Which should provide a response similar to:

{
  "data": {
    "hello": {
      "greeting": "Hello, we've seen 61812 Kusama transfers, and 36596 Polkadot transfers"
    }
  }
}

The number of transfers should change assuming that the processor is continually running with out any issues.

Accounts

You can specify which chain you want in the where clause of your GQL query using the substrateChain_eq

query {
  accounts(
    limit: 100,
    where: {
      substrateChain_eq: "kusama"
    }
  ) {
    id,
    wallet,
    balance,
    substrateChain,
  }
}

Response:

{
  "data": {
    "accounts": [
      {
        "id": "0x2dcc5638cc58a19c9050837a6b9537346df8caea9e7d72e5a242ada57494c4e1",
        "wallet": "5D6kiHiTYaEQpHcNt2bvWRCfCFQSwda9nkSHiWamTqpNJmtj",
        "balance": "1000000000000",
        "substrateChain": "kusama"
      },
      {
        "id": "0x3cd95f701f08e5cd36078c7e63d1254f0e66977175ed55cc5e42af05d0d8af37",
        "wallet": "5DSVJZ55ey4hsAA9x6gRomqaioDovTrWRKeft3CCcov66giz",
        "balance": "1000000000000",
        "substrateChain": "kusama"
      },
      {
        "id": "0xff32d83b31df2d73a27ef6763e297b329977f62e4b87d775eb4a2bae26b11eed",
        "wallet": "5HqKAv7QLmGqk2QgkUV77JA13JTcFbWW65DskVzZECdbRUeT",
        "balance": "1000000000000",
        "substrateChain": "kusama"
      },
      ...
      {
        "id": "0x1df2dac62aeb811610c013b84eabac613a4edc1cf534b7f6ad5e16bac26d45a7",
        "wallet": "5CjyPVqJGZJdzrxXLCQ25Qa4PJgs6xvB5693WcbiouESy8RG",
        "balance": "1000000000000",
        "substrateChain": "kusama"
      }
    ]
  }
}

Chain Type Generation

Currently, in order to support multiple chains with the type generation (typegen). You can generate each chains generated code by commenting/uncommenting the typegen portion of the manifest.yml file.

Example for polkadot typegen

Here is what the typegen portion of the manifest.yml file will look like to generate the Polkadot .ts files

typegen:
  metadata:
    source: wss://kusama-rpc.polkadot.io/
    blockHash: '0x98af7de0c69107bcdd5033d84cfee0e331514dc42e0a45b9d64e82b0dcec13a2'
  events:
    - balances.Transfer
  calls:
    - timestamp.set
  outDir: chain/kusama

# typegen:
#   metadata:
#     source: wss://rpc.polkadot.io/
#     blockHash: '0x85b133210631562ef26d3cb1a0781396ab13fa5e2118a74c4f8ed59c6cf8c9ab'
#   events:
#     - balances.Transfer
#   calls:
#     - timestamp.set
#   outDir: chain/polkadot

In order to generate the Kusama .ts files you can flip the commented section of the typegen i.e.:

# typegen:
#   metadata:
#     source: wss://kusama-rpc.polkadot.io/
#     blockHash: '0x98af7de0c69107bcdd5033d84cfee0e331514dc42e0a45b9d64e82b0dcec13a2'
#   events:
#     - balances.Transfer
#   calls:
#     - timestamp.set
#   outDir: chain/kusama

typegen:
  metadata:
    source: wss://rpc.polkadot.io/
    blockHash: '0x85b133210631562ef26d3cb1a0781396ab13fa5e2118a74c4f8ed59c6cf8c9ab'
  events:
    - balances.Transfer
  calls:
    - timestamp.set
  outDir: chain/polkadot

NOTE: There has been some work done to support multiple metadata sources and generate the type generations given multiple entries/sources here. This work is currently being tested and has yet to be published on NPM as a package and imported into this project for ease of generating the multiple chain types.

Project structure

Hydra tools expect a certain directory layout:

generated - model/server definitions created by codegen. Do not alter the contents of this directory manually.
server-extension - a place for custom data models and resolvers defined via *.model.ts and *.resolver.ts files.
chain - data type definitions for chain events and extrinsics created by typegen.
mappings - mapping module.
.env - hydra tools are heavily driven by environment variables defined here or supplied by a shell.

Development flow

If you modified schema.graphql:

# Run codegen to re-generate model/server files
npm run codegen

# Analyze database state and create a new migration to match generated models
npm run db:create-migration # add -n "myName" to skip the migration name prompt

# Apply the migrations
npm run db:migrate

You might want update the Initial migration instead of creating a new one (e.g. during the development phase when the production database is not yet set up). In that case it convenient to reset the database schema and start afresh:

rm db/migrations/LastUnappliedMigration.ts
npm run db:reset
npm run db:create-migration
npm run db:migrate

To generate new type definitions for chain events and extrinsics:

# Review typegen section of manifest.yml (https://docs.subsquid.io/hydra-typegen)

# Delete old definitions
rm -rf chain

# Run typegen tool
npm run typegen

Self-hosted indexer

It is recommended to use a readily set up indexer if available. It takes some time for a freshly started indexer to get in sync with chain and catch the events.

Have a look at ./indexer/docker-compose.yml for an example of how you can set up a self-hosted version.

Misc

For more details, please checkout https://docs.subsquid.io.

Developer Notes

None known at the moment.

cpurta / multi-squid-api