Giters

omkarb / midgard

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

pipeline status

Midgard API

Midgard is a layer 2 REST API that provides front-end consumers with semi real-time rolled up data and analytics of the THORChain network. Most requests to the network will come through Midgard. This daemon is here to keep the chain itself from fielding large quantities of requests. You can think of it as a “read-only slave” to the chain. This keeps the resources of the network focused on processing transactions.

Running Midgard

Midgard can be run locally with native code or via Docker Compose. Midgard populates the PSQL database with content from the blockchain. Progress is traceable with the Prometheus Metrics propagated on http://localhost:8080/debug/metrics, specifically the measurements midgard_chain_cursor_height v.s. midgard_chain_height. Open http://localhost:8080/v2/doc in your browser.

Config

You can configure Midgard with a big config file, a list of smaller config files, or with environment variables.

The easiest is composing multiple config files with :, later configs overwrite values from earlier ones. A good starting point for config files is in config/ex directory. If you wish to edit the config files you can cp -r ./config/examples ./tmp/config (gitignored) and edit them there.

Second option: a full file is at config/config.json which assumes you are running a local ThorNode and a PSQL. You can copy this file to config/local.json (which is ignored by git) and make desired changes. If you wish to connect to a specific ThorNode the proper urls can be found in config/ex/net-local.json (localhost), config/ex/net-main.json (mainnet), config/ex/net-stage.json (stagenet)

Third option: Overwrite single config values with environment variables

Fields in nested structs are accessed using underscores. Examples:

  • MIDGARD_LISTEN_PORT env variable will override Config.ListenPort value
  • MIDGARD_TIMESCALE_PORT will override Config.TimeScale.Port value
  • MIDGARD_USD_POOLS="A,B,C" will override the UsdPools

Start native Midgard

# One time setup:
docker-compose up -d pg
mkdir -p ./tmp/blockstore

# run midgard
go run ./cmd/midgard/ config/ex/base.json:config/ex/pg.json:config/ex/bs-m.json:config/ex/net-main-9r.json

Docker Compose

Running with Docker Compose it's possible with a single config file at config/local.json or environment variables.

To allow Midgard to connect properly to Postgres do cp config/config.json config/local.json then edit local.json and change timescale.host to "pg".

Then:

# One time setup:
docker-compose up -d pg

docker-compose up --build midgard

Running Local ThorNode

To work on Midgard we don't need or want a proper validator setup, just the full thornode that follows and syncs the thorchain locally.

Clone the thornode repo from: https://gitlab.com/thorchain/thornode

Look up the current version and check it out. If you need the latest verion you are probably fine using the chaosnet-multichain branch.

Start the thornode by running make run-fullnode from build/docker/mainnet.

IMPORTANT! This will create a docker container named thornode and will store data in your home directory, under ~/.thornode. If you have anything important in one or the other, backup first!

To summarize:

git clone https://gitlab.com/thorchain/thornode.git
cd thornode
git checkout chaosnet-multichain
cd build/docker/mainnet
make run-fullnode

For midgard config use:

    "tendermint_url": "http://localhost:26657/websocket",
    "thornode_url": "http://localhost:1317/thorchain",

Upgrading local ThorNode

When the network switches to a newer version your local thornode will stop working: the docker container will be in a crash loop. To upgrade, remove the container, the docker image, pull, and restart:

docker stop thornode
docker rm thornode
docker rmi registry.gitlab.com/thorchain/thornode:mainnet
cd thornode/build/docker/mainnet
git pull
make run-fullnode

Websockets

Websockets is an experimental feature supported for Linux only. If you need to use it for develop using a different OS you may need to run Midgard using Docker.

Testing

docker-compose up -d pgtest
go test -p 1 ./...

External MR Pipeline Testing

To enable test pipelines on an external MR please consider to enable Shared runners

Repo Settings -> CI/CD -> Runners -> enable Shared Runners

State Checks

A cmd that checks the state recreated by Midgard through events and the actual state stored in the Thorchain can be run with:

go run ./cmd/statechecks config/ex/base.json:config/ex/pg.json:config/ex/bs-m.json:config/ex/net-main-9r.json:config/ex/loginfo.json

Connecting to Midgard's PostgreSQL DB

To inspect Midgard's DB (run manual queries etc.) connect with psql. Install postgres client tools; on Debian based systems:

sudo apt install postgres-client

And then:

psql -h localhost -U midgard midgard -p 5432

For test DB use port 5433; the pg2 instance is on port 6432. The password is password. To avoid entering it over and over again, do:

echo '*:*:midgard:*:password' >> ~/.pgpass && chmod 0600 ~/.pgpass

Alternatively, you can use the psql from within the appropriate Docker container (no need to install postgres-client on your machine):

docker exec -it midgard_pg_1 psql -h localhost -U midgard midgard

Trimming the database

Regenerating the database from height 1 can be time consuming. If there is a bug in a later point it's possible to trim back all database tables to just before the problematic point. This is useful to apply a bugfix quickly.

go run ./cmd/trimdb config/config.json HEIGHTORTIMESTAMP

Saving & copying the database

If you'd like to do some (potentially destructive) experiments with the database, it's probably a good idea to make a backup of it first, so you don't have to resync in case things don't go as expected.

Provided that the directory where you checked out Midgard code is named midgard the standard location of the pg database instance will be under /var/lib/docker/volumes/midgard_pg/_data. But you can check this with docker inspect on the appropriate docker container. Like this:

docker inspect midgard_pg_1 | jq -r '.[].Mounts | .[].Source'

Consider treating unset parameters as an error when substituting.

set -u

Creating a backup of the pg instance:

# choose where to put the backup:
backup_dir=/tmp/pgbackup
# query the location of the docker volume:
pg_volume=/var/lib/docker/volumes/midgard_pg/_data

# stop, backup, restart:
docker stop midgard_pg_1
sudo cp -a $pg_volume/ $backup_dir/
docker start midgard_pg_1

Restoring the DB from the backup:

docker stop midgard_pg_1
sudo rsync -ac --del $backup_dir/ $pg_volume/
docker start midgard_pg_1

Of course, you can do this with the pg2 or pgtest instances too.

Monitoring more than one chain

It is possible to rune more than one Midgard instance against different chains (e.g. main/testnet). Create two config files (e.g. mainnet.json, testnet.json):

  • set listen_port to 8080 and 8081
  • edit thornode and tendermint urls
  • set timescale/port to 5432 and 6432
docker-compose up -d pg
docker-compose up -d pg2
go run ./cmd/midgard tmp/mainnet.json
go run ./cmd/midgard tmp/testnet.json

Then you can check depths separately for them:

go run ./cmd/statechecks tmp/mainnet.json
go run ./cmd/statechecks tmp/testnet.json

Generated files

Some GraphQL or OpenApi files are generated.

You need to install a few things once.

  • For redoc-cli do npm install which takes dependencies form package.json
  • For oapi-codegen do go get github.com/deepmap/oapi-codegen/cmd/oapi-codegen which will provide $GOPATH/bin/oapi-codegen

Then from now you can regenerate files with:

make generated

Generating blockstore hashes

Midgard can read blockstore to speed up fetching from ThorNode. Blockstore consists of compressed files containing the raw Bloks in batches of 10K. These batches (chunks) are stored in a remote location. Midgard will download them on startup, but it accepts only if the hashes of the chunks match the predefined values.

To regenerate the hashes and store them in git do these two steps:

Fetch all blocks from thornode to have them locally:

# Stop midgard first.
go run ./cmd/blockstore/dump config

Save the hashes in the git repository:

(cd $blockstore_folder; sha256sum *) > resources/hashes/$chain_id

Format, Lint

You can run these before submit to make sure the CI will pass:

gofmt -l -s -w ./
docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint golangci-lint run -v

Please, use an editor that respects the project's .editorconfig settings. For example, for Visual Studio Code you can install the official EditorConfig extension: https://marketplace.visualstudio.com/items?itemName=EditorConfig.EditorConfig

Managing dependencies

To update a Go library dependency (because a vulnerability was discovered or we want to use a feature from a newer version), do the following:

go get github.com/team/repo@version
go mod tidy

The version can be latest if you want to upgrade to the latest version.

Adding a binary-only dependency

Sometimes we need a go library dependency not because we use it in the code, but because it provides some binary which we use in the build process, for example. You add this kind of dependency the same way as a normal dependency:

go get github.com/deepmap/oapi-codegen@latest

But, to prevent go mod tidy from removing this dependency it should be added to a .go file. In Midgard we use cmd/deps/deps.go for this purpose.

Architecture

The chain package reads the blockchain in chronological order. Blocks are parsed with events and persisted with internal/timeseries. The RDBM is almost a one-to-one mapping of the key-value entries from the THORChain. Aggregated values and tables are created separately in aggregate.go.

Package internal/api defines the HTTP interface. See internal/graphql for the query facilities (provided by internal/timeseries/stat).

Blocks are “committed” with an entry in the block_log table, including a block_timestamp. Queries give consistent [cachable] results when executed with a (time) db.Window within timeseries.LastBlock.

Bookmarks

Direct links:

Documentation:

About

License:Other

Languages

Language:HTML 64.3%Language:Go 33.5%Language:PLpgSQL 2.1%Language:Dockerfile 0.1%Language:Makefile 0.1%