coinbase / mesh-bitcoin

Bitcoin Mesh API Implementation

Mesh Bitcoin

MESH-BITCOIN IS CONSIDERED ALPHA SOFTWARE. USE AT YOUR OWN RISK.

This project is available open source under the terms of the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).

Overview

The mesh-bitcoin repository provides a reference implementation of the Mesh API for Bitcoin in Golang. This repository was created for developers of Bitcoin-like (a.k.a., UTXO) blockchains, who may find it easier to fork this reference implementation than write one from scratch.

Mesh is an open-source specification and set of tools that makes integrating with blockchains simpler, faster, and more reliable. The Mesh API is specified in the OpenAPI 3.0 format.

Requests and responses can be crafted with auto-generated code using Swagger Codegen or OpenAPI Generator, are human-readable (easy to debug and understand), and can be used in servers and browsers.

Features

• Mesh API implementation (both Data API and Construction API)
• UTXO cache for all accounts (accessible using the Mesh /account/balance API)
• Stateless, offline, curve-based transaction construction from any SegWit-Bech32 Address
• Automatically prune bitcoind while indexing blocks
• Reduce sync time with concurrent block indexing
• Use Zstandard compression to reduce the size of data stored on disk without needing to write a manual byte-level encoding

System Requirements

The mesh-bitcoin implementation has been tested on an AWS c5.2xlarge instance. This instance type has 8 vCPU and 16 GB of RAM.

Getting Started

1. Adjust your network settings to the recommended connections.
2. Install and run Docker as directed in the Deployment section below.
3. Run the Testnet:Online command.

Network Settings

To increase the load that mesh-bitcoin can handle, we recommend tunning your OS settings to allow for more connections. On a linux-based OS, you can run these commands (source):

sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_max_syn_backlog=10000
sysctl -w net.core.somaxconn=10000
sysctl -p (when done)

We have not tested mesh-bitcoin with net.ipv4.tcp_tw_recycle and do not recommend enabling it.

You should also modify your open file settings to 100000. This can be done on a linux-based OS with the command: ulimit -n 100000.

Memory-Mapped Files

mesh-bitcoin uses memory-mapped files to persist data in the indexer. As a result, you must run mesh-bitcoin on a 64-bit architecture (the virtual address space easily exceeds 100s of GBs).

If you receive a kernel OOM, you may need to increase the allocated size of swap space on your OS. There is a great tutorial for how to do this on Linux here.

Development

While working on improvements to this repository, we recommend that you use these commands to check your code:

• make deps to install dependencies
• make test to run tests
• make lint to lint the source code
• make salus to check for security concerns
• make build-local to build a Docker image from the local context
• make coverage-local to generate a coverage report

Image Installation

Running these commands will create a Docker image called mesh-bitcoin:latest.

Installing from GitHub

To download the pre-built Docker image from the latest release, run:

curl -sSfL https://raw.githubusercontent.com/coinbase/mesh-bitcoin/master/install.sh | sh -s

Do not try to install mesh-bitcoin using GitHub Packages!

Installing from Source

After cloning this repository, run:

make build-local

Run Docker

Running these commands will start a Docker container in detached mode with a data directory at <working directory>/bitcoin-data and the Mesh API accessible at port 8080.

Required Arguments

MODE - Determines whether Mesh can make outbound connections.

• Type: String
• Options: ONLINE, OFFLINE
• Default: None

NETWORK - The Ethereum network to launch or communicate with.

• Type: String
• Options: MAINNET, ROPSTEN, RINKEBY, GOERLI or TESTNET
• Default: ROPSTEN, but only for backwards compatibility if you use TESTNET

PORT - The port to use for Mesh.

• Type: Integer
• Options: 8080, any compatible port number.
• Default: None

Command Examples

You can run these commands from the command line. If you cloned the repository, you can use the make commands shown after the examples.

Mainnet:Online

Uncloned repo:

docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 -p 8333:8333 mesh-bitcoin:latest

Cloned repo:

make run-mainnet-online

Mainnet:Offline

Uncloned repo:

docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=MAINNET" -e "PORT=8081" -p 8081:8081 mesh-bitcoin:latest

Cloned repo:

make run-mainnet-offline

Testnet:Online

Uncloned repo:

docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 18333:18333 mesh-bitcoin:latest

Cloned repo:

make run-testnet-online

Testnet:Offline

Uncloned repo:

docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "PORT=8081" -p 8081:8081 mesh-bitcoin:latest

Cloned repo:

make run-testnet-offline

Architecture

mesh-bitcoin uses the syncer, storage, parser, and server package from mesh-sdk-go instead of a new Bitcoin-specific implementation of packages of similar functionality. Below you can find an overview of how everything fits together:

Concurrent Block Syncing

To speed up indexing, mesh-bitcoin uses concurrent block processing with a "wait free" design (using the channels function instead of the sleep function to signal which threads are unblocked). This allows mesh-bitcoin to fetch multiple inputs from disk while it waits for inputs that appeared in recently processed blocks to save to disk.

Test the Implementation with the mesh-cli Tool

To validate mesh-bitcoin, install mesh-cli and run one of these commands:

• mesh-cli check:data --configuration-file mesh-cli-conf/testnet/config.json - This command validates that the Data API information in the testnet network is correct. It also ensures that the implementation does not miss any balance-changing operations.
• mesh-cli check:construction --configuration-file mesh-cli-conf/testnet/config.json - This command validates the blockchain’s construction, signing, and broadcasting.
• mesh-cli check:data --configuration-file mesh-cli-conf/mainnet/config.json - This command validates that the Data API information in the mainnet network is correct. It also ensures that the implementation does not miss any balance-changing operations.

Read the How to Test your Mesh Implementation documentation for additional details.

Contributing

You may contribute to the mesh-bitcoin project in various ways:

• Asking Questions
• Providing Feedback
• Reporting Issues

Read our Contributing documentation for more information.

You can also find community implementations for a variety of blockchains in the mesh-ecosystem repository.

Documentation

You can find the Mesh API documentation here.

Check out the Getting Started section to start diving into Mesh.

Related Projects

• mesh-sdk-go — The mesh-sdk-go SDK provides a collection of packages used for interaction with the Mesh API specification.
• mesh-specifications — Much of the SDK code is generated from this repository.
• mesh-cli — Use the mesh-cli tool to test your Mesh API implementation. The tool also provides the ability to look up block contents and account balances.

Sample Implementations

You can find community implementations for a variety of blockchains in the mesh-ecosystem repository.

License

This project is available open source under the terms of the Apache 2.0 License.

© 2022 Coinbase