41north / besu-exflo

A plugin for the Besu enterprise-grade Ethereum client with the aim of making it easier to extract chain data into a variety of different data stores and processing pipelines.

A plugin for the Besu enterprise-grade Ethereum client with the aim of making it easier to extract chain data into a variety of different data stores and processing pipelines.

Written with ❤️ in Kotlin.

⚠️ Warning: This project is **DEPRECATED**, use it at your own risk!

🗒️ Features

Exflo can extract the following information from a Besu archive instance into either a Postgres database or a Kafka topic:

• Block headers.
• Transactions.
• Transaction traces.
• Log events for standards-compliant ERC20, ERC721, ERC777 and ERC1155 tokens.
• A detailed breakdown of Ether movements e.g. block rewards, tx fees, simple ether transfers and so on.
• Contract creations and self-destructs.
• Per block account state changes.

Some screenshots of captured data:

Plugin	Screenshots
Postgres
Kafka

⏲️ Try Exflo in 10 seconds

We offer two docker-compose files ready to launch Exflo configured for Ethereum's Ropsten network (and easily configurable to other networks).

Postgres

docker-compose -f docker/exflo/docker-compose.postgres.yml up

Kafka

docker-compose -f docker/exflo/docker-compose.kafka.yml up

Wait for docker to properly initialize each service. Once everything is ready navigate to http://localhost:8082, you will be greeted with either pgweb or kafkahq.

🚆 Development quick start

Ensure you have the following programs installed in your system:

• Java 11 or higher (AdoptOpenJDK, Zulu Community or OpenJDK)
• Docker
• Docker Compose
• direnv
• (Optional but highly recommendable) IntelliJ IDEA Community or Ultimate

Clone the repository:

git clone [e-mail]:41north/exflo.git

Generate Intellij's run configurations (it will read from the file intellij-run-configs.yml and auto create XML configurations to ease common operations inside the IDE):

./gradlew generateIntellijRunConfigs

Decide which data store you want to run:

Postgres

Terminal:

docker-compose -f docker-compose.exflo-postgres.yml up

Intellij:

Kafka

Terminal:

docker-compose -f docker-compose.exflo-kafka.yml up

Intellij:

Wait for each docker service to initialize.

After that, you can start processing with Exflo:

Postgres

Terminal:

./gradlew runPostgres

Intellij:

Kafka

Terminal:

./gradlew runKafka

Intellij:

You can now check the logs or open your browser and navigate to http://localhost:8082 and you will be greeted with either pgweb or kafkahq respectively.

Exflo will start processing immediately!

Usage with Besu

There are two possible ways of running Exflo with Besu.

Bundled docker images

We have provide a docker image that contains a pre-packaged version of Besu to make your life easier. This is the recommended way of running Exflo as you don't need to worry about placing the plugin in the correct folder path. Below is an an example using docker-compose:

version: "3.7"
services:
  besu:
    image: 41north/exflo:latest
    environment:
      BESU_LOGGING: INFO
      BESU_NETWORK: ROPSTEN
      BESU_SYNC_MODE: FULL
      BESU_DATA_PATH: /opt/besu/data
      BESU_PLUGIN_EXFLO_KAFKA_ENABLED: "false"
      BESU_PLUGIN_EXFLO_POSTGRES_ENABLED: "true"
      BESU_PLUGIN_EXFLO_POSTGRES_JDBC_URL: jdbc:postgresql://postgres/exflo?user=exflo&password=exflo

Have a look at the usage section for a full list of parameters.

JAR

If you are already using Besu with other plugins it is possible to include Exflo as a jar. Below are instructions assuming you are running Besu within docker:

1. Go to releases and download the tar or zip file and extract it.
2. Besu docker image relies upon a /etc/besu/plugins folder where it plugin jars. Ensure the Exflo jar is in this folder.

Here's an example of a possible docker configuration using docker-compose syntax and a bind mount for the plugin folder:

version: "3.7"
services:
  besu:
    image: hyperledger/besu:1.4.4
    volumes:
      - ./path/to/exflo-jar/:/etc/besu/plugins
    environment:
      BESU_LOGGING: INFO
      BESU_NETWORK: ROPSTEN
      BESU_SYNC_MODE: FULL
      BESU_DATA_PATH: /opt/besu/data
      BESU_PLUGIN_EXFLO_KAFKA_ENABLED: "false"
      BESU_PLUGIN_EXFLO_POSTGRES_ENABLED: "true"
      BESU_PLUGIN_EXFLO_POSTGRES_JDBC_URL: jdbc:postgresql://postgres/exflo?user=exflo&password=exflo

Have a look at the usage section for more information.

💻 Contribute

We welcome any kind of contribution or support to this project but before to do so:

• Read our development guide to understand how to properly develop on the codebase.
• Make sure you have read the contribution guide for more details on how to submit a good PR (pull request).

In addition you can always:

• Add a GitHub Star 🌟 to the project.
• ETH donations to this address!
• Tweet about this project.
• Write a review or tutorial:
  • Exflo: A plugin for Hyperledger Besu

❔ FAQ

Why Besu?

We chose Besu for several reasons:

1. It is enterprise ready.
2. It is written in Java, which allows us to work in Kotlin
3. It has a modern codebase with good patterns and documentation.
4. There is a plugin system allowing for customisation and integration without the need to maintain a fork.
5. Their community is open and welcoming (we recommend you join!).

Why not use the Web3 interface that every Ethereum client has?

If you have ever tried this, you will quickly realise that extracting even just the basic information from an Ethereum client via the Web3 interface requires a lot of requests, and some non-trivial logic to do well. On top of that, depending on the client (we won't name anyone in particular), you may find that under heavy load, such as when syncing for the first time, your client may become unstable and periodically core dump. Maximising throughput, whilst keeping the client happy, quickly becomes a tedious exercise.

Put simply it has been our experience that pulling via the Web3 interface is sub-optimal for a variety reasons which are better explored in a blog post.

📬 Get in touch

Exflo has been developed initially by °41North.

If you think this project would be useful for your use case and want to talk more about it you can reach out to us via our contact form or by sending an email to [e-mail]. We try to respond within 48 hours and look forward to hearing from you.

✍️ License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.