BDS
Introduction
Blockchain Data Service (BDS) is that JD Cloud uses technical ways to store the chain-like and unstructured data of blockchain and synchronizes them to database service in real time, meanwhile provides a tool of data visualization.
Splitter is a separate module of the open source project which is blockchain data service (BDS) and provides data analysis services
Splitter is responsible for consuming blockchain data from message queue (kafka) and inserting data into persistent data storage services (relational database, data warehouse, etc.).
Architecture
Environment Deployment
Install BDS
Environment initialization
Before compiling and running BDS, you must install go's compilation environment locally: go install
Install Splitter steps
- Set the path of project :
$GOPATH/src/github.com/jdcloud-bds/bds/
- Input
go build -v github.com/jdcloud-bds/bds/cmd/bds-splitter
,compile to get executable file bds-splitter - Build new configuration file splitter.conf, see
/config/splitter_example.conf
configuration file template - Run programe
./bds-splitter -c splitter.conf
Install confluent and kafka
Install kafka
See kafka
Modify config/server.properties
- message.max.bytes=1048576000
Install confluent
see confluent
Unzip the confluent package and run Confluent REST Proxy
Modify /etc/kafka-rest/kafka-rest.properties
- max.request.size = 1048576000
- buffer.memory = 1048576000
- send.buffer.bytes = 1048576000
Database
Database we now support SQL Server, PostgreSQL, you can choose one as a data storage method.
SQL Server
PostgreSQL
After you run the database, you need to manually create new database and use the database name initialization splitter.conf.
Install Grafana
See Grafana Official
Source code
Development Steps
- Define the data structure of Kafka messages.
- Define table structure.
- Analyze Kafka message and store data in database.