ZettaStor DBS

Content Index

About ZettaStor DBS
- System Architecture Diagram
- Schematic Diagram
Technical Advantages
Versions and Features
User Documentation
- Operating System Installation
- Getting Started
- Advanced Configuration
- User Manual
Developer Documentation

What is ZettaStor DBS

ZettaStor DBS provides enterprise-level business storage solutions with high availability, high performance, easy expansion and easy maintenance for large-scale virtualization, private cloud and container environments. ZettaStor DBS, the solid digital base for cloud-based core applications, is independently developed by Pengyun Network.

ZettaStor DBS comes in two distributions: the Community Edition and Enterprise Edition. The source code for the Community Edition is distributed under the Apache License 2.0 license.

System Architecture Diagram

Schematic Diagram

Technical Advantages

ZettaStor DBS is the first product developed by Pengyun Network. Version 1.0 was officially released in 2015 and mainly adopted by cloud computing companies.

The first commercial version of DBS is developed according to the standard requirements of telecom operators, and has passed the tests of China's three major telecom operators for telecom equipment applications. In recent years, DBS has been polished for the financial industry, its stability and reliability has been further strengthened and has been operating stably in key business scenarios for 4 years.

ZettaStor DBS has completed the leap from telecom-grade to financial-grade. It is a distributed storage product with a high starting point and high requirements, which can meet the new and changing needs of the market.

Decentralized Architecture: A symmetric distributed architecture based on the decentralized blockchain network, eliminates traditional limitations and disadvantages on system scale, I/O performance, stability and reliability.
Large-scale Node Deployment: Stable high-performance is constantly maintained at optimal conditions, with the deployment of tens of thousands of nodes. The product's capacity and performance scale linearly with more storage nodes.
Sub-millisecond Latency: Minimized I/O processing path by directly reading/writing to disk, using an optimized network data path. It is capable of achieving sub-millisecond latencies with magnetic storage devices as primary storage configuration.
Recovery-in-place: When a hard disk or node failure occurs, a healthy one can immediately take over. The fault recovery time is less than 1 second, and there will be no impact on I/O performance.
Secure and Controllable: Complete intellectual property ownership with independent research-and-development, compatible with China-made hardware, operating systems and localized storage solutions.

Versions and Features

The following table compares some key features between various versions of ZettaStor DBS.

Features	Community Edition	Enterprise Edition
Volume Extension
Volume Shrink
Volume Clone
Volume Snapshot
iSCSI Protocol
PYD Protocol
Access Control
Data Access Control Strategy
Data Reconstruction Strategy
Load Balancing Strategy
Cache Boosting
System-wide Power Failure Protection
Disk Indicator
Disk Roaming
Alarm Management
User Management
Log Auditor
GUI Operation

Quick Start

I. Source Code Overview

The repositories of ZettaStor DBS must be organized in a hierarchy structure. This project has implemented Continuous Integration (CI) to ensure the quality and stability of the code. Please note that the CI status below may change as new commits are made to the codebase. If you have any questions or concerns, please use the GitHub Issues to report problems.

pengyun-root	pengyun-lib







	pengyun-dbs

II. Downloading the Source

Please use the following commands to download the source code:

export ROOT_PATH=git@github.com

git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-root
pushd pengyun-root

git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-lib
pushd pengyun-lib
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-core
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-database_core
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-models
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-dih_model
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-dih_client
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-query_log
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-configuration_common
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-monitor_common
popd

git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-dbs
pushd pengyun-dbs
git clone -b 1.0-OS $ROOT_PATH:zettastor/dbs-dnmodel
git clone -b 1.0-OS $ROOT_PATH:zettastor/dbs-models_related
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-driver_core
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-coordinator
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-infocenter
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-drivercontainer
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-deployment_daemon
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-datanode_core
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-datanode_service
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-datanode
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-webservice_adapter
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-utils
git clone -b 1.0-OS $ROOT_PATH:zettastor/pengyun-console
popd

popd

III. Setup a development environment

If you're in a UNIX-like environment, the packages required for compilation can be installed by the folowing commands

Note
The following instructions assume that you already have sufficient privileges, we will not go into details about using su or sudo and other privilege escalation operations.

RHEL/CentOS 7

yum install epel-release
yum -y install java-1.8.0-openjdk-devel thrift curl unzip wget perl-Data-Dumper perl-XML-Simple

# Install a newer version of Apache Maven
curl -LO https://downloads.apache.org/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz
tar -xvf apache-maven-3.5.4-bin.tar.gz --directory /opt
ln -s /opt/apache-maven-3.5.4 /opt/maven
chown -R root:root /opt/maven
echo '# Apache Maven Environment Variables' > /etc/profile.d/maven.sh
echo 'export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk/' >> /etc/profile.d/maven.sh
echo 'export PATH=/opt/maven/bin:${PATH}' >> /etc/profile.d/maven.sh

# Install a newer version of Protocol Buffers
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v3.5.1/protoc-3.5.1-linux-x86_64.zip
unzip protoc-3.5.1-linux-x86_64.zip -d /usr/local

RHEL/CentOS 8

yum install epel-release
yum install maven compat-openssl10 protobuf-compiler
yum install https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/t/thrift-0.9.1-15.el7.x86_64.rpm

RHEL 9

yum install maven unzip
yum install http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/compat-openssl10-1.0.2o-3.el8.x86_64.rpm
yum install https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/t/thrift-0.9.1-15.el7.x86_64.rpm
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v3.5.1/protoc-3.5.1-linux-x86_64.zip
unzip protoc-3.5.1-linux-x86_64.zip -d /usr/local

Debian 10/11, Ubuntu 18/20

apt-get update
apt-get install curl openjdk-11-jdk maven protobuf-compiler
curl -LO http://archive.ubuntu.com/ubuntu/pool/universe/t/thrift-compiler/thrift-compiler_0.9.1-2.1_amd64.deb
dpkg -i thrift-compiler_0.9.1-2.1_amd64.deb

SUSE/SLES 15

zypper install curl unzip maven thrift
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v3.5.1/protoc-3.5.1-linux-x86_64.zip
unzip protoc-3.5.1-linux-x86_64.zip -d /usr/local

macOS Catalina (10.15) or higher

# Install Homebrew package manager
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install JDK
brew install openjdk@11
ln -sfn $(brew --prefix)/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk

# Install Apache Maven
curl -LO https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
tar -xvf apache-maven-3.6.3-bin.tar.gz
mv apache-maven-3.6.3 /opt/
export M2_HOME="/opt/apache-maven-3.6.3"
export PATH="${M2_HOME}/bin:${PATH}"

# Install Protocol Buffers
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v3.5.1/protoc-3.5.1-osx-x86_64.zip
unzip protoc-3.5.1-osx-x86_64.zip -d /usr/local

# Install Apache Thrift
brew install thrift@0.9
export PATH="/usr/local/opt/thrift@0.9/bin:$PATH"

Other Architecture and Platform

As a general rule, the simplest way is to download a pre-built binary. If you would like to build from source code, please refer to the links below for details.

Apache Thrift: To build Thrift from source look at installation instructions. Pay attention to the OS notes, there are are some system specific requirements.
Protocol Buffers (Protobuf): First check whether you can download a Protobuf 3.5.1 pre-built binary. If you would like to build protoc binary from source, see the installation instructions.

IV. Compiling the code

Verification of Requirements

To compile ZettaStor DBS, you need:

Java Development Kit (JDK) 11
Apache Maven 3.5 or higher
Apache Thrift 0.9.x
Protocol Buffers 3.5.1

Please make sure that the following command is in the PATH environment variable of the system and returns the correct version number, for example:

$ mvn --version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 11.0.17, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64

$ thrift --version
Thrift version 0.9.1

$ protoc --version
libprotoc 3.5.1

Building with Maven

To build the package, use the following commands in the directory where pengyun-root/pom.xml is located

# Update version number from system environment
mvn versions:set-property -Dproperty=libthrift.version -DnewVersion=$(thrift --version | awk '{print $3}')
mvn versions:set-property -Dproperty=protobuf.version -DnewVersion=$(protoc --version | awk '{print $2}')
mvn clean install -Dproguard=off

It is also possible to run maven with multiple threads and skip test to speed up the builds

mvn -T 1C clean install -Dproguard=off -DskipTests

V. Build Installation Package

When compilation finishes, use the following command in the pengyun-root/pengyun-dbs/ directory to make a new installation package:

cd pengyun-root/pengyun-dbs/
mkdir -p /opt/deploy/
perl bin/CreateDeployPackage.pl -d /opt/deploy

When the packaging process finishes, a new installation package will be saved in the /opt/deploy directory. For more detailed use of the installation package, please refer to Getting Started.

Where can I find out more?

ZettaStor DBS Website

How to Contribute

Submit Your Code

After finishing the development of your code, you should submit a pull request to 1.0-OS branch and fill out a pull request template.

An automated code style check has been added in the project compilation process. Please check the compilation result before submitting the code to ensure that there are no code style errors. While the prerequisites above must be satisfied prior to having your pull request reviewed, the reviewer may ask you to complete additional design work, tests, or other changes before your pull request can be ultimately accepted.

Coding standard

Source code should be viewed and edited with your editor set to use two spaces per tab, with one tab used per indentation level. Spaces are used for other alignment within a line.

Most parts of the code follow Google Java Style; some parts of the code follow Oracle's Code Conventions -- mostly depending on who wrote the original version. Above all else, be consistent with what you modify, and keep whitespace changes to a minimum when modifying existing source. For new code, use Google Java Style.

License

Apache License 2.0