danielqsj/datakit

Website • Doc

Read this in other languages.

DataKit is an open source, integrated data collection agent, which provides full platform (Linux/Windows/macOS) support and has comprehensive data collection capability, covering various scenarios such as host, container, middleware, tracing, logging and security inspection.

Key Features

Support collection of metrics, logging and tracing
Fully support Kubernetes ecology
Pipeline: Simple structured data extraction
Supports third-party data import:
- Telegraf
- Prometheus
- Statsd
- Fluentd
- Function
- Tracing related(OpenTelemetry/DDTrace/Zipkin/Jaeger/Skywalking)

Changelog

All DataKit changelog refers to here.

Minimal Requirements

OS	Arch	Install Path
Linux Kernel 2.6.23+	amd64/386/arm/arm64	`/usr/local/datakit`
macOS 10.12+(Why)	amd64	`/usr/local/datakit`
Windows 7+/Server 2008R2+	amd64/386	64-bit：`C:\Program Files\datakit` 32-nit：`C:\Program Files(32)\datakit`

Install DataKit

We can directly obtain the DataKit installation command from guance cloud. Most of the installation commands seems like that:

Linux & Mac

DK_DATAWAY="https://openway.guance.com?token=<YOUR-TOKEN>" bash -c "$(curl -L https://static.guance.com/datakit/install.sh)"

Windows

$env:DK_DATAWAY="https://openway.guance.com?token=<YOUR-TOKEN>";Set-ExecutionPolicy Bypass -scope Process -Force; Import-Module bitstransfer; start-bitstransfer -source https://static.guance.com/datakit/install.ps1 -destination .install.ps1; powershell .install.ps1;

Kubernetes DaemonSet

For more documentations about DataKit installation, see here.

Install unstable release

We also released the unstable DataKit, we can install via

Linux & Mac

DK_DATAWAY="https://openway.guance.com?token=<YOUR-TOKEN>" bash -c "$(curl -L https://static.guance.com/datakit/community/install.sh)"

Windows

$env:DK_DATAWAY="https://openway.guance.com?token=<YOUR-TOKEN>";Set-ExecutionPolicy Bypass -scope Process -Force; Import-Module bitstransfer; start-bitstransfer -source https://static.guance.com/datakit/community/install.ps1 -destination .install.ps1; powershell .install.ps1;

Kubernetes DaemonSet

# We should use the unstable version yaml
wget https://static.guance.com/datakit/community/datakit.yaml

Build From Source

DataKit building relies on some external tools/libs, we must install them all before compile the source code.

We do not support build DataKit on Windows.

Go-1.16.4+
gcc-multilib: Used to build Oracle input(apt-get install gcc-multilib)
tree: After building datakit, tree used to show all bianries(apt-get install tree)
packr2: Used to package resources(mainly documents)
goyacc: Used to build grammar for Pipleine(go get -u golang.org/x/tools/cmd/goyacc)
Docker: Used to build DataKit image
lint related:
- gofumpt: Used to format go source code(go install mvdan.cc/gofumpt@latest)
- golangci-lint 1.42.1
eBPF related:
- clang 10.0+
- llvm 10.0+
- apt install go-bindata
Documentation exporting:
- waque 1.13.1+

Build

Clone

git clone https://github.com/DataFlux-cn/datakit.git

Building

cd datakit
make

If building ok, all binaries are generated under dist:

dist/
├── datakit-linux-amd64
│   ├── datakit             # DataKit main binary
│   └── externals      
│       ├── datakit-ebpf    # eBPF collector
│       ├── logfwd          # logfwd collector
│       └── oracle          # Oracle collector
└── local
    ├── installer-linux-amd64 # installer used fo Linux 
    └── version               # version descriptor

We can build all platforms(Linux/Mac/Windows) with following command:

make testing

Basic Usage

We can use help command to see more usage of DataKit:

datakit help

# Or

./dist/datakit-linux-amd64/datakit help

Contributing

Before contributing, check out some guideline of DataKit:

Read architecure introduciton
Read development guideline

Full Documentation

For full documents of DataKit, see DataKit Doc Repository。

danielqsj / datakit