falcon-log-agent

Synopsis

Falcon-Log-Agent is an open-source log collection tool designed to crawl and log feature information from streaming logs.

The acquired feature information is associated with the open source Open-Falcon monitoring system. It can be used for measuring business indicators and for building stability.

Highlights

Powerful and flexible
Calculating with high efficiency
Friendly strategy configuration supporting secondary development
Support for a variety of computing methods（cnt、avg、sum、max、min etc.）
Development language: entire agent written in Golang

Data Model

The data model is similar to the data model in Open-Falcon: metric and endpoint with a couple of key value tags. Here are two examples:

{
    metric: load.1min,
    endpoint: open-falcon-host,
    tags: srv=falcon,idc=aws-sgp,group=az1,
    value: 1.5,
    timestamp: `date +%s`,
    counterType: GAUGE,
    step: 60
}
{
    metric: net.port.listen,
    endpoint: open-falcon-host,
    tags: port=3306,
    value: 1,
    timestamp: `date +%s`,
    counterType: GAUGE,
    step: 60
}

Geting Started

clone & build

git clone https://github.com/didi/falcon-log-agent.git && cd falcon-log-agent && sh build.sh

change configs

# base config
cp cfg/dev.cfg cfg/cfg.json
vim cfg/cfg.json

# strategy config
cp cfg/strategy.dev.json cfg/strategy.json
vim cfg/strategy.json

start & stop & status service

 # start
./control start

# stop
./control stop

# status
./control status

BasicConfig

{
	# configs about log
    "log" : {
        "log_path" : "/var/log/log-agent",
        "log_level" : "INFO",
        "log_rotate_size" : 200,
        "log_rotate_num" : 10
    },

    # configs about service
    "http" : {
        "http_port" : 8003
    },

    # configs about strategy & update
    "strategy" : {
        "update_duration" : 60,
        "default_degree" : 6
    },

    # configs about worker
    "worker" : {
        "worker_num" : 10,
        "queue_size" : 1024000,
        "push_interval" : 1,
        "push_url" : "http://127.0.0.1:1988/v1/push"
    },

    # configs about resource self-limited
    "max_cpu_rate": 0.2,
    "max_mem_rate": 0.05
}

StrategyConfig

[
    {
        "id":1,
        "name":"流量500错误数",
        "file_path":"/home/work/myService/log/access.log",
        "time_format":"dd/mmm/yyyy:HH:MM:SS",
        "pattern":"service error 500, num=(\\d+)",
        "exclude":"unimport-request",
        "step":10,
        "tags":{
            "provice" : "province=(\\s+)"
        },
        "func":"cnt",
        "degree":6,
        "comment":"i'm comment"
    }
]

Support TimeFormat List

dd/mmm/yyyy:HH:MM:SS
dd/mmm/yyyy HH:MM:SS
yyyy-mm-ddTHH:MM:SS
dd-mmm-yyyy HH:MM:SS
yyyy-mm-dd HH:MM:SS
yyyy/mm/dd HH:MM:SS
yyyymmdd HH:MM:SS
mmm dd HH:MM:SS

And you can also add timeFormat in src/common/utils/util.go

will0815 / falcon-log-agent