UNDER DEVELOPMENT NOT YET RELEASED

LinearLogFlow

LinearLogFlow is a service that runs on your servers and sends all your json-formatted log files to Elasticsearch.

Every line in the log files are expected to be a json object.

Example of a log file

{"@timestamp":"2015-04-22T15:02:45+02:00","level":"Info","message":"Starting","version":"1.0"}
{"@timestamp":"2015-04-22T15:02:46+02:00","level":"Info","message":"Found 5 servers", "servers":5}
{"@timestamp":"2015-04-22T15:02:48+02:00","level":"Info","message":"User logged on","user":"HCanber"}

That's it. The line can contain anything as long as it's a valid json object. LinearLogFlow will take everything on the line and push it to Elasticsearch

LinearLogFlow
LogFlow

ELK Stack

The ELK Stack is well known is a setup combining Elasticsearch for indexing log files, Logstash for transforming and pushing logs into Elasticsearch, and Kibana for analysing and visualizing the data.

In the .Net world, Logstash can be replaced with LogFlow. In LogFlow the configuration and the transformation of log files is handled with .Net code (that you have to write). Excellent choice for transforming IIS log files for example.

However, if your log files already are in the correct json format LinearLogFlow pushes your log lines as-is to Elasticsearch.

SELK stack

The absolutely easiest way to write json log files is using Serilog. So the SELK stack is: Serilog, Elasticsearch, LinearLogFlow, Kibana

Install

.Net Framework is required on the server.

Download LinearLogFlow
Copy it to the a directory on the server
Edit logs.config and Nlog.config, see below
Execute LogFlow.exe install. Add --sudo if you don't have administrator privileges.
Start the service LogFlow.exe start or manually in the Services Manager (run services.msc)

Start & Stop

When LinearLogFlow has been installed as described above, you can start and stop the service by executing

LogFlow.exe start
LogFlow.exe stop
or manually in the Services Manager (run services.msc)

Uninstall

LogFlow.exe uninstall. Add --sudo if you don't have administrator privileges.

Help

LogFlow.exe help See TopShelf Documentation for more information.

Configuration

The logs.config file is where to specify the Elasticsearch server to connect to, and where the log files can be found.

Example of a logs.config file

<config>
  <server uri="http://elasticserver:9200">
    <index indexName="log-{yyyyMM}">
      <log type="serviceA" path="C:\ServiceA\logs\log-*.txt" />
      <log type="systemX"  path="D:\SystemX\log-*.txt" />
    </index>
  </server>
</config>

In the example above, two types of log files for two different systems will be collected and sent to the specified Elasticsearch server. The index name is based on each lines timestamp field (by default it is @timestamp´) so we will get a new index every month. The logs for serviceA will be indexed with the _type field set to serviceA and the logs for systemX will have _type set to systemX.

Index Names

Index names can either be fixed, like indexName="log", meaning all logs will end up in the same index (not recommended).

<index indexName="log">

Or based on each line's timestamp by specifying a custom date format string between {...}

<index indexName="log-{yyyyMM}">

log-yyyyMM will create a new index every month.

Change timestamp property

If the indexName contains a custom date format string, for example log-{yyyyMM} then every line must contain a timestamp property. By default @timestamp, timestamp, datetime, date, time will be used (in that order and in any casing). So if a line in your log file looks like below you do not need to change anything.

{"@timestamp":"2015-04-22T15:02:45+02:00","level":"Info","message":"Starting","version":"1.0"}

To specify another timestamp property, for example "thedate", set timestamp="thedate":

<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" timestamp="thedate" />

With this setting you're now able to index lines like this:

{"thedate":"2015-04-22T15:02:45+02:00","level":"Info","message":"Starting","version":"1.0"}

Use | as a separator to specify more than one field:

<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" timestamp="thedate|now|@timestamp" />

With this setting you're now able to index lines like this:

{"now":"2015-04-22T15:02:45+02:00","level":"Info","message":"Starting","version":"1.0"}
{"@timestamp":"2015-04-22T15:02:46+02:00","level":"Info","message":"Found 5 servers", "servers":5}
{"thedate":"2015-04-22T15:02:48+02:00","level":"Info","message":"User logged on","user":"HCanber"}

Encoding

By default the encoding for UTF-8, UTF-16 (Big and Little endian) and UTF-32 (Big and Little endian) will be detected automatically if the file starts with a Byte Order Mark (BOM). If no BOM is found, UTF-8 is used. To specify another set encoding to one of the following values ascii, utf-8, utf-16be, utf-16le (or unicode), utf-32le, utf-32be, utf-7.

Example

<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" encoding="ascii" />

You may also specify a default encoding using the property defaultEncoding on the index element that applies to all child log elements, unless they overrides it.

Example

<config>
  <server uri="http://elasticserver:9200" defaultEncoding="utf-16le">
    <index indexName="log-{yyyyMM}">
      <log type="serviceA" path="C:\ServiceA\logs\log-*.txt" encoding="ascii" />
      <log type="systemX"  path="D:\SystemX\log-*.txt" />
    </index>
  </server>
</config>

The encoding UTF-16 Little Endian will be used for systemX logs and as serviceA has specified encoding ASCII will be used for its files.

TTL – Time To Live

If a value has been specified for ttl, the property _ttl will be set in every line sent to Elasticsearch. The format is a value followed one of the units ms, s, h, d, w.

Example

<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" ttl="4w" />

Every line for serviceA will contain _ttl:"4w" meaning a line will be deleted from Elasticsearch automatically after 4 weeks. By default _ttl is not set.

You may also specify a default ttl using the property defaultTtl on the index element that applies to all child log elements, unless they overrides it.

Example

<config>
  <server uri="http://elasticserver:9200" defaultTtl="5d">
    <index indexName="log-{yyyyMM}">
      <log type="serviceA" path="C:\ServiceA\logs\log-*.txt" ttl="4w" />
      <log type="systemX"  path="D:\SystemX\log-*.txt" />
    </index>
  </server>
</config>

Every line from serviceA will contain _ttl:"4w" and every line from systemX will contain _ttl:"5d" meaning a line will be deleted from Elasticsearch automatically after 4 weeks and 5 days, respectively.

Note! If the log line already contains a _ttl value, it will be sent to Elasticsearch. If ttl on <log ... /> or defaultTtl on <index ... > has been specified it will overwrite any existing _ttl value.

See _ttl field in Elasticsearch Documentation for more information.

Include `@source`

Set addSource="true" to include the machine name hosting the LinearLogFlow instance in a @source property.

<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" addSource="true" />

If LinearLogFlow is running on the machine ProductionServer1 then every line in ElasticSearch that LinearLogFlow posts will contain @source: "ProductionServer1"

Mapping

To specify how properties should be indexed you may specify a mapping per type. You do this by creating a separate file and point to that in the mapping property.

<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" mapping="mapping.json" />

Create the mapping json file without the type property, in this example without a tweet property:

{
	"properties" : {
		"correlationId " : {
			"type": "string",
			"index": "not_analyzed"
		}
	}
}

This means that the same mapping file may be used for different types:

<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" mapping="mapping.json" />
<log type="systemX"  path="D:\SystemX\log-*.txt" mapping="mapping.json" />

You may also specify defaultMapping on the index element that will be applied to all log elements. If a log element specifies a mapping it will be applied after the defaultMapping.

<index indexName="log-{yyyyMM}" defaultMapping="mapping.json">
  <log type="serviceA" path="C:\ServiceA\logs\log-*.txt" />
  <log type="systemX"  path="D:\SystemX\log-*.txt" mapping="systemXmapping.json" />
</index>

In the example above the mapping in mapping.json will be applied for the serviceA type. For systemX mapping.json will be applied first, and then the mapping in systemXmapping.json will be merged. The merging is done by Elasticsearch (meaning LinearLogFlow first will PUT mapping.json and then PUT systemXmapping.json)

Note! The mapping will be put when a new index is created. It will also be put the first time LinearLogFlow writes a log line to an index, no matter if the index existed or not, after it has been restarted.

Example A log line is to be inserted into index log-201504. As it's a new index, it's created (potentially with an index template, see below). After that, the mapping is put into the index. The log line is inserted. Let's say that LinearLogFlow at this point is restarted. WHen LinearLogFlow is up and running it encounters a new log line that is to be inserted into the existing index log-201504. As it's the first time a line is inserted into that index, after LinearLogFlow was started, the mapping will be PUT again.

####Separating mappings into several files You may specify more than one mapping file for defaultMapping and mapping by separating the paths with |. The mappings will be put to the type mapping in the specified order.

<config>
	<server uri="http://elasticserver:9200" defaultTtl="5d">
		<index indexName="service-log-{yyyyMM}" defaultMapping="mapping.json|services-mapping.json">
			<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" />
			<log type="serviceB"  path="D:\serviceB\log-*.txt" mapping="serviceB-mapping.json" />
		</index>
		<index indexName="appe-log-{yyyyMM}" defaultMapping="mapping.json|apps-mapping.json">
			<log type="appX" path="C:\AppX\log-*.txt" />
			<log type="appY" path="C:\AppY\log-*.txt" mapping="specialApp-mapping.json|appY-mapping.json"/>
		</index>
	</server>
</config>

In the example above the types will receive the mappings in this order:

Type	Mapping
serviceA	`mapping.json` `services-mapping.json`
serviceB	`mapping.json` `services-mapping.json` `serviceB-mapping.json`
appX	`mapping.json` `apps-mapping.json`
appY	`mapping.json` `apps-mapping.json` `specialApp-mapping.json` `appY-mapping.json`

See http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html and http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html for more information on mapping in Elasticsearch.

Index template

An index template can be specified on the index element by specifying a file on the indexTemplate property. The file must be a json file and follow the format described in Elasticsearch documentation. The template will be written when a new index is created.

<index indexName="log-{yyyyMM}" indexTemplate="template.json">

Posting to a cluster

Set isCluster="true" to specify that the specified server belongs to a cluster to be able to failover to other nodes in the cluster, if the specified server goes down.

<server uri="http://elasticserver:9200" isCluster="true">

To specify more seed nodes separate them with |.

<server uri="http://elasticserver1:9200|http://elasticserver2:9200" isCluster="true">

Posting different logs to different servers

You may specify more than one server element, to post to different servers

<config>
  <server uri="http://elasticserverA:9200">
    <index indexName="log-{yyyyMM}">
      <log type="serviceA" path="C:\ServiceA\logs\log-*.txt" />
    </index>
  </server>
  <server uri="http://elasticserverB:9200">
    <index indexName="log-{yyyyMM}">
      <log type="systemX"  path="D:\SystemX\log-*.txt" />
    </index>
  </server>
</config>

In this example files for serviceA will be posted to elasticserverA and files for systemX will be posted to elasticserverB.

Post multiple log files to the same index type

It's possible to have several log elements have the same type property to insert logs from different locations into the same index and type. The name property must then be specified on at least one of the log elements, in order to make the log elements distinguishable from each other

<log type="serviceA" path="C:\ServiceA\logs\log-*.txt" />
<log type="serviceA" name="systemX"  path="D:\SystemX\log-*.txt" />

The name property is an optional property, which defaults to the value of type, so in the example above, the first log element will have type="serviceA" and name="serviceA". The second log element will have type="serviceA" and name="systemX".

Note! Name property has no effect on Elasticsearch. It's only used internally in LinearLogFlow.

Configuration – Reference

Configuration is specified in a xml file called logs.config

<config>
  <server uri="http://localhost:9200" [isCluster="true|false"]>
    <index indexName="log-{yyyyMM}" [indexTemplate="indexTemplate.json"] [defaultEncoding="utf-8"] [defaultTtl="31d"] [defaultMapping="mapping.json"] >
      <log type="indexType" path="C:\logs\log-*.txt" [name="indexType"] [encoding="utf-8"] [ttl="31d"] [mapping="mapping.json"] [addSource="true|false"] [timestamp="@timestamp"]/>
    </index>
  </server>
</config>

Element	Property		Description
`server`		Required	At least one `server` element is required. To index to different Elasticsearch servers/clusters the config file may contain more than one `server` element. More info
	`uri`	Required	The uri to the server. Example: `http://localhost:9200`. For clusters specify seed nodes by separating them by `\|`. Example: `http://server1:9200\|http://server2:9200` More info
	`isCluster`	Optional Default: `false`	Set to `true` to specify that the specified server/servers are part of a cluster. More info
`index`		Required	Every `server` element must contain at least one `index` element. More than one is allowed.
	`indexName`	Requried	The name of the index. May use custom date format specifiers to create new indices based on the timestamp on each log line. Example: `log-{yyyyMM}` More info
	`indexTemplate`	Optional	A json file containing the settings and mappings for the index More info
	`defaultEncoding`	Optional Default: `utf-8`	The default encoding of the log files. By default UTF-8, UTF-16 (Big and Little endian) and UTF-32 (Big and Little endian) will be detected automatically if the file starts with a Byte Order Mark (BOM). If no BOM is found, UTF-8 is used. Example: `ascii` More info
	`defaultTtt`	Optional	If specified the value will be written in the `_ttl` property on each line/document. The format is a value followed one of the units `ms`, `s`, `h`, `d`, `w`. Example: `4w` More info
	`defaultMapping`	Optional	A path to a json file containg the mapping that will be applied to all index types inserted by LinearLogFlow. More than one may be specified by separating them by `\|`. Example: `mapping.json` and `mapping.json\|mapping2.json` More info
`log`		Required	Every `index` element must contain at least one `log` element. More than one is allowed. Every `log` must be unique. The uniqueness is determined by the value of `name` which defaults to the value of `type`. If two or more share the same `type` value, `name` must be manually specified to ensure uniqueness.
	`type`	Required	The index type or mapping type under which the log will be inserted. Example: `SystemA` See Elasticsearch documentation
	`path`	Required	The path to where log files can be found. May contain the wildcard `` in the file name, but not in the directory name. Example: `C:\logs\log-.txt`. To collect files from more than one directory, use a `log` element for every directory and set different `name` values on every element.
	`name`	Optional Defaults to the value of `type`	The name is optional as long as all `log` elements have unique `type` values. If two or more `log` share the same `type` value then a `name` must be specified to make them distinguishable from each other. Note The name is only used internally by LinearLogFlow and is never written to Elasticsearch. More info
	`encoding`	Optional Default: `utf-8`	The encoding of the log files. By default UTF-8, UTF-16 (Big and Little endian) and UTF-32 (Big and Little endian) will be detected automatically if the file starts with a Byte Order Mark (BOM). If no BOM is found, UTF-8 is used. Overrides `defaultEncoding`, if it has been specified. Example: `ascii` More info
	`ttl`	Optional	If specified the value will be written in the `_ttl` property on each line/document. The format is a value followed one of the units `ms`, `s`, `h`, `d`, `w`. Overrides `defaultTtl`, if it has been specified. Example: `4w` More info
	`mapping`	Optional	A path to a json file containg the mapping that will be applied to the index for the specified `type`. More than one may be specified by separating them by `\|`. If `defaultMapping` has been specified on the parent `index` element these will be applied first. Example: `mapping.json` and `mapping.json\|mapping2.json`. More info
	`addSource`	Optional Default: `false`	If set to `true` the machine name that hosts the LinearLogFlow instance will be ritten in the `@source` property. More info
	`timestamp`	Optional Default: `@timestamp`, `timestamp`	If `indexName` contains a custom date format string, for example `log-{yyyyMM}` then every line must contain a timestamp property. By default `@timestamp`, `timestamp`, `datetime`, `date`, `time` will be used (in that order and in any casing). Specify this if the timestamp is in another property in your log files. If the log files can contain the timestamp in different propertys, separate the names with `\|`. Example: `thedate\|now` More info

LogFlow

LinearLogFlow is based on LogFlow.

HCanber / LinearLogFlow

LinearLogFlow

ELK Stack

SELK stack

Install

Start & Stop

Uninstall

Help

Configuration

Index Names

Change timestamp property

Encoding

TTL – Time To Live

Include `@source`

Mapping

Index template

Posting to a cluster

Posting different logs to different servers

Post multiple log files to the same index type

Configuration – Reference

LogFlow

About

Languages

LinearLogFlow

ELK Stack

SELK stack

Install

Start & Stop

Uninstall

Help

Configuration

Index Names

Change timestamp property

Encoding

TTL – Time To Live

Include @source

Mapping

Index template

Posting to a cluster

Posting different logs to different servers

Post multiple log files to the same index type

Configuration – Reference

LogFlow

About

Languages

Include `@source`