Load pino logs into Elasticsearch.
Install
npm install pino-elasticsearch -g
Usage
pino-elasticsearch
To send pino logs to elasticsearch:
cat log | pino-elasticsearch --node http://localhost:9200
Flags
-h | --help Display Help
-v | --version display Version
-n | --node the URL where Elasticsearch is running
-i | --index the name of the index to use; default: pino
will replace %{DATE} with the YYYY-MM-DD date
-t | --type the name of the type to use; default: log
-f | --flush-bytes the number of bytes for each bulk insert; default: 1000
-b | --bulk-size the number of documents for each bulk insert [DEPERCATED]
-l | --trace-level trace level for the elasticsearch client, default 'error' (info, debug, trace).
| --es-version specify the major version number of Elasticsearch (eg: 5, 6, 7)
(this is needed only if you are using Elasticsearch <= 7)
Usage as module
const pino = require('pino')
const pinoElastic = require('pino-elasticsearch')
const streamToElastic = pinoElastic({
index: 'an-index',
consistency: 'one',
node: 'http://localhost:9200',
'es-version': 7,
'flush-bytes': 1000
})
const logger = pino({ level: 'info' }, streamToElastic)
logger.info('hello world')
// ...
ECS support
If you want to use Elastic Common Schema, you should install @elastic/ecs-pino-format
, as the ecs
option of this module has been removed.
const pino = require('pino')
const ecsFormat = require('@elastic/ecs-pino-format')()
const pinoElastic = require('pino-elasticsearch')
const streamToElastic = pinoElastic({
index: 'an-index',
consistency: 'one',
node: 'http://localhost:9200',
'es-version': 7,
'flush-bytes': 1000
})
const logger = pino({ level: 'info', ...ecsFormat }, streamToElastic)
logger.info('hello world')
// ...
You can then use Kibana to
browse and visualize your logs.
Note: This transport works only with Elasticsearch version ≥ 5.
Dynamic index
It is possible to customize the index name for every log line just providing a function to the index
option:
const pino = require('pino')
const pinoElastic = require('pino-elasticsearch')
const streamToElastic = pinoElastic({
index: function (logTime) {
// the logTime is a ISO 8601 formatted string of the log line
return `awesome-app-${logTime.substring(5, 10)}`
},
consistency: 'one',
node: 'http://localhost:9200'
})
// ...
The function must be sync, doesn't throw and return a string.
Authentication
If you need to use basic authentication to connect with the Elasticsearch cluster, pass the credentials in the URL:
cat log | pino-elasticsearch --node https://user:pwd@localhost:9200
Setup and Testing
Setting up pino-elasticsearch is easy, and you can use the bundled
docker-compose.yml
file to bring up both
Elasticsearch and
Kibana.
You will need docker and
docker-compose, then in this project
folder, launch docker-compose up
.
You can test it by launching node example | pino-elasticsearch
, in
this project folder. You will need to have pino-elasticsearch
installed globally.
Acknowledgements
This project was kindly sponsored by nearForm.
License
Licensed under MIT.