rotating-file-stream

Description

Creates a stream.Writable to a file which is rotated. Rotation behaviour can be deeply customized; optionally, classical UNIX logrotate behaviour can be used.

Usage

var rfs    = require('rotating-file-stream');
var stream = rfs('file.log', {
    size:     '10M', // rotate every 10 MegaBytes written
    interval: '1d',  // rotate daily
    compress: 'gzip' // compress rotated files
});

Installation

With npm:

npm install rotating-file-stream

API

require('rotating-file-stream');

Returns RotatingFileStream constructor.

Class: RotatingFileStream

Extends stream.Writable.

[new] RotatingFileStream(filename, options)

Returns a new RotatingFileStream to filename as fs.createWriteStream does. The file is rotated following options rules.

filename {String|Function}

The most complex problem about file name is: "how to call the rotated file name?"

The answer to this question may vary in many forms depending on application requirements and/or specifications. If there are no requirements, a String can be used and default rotated file name generator will be used; otherwise a Function which returns the rotated file name can be used.

function filename(time, index)

time: {Date} If rotation by interval is enabled, the start time of rotation period, otherwise the time when rotation job started. If null, the not-rotated file name must be returned.
index {Number} The progressive index of rotation by size in the same rotation period.

An example of a complex rotated file name generator function could be:

function pad(num) {
    return (num > 9 ? "" : "0") + num;
}

function generator(time, index) {
    if(! time)
        return "file.log";

    var month  = time.getFullYear() + "" + pad(time.getMonth() + 1);
    var day    = pad(time.getDate());
    var hour   = pad(time.getHours());
    var minute = pad(time.getMinutes());

    return "/storage/" + month + "/" +
        month + day + "-" + hour + minute + "-" + index + "-" + filename;
}

var rfs    = require('rotating-file-stream');
var stream = rfs(generator, {
    size:     '10M',
    interval: '1d'
});

Note: If both rotation by interval and rotation by time are used, returned rotated file name must be function of both parameters time and index. Alternatively, rotationTime option can be used (to see below).

function filename(index)

index {Number} The progressive index of rotation. If null, the not-rotated file name must be returned.

If classical logrotate behaviour is enabled rotated file name is only a function of index.

Note: If part of returned destination path does not exists, the rotation job will try to create it.

options {Object}

compress: {String|Function|True} (default: null) Specifies compression method of rotated files.
interval: {String} (default: null) Specifies the time interval to rotate the file.
path: {String} (default: null) Specifies the base path for files.
rotate: {Integer} (default: null) Enables the classical UNIX logrotate behaviour.
rotationTime: {Boolean} (default: null) Makes rotated file name with time of rotation instead of start time of period.
size: {String} (default: null) Specifies the file size to rotate the file.
highWaterMark: {Number} (default: 16K) Proxied to new stream.Writable
mode: {Integer} (default: 0o666) Proxied to fs.createWriteStream

path

If present, it is prepended to generated file names.

size

Accepts a positive integer followed by one of these possible letters:

B: Bites
K: KiloBites
M: MegaBytes
G: GigaBytes

  size: '300B', // rotates the file when its size exceeds 300 Bytes
                // useful for tests

  size: '300K', // rotates the file when its size exceeds 300 KiloBytes

  size: '100M', // rotates the file when its size exceeds 100 MegaBytes

  size: '1G', // rotates the file when its size exceeds a GigaByte

interval

Accepts a positive integer followed by one of these possible letters:

s: seconds. Accepts integer divider of 60.
m: minutes. Accepts integer divider of 60.
h: hours. Accepts integer divider of 24.
d: days

  interval: '5s', // rotates the file at seconds 0, 5, 10, 15 and so on
                  // useful for tests

  interval: '5m', // rotates the file at minutes 0, 5, 10, 15 and so on

  interval: '2h', // rotates the file at midnight, 02:00, 04:00 and so on

  interval: '1d', // rotates the file at every midnight

compress

Due the nature of Node.js compression may be done with an external command (to use other CPUs than the one used by Node.js) or with internal code (to use the CPU used by Node.js). This decision is left to you.

Following fixed strings are allowed to compress the files with internal libraries:

bzip2 (not implemented yet)
gzip

To enable external compression, a function can be used or simply the boolean true value to use default external compression. The function should accept source and dest file names and must return the shell command to be executed to compress the file. The two following code snippets have exactly the same effect:

var rfs    = require('rotating-file-stream');
var stream = rfs('file.log', {
    size:     '10M',
    compress: true
});

var rfs    = require('rotating-file-stream');
var stream = rfs('file.log', {
    size:     '10M',
    compress: function(source, dest) {
        return "cat " + source + " | gzip -c9 > " + dest;
    }
});

Note: The shell command to compress the rotated file should not remove the source file, it will be removed by the package if rotation job complete with success.

rotationTime

As specified above, if rotation by interval is enabled, the parameter time passed to rotatle name generator is the start time of rotation period. Setting this option to true, parameter time passed is time when rotation job started.

rotate

If specified, classical UNIX logrotate behaviour is enabled and the value of this option has same effect in logrotate.conf file.

Events

Custom Events are emitted by the stream.

var rfs    = require('rotating-file-stream');
var stream = rfs(...);

stream.on('error', function(err) {
    // here are reported blocking errors
    // once this event is fired, the stream will be closed as well
});

stream.on('open', function() {
    // no rotated file is open (fired after each rotation as well)
});

stream.on('rotation', function() {
    // rotation job started
});

stream.on('rotated', function(filename) {
    // rotation job completed with success and produced given filename
});

stream.on('warning', function(err) {
    // here are reported non blocking errors
});

Rotation logic

Regardless of when and why rotation happens, the content of a single stream.write will never be split among two files.

by size

Once the not-rotated file is opened first time, its size is checked and if it is greater or equal to size limit, a first rotation happens. After each stream.write, the same check is performed.

by interval

The package sets a Timeout to start a rotation job at the right moment.

Under the hood

Logs should be handled so carefully, so this package tries to never overwrite files.

At stream creation, if the not-rotated log file already exists and its size exceeds the rotation size, an initial rotation attempt is done.

At each rotation attempt a check is done to verify that destination rotated file does not exists yet; if this is not the case a new destination rotated file name is generated and the same check is performed before going on. This is repeated until a not existing destination file name is found or the package is exhausted. For this reason the rotated file name generator function may be called several times for each rotation job.

Once an error event is emitted, nothing more can be done: the stream is closed as well.

Compatibility

This package is written following Node.js 4.0 specifications always taking care about backward compatibility. The package it tested under following versions:

6.0
5.0
4.2
4.1
4.0
0.12
0.11

Required: Node.js 0.11

Licence

MIT Licence

Bugs

Do not hesitate to report any bug or inconsistency @github.

ChangeLog

2016-12-05 - v1.1.1
- Dependencies update
2016-10-18 - v1.1.0
- Added classical UNIX logrotate tool behaviour.
- Dependencies update
2016-04-29 - v1.0.5
- Tested on node v6.0
- Fixed a bug on rotation with interval and compression
2015-11-09 - v1.0.4
- Tested on node v5.0
- Fixed bug on initial rotation with interval
2015-10-25 - v1.0.3
- Tested on node v4.2
- Dependencies update
2015-10-09 - v1.0.2
- README update
2015-10-08 - v1.0.1
- README fix
2015-10-08 - v1.0.0
- Async error reporting refactory
2015-10-07 - v0.1.0
- Internal gzip compression
2015-10-06 - v0.0.5
- External compression
2015-09-30 - v0.0.4
- Added path option
- Missing path creation
2015-09-29 - v0.0.3
- Rotation by interval
- Buffer optimization (thanks to allevo)
2015-09-17 - v0.0.2
- Rotation by size
2015-09-14 - v0.0.1
- README.md
2015-09-10 - v0.0.0
- Embryonal stage

dtryon / rotating-file-stream