qpp-shared-logger-node

A simple configurable wrapper around the winston application logger and the morgan http access logger.

This logger is opinionated. Configure it with a minimal set of options, and get Splunk-searchable, PII-scrubbed, and QPP-compliant logs with minimal effort.

Installation

$ npm install qpp-shared-logger-node --save

Usage

This module accepts configuration, sets up the logger, then yields a winston Logger object for you to use.

It is recommended that you set up your project with a log.js module that you can import from the rest of your project; log.js is where you will configure the shared logger and expose the winston Logger.

First, pass your configuration to the module, then get access to the Logger object.

log.js:

const sharedLogger = require('qpp-shared-logger-node');

sharedLogger.configure({
    projectSlug: 'my-app-name'
});

module.exports = sharedLogger.logger; // a winston Logger

In other parts of your code:

const logger = require('log');

logger.info('fetching data', { id: object.id, sort: 'asc' });

Logging with context fields

The contextLogger function returns a logger object that includes a set of fields with each log message. The fields are merged with those passed to a logging method.

For example, this can be used to log a request id in all entries related to a single client request:

req.logger = sharedLogger.contextLogger({ requestId: uuidv1(), url: req.url });
req.logger.info('started');
req.logger.error(message, { error: err });

Field values in the log call take priority over values in the context fields.

Context Logging Defaults

There are defaults that should be added to the context of the beginning with each request if possible.

Attribute Name	Attribute Value	Description
requestId	UUID	This is a unique ID that will allow for tracking events throughout the livetime of the request. This can be populated either through uuid or express request id or some other preferred UUID generator method.
oktaId	string	This value should be pulled from the auth service if possible
applicationBuild or gitSHA	string	This should be the build id or the git SHA of the current running application

const requestContext = {
  requestId: uuidv1(),
  oktaId: AuthService(), // The auth service should return the okta id.
  applicationBuild: process.env.BUILD_ID // what constant the application would be storing this information
}

req.logger = sharedLogger.contextLogger(requestContext);

Configuration

Key	Description	Allowed Values
projectSlug (Required)	The name of your app, to be included in the log metadata and the log file path.	any string

Environment Defaults

This module will read process.env.NODE_ENV to determine your application's "environment", and configure these settings out-of-the-box:

Environment	Application log path	Log level
development	console	debug
test	`./app.log`	silly
production	`/var/log/qpp-service/{{ projectSlug }}/app.log`	info

Environment	Access log path	Format
development	console	dev
test	`./access.log`	combined
production	/var/log/qpp-service/{{ projectSlug }}/access.log	combined

Advanced Application Log Configuration

You may override the defaults:

Key	Description	Allowed Values	Default
addlFormats	This key accepts an array of Winston compatible logform formats that are then added to the logger. This allows you to provide formats for unique scenarios that are not covered by the default formats.	An array of logform formats.	[]
format	Format logs should be written in. Default is `json` format. More on winston fromats can be found https://www.npmjs.com/package/winston#formats.	`json`, `simple`, `prettyPrint`, `logstash`	`json`
environment	Override the "node environment" that your app is running in. Using the conventional values will automatically configure various log settings for you. Conventional values are `development`, `test`, and `production`. Deployed code should run with `NODE_ENV=production`	`process.env.NODE_ENV`
logDirectory	A valid directory where the log file should be written, or "console" to write to stdout. Will `throw` if the directory does not exist or is not writable.	`console`, or an absolute dir	built from environment
logFilenamePrefix	The application log filename will be built from this prefix, by default it will be `app.YYYYMMDD.log`	any string	`app`
datePattern	A string representing the moment.js date format to be used for rotating. The meta characters used in this string will dictate the frequency of the file rotation. For example, if your datePattern is simply 'HH' you will end up with 24 log files that are picked up and appended to every day. (default: 'YYYYMMDD')	any string containing a standard date pattern	`YYYYMMDD`
logFileExtension	The log file extension to use.	string	`log`
logLevel	All log messages at this level or higher will be logged. `none` effectively turns off logging.	`none`, `error`, `warn`, `info`, `verbose`, `debug`, `silly`	chosen based on environment
logTimestamps	Add timestamps to log entries	`true` or `false`	`true`
logColorize	If `true`, log messages will be sent colorized (most valuable when logging to the `console`)	`true` or `false`	`false`
redactKeys	An array of keys to scrub from the log metadata	an array of lowercase strings	`['email', 'firstname', 'lastname', 'password', 'ptan', 'tin', 'userid', 'username']`
redactRegexes	An array of regular expressions representing string values to scrub	an array of regular expressions (string or `RegExp`)	`[]`
maxDays	The maximum number of days to keep logs for.	A number, in days	0 (No deletion)
rotationMaxsize	The max size the log file should reach before it is rotated.	a size, in bytes. For example, 1M = 1000000. Or 'none' to never rotate logs	50000000 (50M)
splunkSettings	Adding the Splunk configuration settings will add Splunk http transport via the winston-splunk-httplogger package	`object`	`undefined`

Splunk Transport Configuration

Key	Description	Allowed Values	Default
url	URL string to pass to `url.parse`. This will try to set `host`, `path`, `protocol`, `port`, `url`. Any of these values will be overwritten if the corresponding property is set on `config`	`http://localhost:8888`	`undefined`
token	The Splunk HTTP Event Collector token

WARNING - If the Splunk transport fails to connect to Splunk, log messages will be lost during the outage. DO NOT rely on this logger alone if you need to have guaranteed delivery of all logs to Splunk.

Advanced HTTP Access Log Configuration

Override defaults.

Key	Description	Allowed Values	Default
accessLog.logDirectory	A valid directory where the log file should be written, or "console" to write to stdout. Will `throw` if the directory does not exist or is not writable.	`console`, or an absolute dir	built from environment
accessLog.logFilenamePrefix	The access log filename will be built from this prefix, by default it will be `access.YYYYMMDD.log`	any string	`access`
accessLog.format	The log format. Note that `combined` and `common` are well-known as NCSA log formats.	`combined`, `common`, `dev`, `short`, `tiny`, `none`	chosen based on environment
accessLog.rotationMaxsize	The max size the log file should reach before it is rotated.	a size, in bytes. For example, 1M = 1000000. Or 'none' to never rotate logs	50000000 (50M)
accessLog.maxFiles	The maximum number of rotated logs to keep around. Logs rotated after this will be removed.	a count, in number of files	None (No deletion)

Considerations for production logging

Splunk ingesters are set up to find log files under /var/log, so always send your production logs here or to a subdirectory within.
To differentiate logs emitted from the DEV or IMP or PROD deployed environments, be sure the name is embedded in one or more of the Splunk-searchable fields. For example, configure your log directory with the "deployed environment" embedded as a subdir: /var/log/qpp-service/{{ dev|imp|prod }}/{{ projectSlug }}/

So on Halloween, Splunk will index the source of log messages as /var/log/qpp-service/imp/projectSlug/app.20171031.log. A sample Splunk query to pick up your IMP logs: source=/var/log/qpp-service/imp/projectSlug/*.
Suggestion: use an environment var (not NODE_ENV) to communicate the name of the environment from the OS to the app. The auth service, for example, uses the ENVIRONMENT var.

PII (Personally-Identifiable Information) and log scrubbing

Log messages are composed of a message (String) and metadata (key-value). Keys in the metadata that match one of the redactKeys will not have their value logged. Note that the logger does not search the String message for data to scrub. Please keep log messages simple, and add metadata with proper keys.

Compatibility

Tested with node v10.15.3 and node v12.15.0

Development

$ brew install gitleaks # Installs gitleaks for pre-commit secret detection. If not running on Mac, see https://github.com/zricethezav/gitleaks/releases 
$ npm install       # install dependencies
$ npm  test         # run tests, also report coverage in ./coverage/index.html
$ npm run  format   # run prettier to format code
$ npm run lint      # run eslint to check code

Publishing

The qpp-shared-logger-node npm package can be published to npmjs.com via GitHub workflow. Make sure that an NPM_TOKEN is configured in the repository's Secrets setttings. The token must belong to a member of the npm package's maintainers team. To publish, make sure the version number in package.json is updated, then create a Release in GitHub with the "pre-release" checkbox checked. Publishing should happen automatically after that.

If the GitHub Workflow is not working, a member of the npm package's maintainers team can publish via command line on a local computer.

npm login
git checkout master
git pull
npm publish

Want to Contribute?

Want to file a bug or contribute some code? Read up on our guidelines for contributing.

Public Domain

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

See the formal LICENSE file.

KyleApfel / qpp-shared-logger-node