Patman

Overview

Patman is a tiny command-line tool designed for processing and manipulating raw text data. It excels in tasks like log filtering and aggregation, offering a range of operators for various annoying text operations. Its reason for existence is in all those cases where grep and sed are not enough, but a dedicated script language is overkill or too slow.

Have you ever tried to parse GBs of logs while debugging a production incident?

Installation

Currently the best way to install Patman is to use it as a library. You will need to have Go installed on your system.

go get github.com/lucagez/patman

Then create a main.go file with the following contents:

package main

import "github.com/lucagez/patman"

func main() {
	patman.Run()
}

Extending Patman natively

Patman can be extended with custom operators by implementing the Operator interface. The following example shows how to implement a uppercase operator.

package main

import (
	"strings"

	"github.com/lucagez/patman"
)

func Upper(line, arg string) string {
	return strings.ToUpper(line)
}

func main() {
	patman.Register("upper", patman.OperatorEntry{
		Operator: Upper,
		Usage:    "converts line to uppercase",
		Example:  "echo 'hello' | patman 'upper(/)' # HELLO",
	})
	patman.Run()
}

The operator can then be used as follows:

echo hello | patman 'upper(/)' # HELLO

The new operator will then be available also in the patman command help message.

Usage

The basic structure of a Patman command is:

patman [options] | '[operator1] |> [operator2] |> ...'

The patman command takes in a list of operators and applies them to the input data. The |> symbol is used to pipe the output of one operator to the next. The patman command can be used in a standard unix pipeline with other commands.

Examples

Let's use as an example a log file containing the following lines:

2018-01-01 00:00:00 ERROR: Something went wrong.
2018-01-01 00:00:00 INFO: Something went right.
2018-01-01 00:00:00 ERROR: Something went wrong again.

Match all lines containing the word "ERROR" and replace it with "WARNING":

cat logs.txt | patman 'matchline(ERROR) |> replace(WARNING)'

Match all error lines and output a csv a timestamp and message columns:

cat logs.txt | patman 'matchline(ERROR)' | patman -format csv \
    'split(,/1) |> name(timestamp)' \
    'split(: /1) |> name(message)'

This will output a csv file with the following contents:

timestamp,message
2018-01-01 00:00:00,Something went wrong.
2018-01-01 00:00:00,Something went wrong again.

Initialization Options

-help, -h: Show help message.
-file: Specify the input file (default: stdin).
-index: Define the index property for log aggregation.
-format: Set the output format (default: stdout). One of stdout, csv, json or a custom formatted string.
-mem: Buffer size in MB for parsing larger file chunks.
-delimiter: Custom delimiter for splitting input lines.
-join: Custom delimiter for joining output (default: \n).
-buffer: Size of the stdout buffer when flushing (default: 1).

Operators and Aliases

Patman includes a variety of operators for text manipulation:

name/n

Assigns a name to the output of an operator, useful for log aggregation and naming columns in csv or json formats. Usage:

echo something | patman 'name(output_name)'

match/m

Matches the first instance of a regex expression. Usage:

echo something | patman 'match(expression)'

matchall/ma

Matches all instances of a regex expression. Usage:

echo something | patman 'matchall(expression)'

replace/r

Replaces text matching a regex expression with a specified string. Usage:

echo something | patman 'replace(expression/replacement)'

named_replace/nr

Performs regex replacement using named capture groups. Usage:

echo something | patman 'named_replace(expression/replacement)'

matchline/ml

Matches entire lines that satisfy a regex expression. Usage:

echo something | patman 'matchline(expression)'

notmatchline/nml

Returns lines that do not match a regex expression. Usage:

echo something | patman 'notmatchline(expression)'

split/s

Splits a line by a specified delimiter and selects a part based on index. Usage:

echo something | patman 'split(delimiter/index)'

filter/f

Filters lines containing a specified substring. Usage:

echo something | patman 'filter(substring)'

js

Executes a JavaScript expression, passing x as the argument. Usage:

echo something | patman 'js(expression)'

# e.g. uppercase
echo something | patman 'js(x.toUpperCase())' # SOMETHING

explode

Splits a line by a specified delimiter and joins resulting parts with a newline character. Usage:

echo something | patman 'explode(delimiter)'

# e.g. split by any char
echo something | patman 'explode(\.*/)'
# s
# o
# m
# e
# t
# h
# i
# n
# g

License

The MIT License (MIT)

lucagez / patman