forkkit / script

Making it easy to write shell-like scripts in Go

import "github.com/bitfield/script"

What is script?

script is a Go library for doing the kind of tasks that shell scripts are good at: reading files, executing subprocesses, counting lines, matching strings, and so on.

Why shouldn't it be as easy to write system administration programs in Go as it is in a typical shell? script aims to make it just that easy.

Shell scripts often compose a sequence of operations on a stream of data (a pipeline). This is how script works, too.

Read more: Scripting with Go

Quick start: Unix equivalents

If you're already familiar with shell scripting and the Unix toolset, here is a rough guide to the equivalent script operation for each listed Unix command.

Unix / shell	script equivalent
(any program name)	Exec()
[ -f FILE ]	IfExists()
>	WriteFile()
>>	AppendFile()
$*	Args()
basename	Basename()
cat	File() / Concat()
cut	Column()
dirname	Dirname()
echo	Echo()
grep	Match() / MatchRegexp()
grep -v	Reject() / RejectRegexp()
head	First()
find -type f	FindFiles
ls	ListFiles()
sed	Replace() / ReplaceRegexp()
sha256sum	SHA256Sum() / SHA256Sums()
tail	Last()
uniq -c	Freq()
wc -l	CountLines()
xargs	ExecForEach()

Some examples

Let's see some simple examples. Suppose you want to read the contents of a file as a string:

contents, err := script.File("test.txt").String()

That looks straightforward enough, but suppose you now want to count the lines in that file.

numLines, err := script.File("test.txt").CountLines()

For something a bit more challenging, let's try counting the number of lines in the file that match the string "Error":

numErrors, err := script.File("test.txt").Match("Error").CountLines()

But what if, instead of reading a specific file, we want to simply pipe input into this program, and have it output only matching lines (like grep)?

script.Stdin().Match("Error").Stdout()

That was almost too easy! So let's pass in a list of files on the command line, and have our program read them all in sequence and output the matching lines:

script.Args().Concat().Match("Error").Stdout()

Maybe we're only interested in the first 10 matches. No problem:

script.Args().Concat().Match("Error").First(10).Stdout()

What's that? You want to append that output to a file instead of printing it to the terminal? You've got some attitude, mister.

script.Args().Concat().Match("Error").First(10).AppendFile("/var/log/errors.txt")

Real use cases

Let's use script to write a program that system administrators might actually need. One thing I often find myself doing is counting the most frequent visitors to a website over a given period of time. Given an Apache log in the Common Log Format like this:

212.205.21.11 - - [30/Jun/2019:17:06:15 +0000] "GET / HTTP/1.1" 200 2028 "https://example.com/ "Mozilla/5.0 (Linux; Android 8.0.0; FIG-LX1 Build/HUAWEIFIG-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.156 Mobile Safari/537.36"

we would like to extract the visitor's IP address (the first column in the logfile), and count the number of times this IP address occurs in the file. Finally, we might like to list the top 10 visitors by frequency. In a shell script we might do something like:

cut -d' ' -f 1 access.log |sort |uniq -c |sort -rn |head

There's a lot going on there, and it's pleasing to find that the equivalent script program is quite brief:

package main

import (
	"github.com/bitfield/script"
)

func main() {
	script.Stdin().Column(1).Freq().First(10).Stdout()
}

Let's try it out with some sample data:

cd examples/visitors
go run main.go <access.log

16 176.182.2.191
 7 212.205.21.11
 1 190.253.121.1
 1 90.53.111.17

Documentation

See pkg.go.dev for the full documentation, or read on for a summary.

Sources

These are functions that create a pipe with a given contents:

Source	Contents
Args	command-line arguments
Echo	a string
Exec	command output
File	file contents
FindFiles	recursive file listing
IfExists	do something only if some file exists
ListFiles	file listing (including wildcards)
Slice	slice elements, one per line
Stdin	standard input

Filters

Filters are methods on an existing pipe that also return a pipe, allowing you to chain filters indefinitely. The filters modify each line of their input according to the following rules:

Filter	Results
Basename	removes leading path components from each line, leaving only the filename
Column	Nth column of input
Concat	contents of multiple files
Dirname	removes filename from each line, leaving only leading path components
EachLine	user-supplied function
Echo	all input replaced by given string
Exec	filtered through external command
ExecForEach	execute given command template for each line of input
First	first N lines
Freq	frequency count of unique input lines, most frequent first
Join	replace all newlines with spaces
Last	last N lines
Match	matching lines
MatchRegexp	matching lines
Reject	non-matching lines
RejectRegexp	non-matching lines
Replace	matching lines replaced with string
ReplaceRegexp	matching lines replaced with string
SHA256Sums	SHA-256 hashes of each listed file

Sinks

Sinks are methods that return some data from a pipe, ending the pipeline and extracting its full contents in a specified way:

Sink	Destination	Results
AppendFile	appended to file, creating if it exists	bytes written, error
Bytes		data as []byte, error
CountLines		number of lines, error
Read	given []byte	bytes read, error
SHA256Sum		SHA-256 hash, error
Slice		data as []string, error
Stdout	standard output	bytes written, error
String		data as string, error
WriteFile	specified file, truncating if it exists	bytes written, error

Examples

Since script is designed to help you write system administration programs, a few simple examples of such programs are included in the examples directory.

Contributing

See the contributor's guide for some helpful tips if you'd like to contribute to the script project.

Links

• Scripting with Go
• Code Club: Script
• Bitfield Consulting
• Go books by John Arundel

Gopher image by MariaLetta