load-balance-lines

Parallelize newline-delimited data processing by load balancing lines between multiple processes

Summary

Install
Basic use
Simple demo
Real case demo
Options
- Number of processes
- Silent

Install

# Make the executable accessible within your project npm scripts as load-balance-lines
# or, out of npm scripts, as ./node_modules/.bin/load-balance-lines
npm i load-balance-lines
# or globally
npm i -g load-balance-lines

Basic use

Take a huge pile of data with atomic data elements separated by newline breaks, typically NDJSON.

# Make sure your executable is... executable
chmod +x /path/to/my/executable
# and let's go!
cat data.ndjson | load-balance-lines /path/to/my/executable some args

or without the cat command, using <

load-balance-lines /path/to/my/executable some args for the executable < data.ndjson

Simple demo

see test

Real case demo

For the needs of wikidata-rank, we need to parse a full dump of Wikidata

get the latest dump (currently 31G gzipped)

wget -c https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz

Use nice to use the maximum amount of CPU possible while letting the priority to other processes
Use pigz to decompress it using threads (drop-in replacement to the single threaded gzip)

nice pigz -d < latest-all.json.gz | nice load-balance-lines /path/to/wikidata-rank/scripts/calculate_base_scores

Options

Number of processes

By default, there will be as many processes as CPU cores, but it can be modified by setting an environment variable

export LBL_PROCESSES=4 ; cat data.ndjson | load-balance-lines ./my/script

Verbose

By default, the load balancer is silent to let stdout free for sub-processes outputs, but you can get some basic informations by setting LBL_VERBOSE

export LBL_VERBOSE=true ; cat data.ndjson | load-balance-lines ./my/script

About