mycaule / bash-scripts

Useful Bash scripts and one-liners

Bash scripts

Power tools to automate stuff at work as a backend engineer or as a data engineer.

From my experience working with AWS, Git, Slack, JIRA, Confluence.

Focuses on minimalism and productivity.

CLI Tools

• aws-cli
• awslogs
• aws-shell
• csvkit
• httpie
• franz
• postman
• httpie
• jq
• git-standup
• jira-cmd
• ncdu
• ncdu-s3
• puppeteer
• ngrok
• socat
• hub
• mssql-tools
• psql

Slack automation

• slack-cli

Git Functions

$./git-clean-secrets.sh

Time Functions

$./calendar.sh

Thu	Fri	Sat	Sun	Mon	Tue	Wed	Thu
 30	 31	 1	 2	 3	 4	 5	 6

$./timer-pv.sh 2

AWS Functions

$./aws-bucket-list.sh s3://ryft-public-sample-data/

$./aws-bucket-list-csv.sh s3://ryft-public-sample-data/

# Connect through local DB Client (Datagrip, DBeaver, Kopf, Cerebro)
# - <bastion-name>
# - <front-name>
$./aws-tunnel-db.sh "elastic"
$./aws-tunnel-db.sh "rds"

SSH tricks

Live tail multiple remote files

With multitail

multitail -l 'ssh <user>@<node1> tail -f /path/to/file.log"' -l 'ssh <user>@<node2> "tail -f /path/to/file.log"'

Difference of two remote files

diff <(ssh <user>@<node1> 'cat /path/to/file.conf') <(ssh <user>@<node2> 'cat /path/to/file.conf')

See: http://xmodulo.com/how-to-diff-remote-files-over-ssh.html

JIRA Functions

jira worklogadd XXX-72 1
jira comment XXX-72 "Hello this is done."

Crontab tricks

Trigger a check in Healthchecks.io.

$ crontab -l
PATH=/path/to/.nvm/versions/node/v8.0/bin
MAILTO="[e-mail]"
0 17 * * 1-5 jira worklogadd XXX-20 6 <subject> && curl -fsS --retry 3 https://hchk.io/<uuid> > /dev/null

Programming

...

Docker Setup

sudo curl -sSL https://get.docker.com/ | sh
sudo usermod -aG docker michel

# Delete all containers
docker rm $(docker ps -a -q)
# Delete all images
docker rmi $(docker images -q)

RocksDB ldb

docker pull arschles/rocksdb-ldb
docker run -v $PWD:/pwd -w /pwd arschles/rocksdb-ldb /rocksdb/tools/ldb --db=. scan

Monitoring

$ crontab -l
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/home/michel/.local/bin/:/home/michel/Scripts/monitoring

0 9,13,18 * * 1-5 healthchecks.sh
0 9,13 * * 1-5 wakeup.sh

References

• https://github.com/agarrharr/awesome-cli-apps
• https://github.com/alebcay/awesome-shell
• Common CRON mistakes
• Slack API - Attaching content and links to message
• Bash stuff : crontab guru, SO #869589
• SO #3004811, SE #230673
• Unit tests with bats