DevOps Ninja

This is a collection of some very useful command-line commands that eases the life of a DevOps Engineer.
Please Feel free to fork and/or PR if you have any additions.

Checking ports

Show port and PID - netstat -tulpn
Show process and listening port - ss -ltp
Show ports that are listening - ss -ltn
Show real time TCP and UDP ports - ss -stplu
Show all established connections lsof -i
Show listening connections - lsof -ni | grep LISTEN

Linux Commands

Copy the content of a folder to an existing folder - cp -a /source/. /dest/
Delete everything in a directory - rm /path/to/dir/*
Remove all sub-directories and files - rm -r /path/to/dir/*
Find and replace whole words in vim - :%s/\<word\>\C/newword/g
To remove a folder with all its contents (including all interior folders): rm -rf /path/to/directory
To remove all the contents of the folder (including all interior folders) but not the folder itself: rm -rf /path/to/directory/*
or, if you want to make sure that hidden files/directories are also removed: rm -rf /path/to/directory/{*,.*}
To remove all the "files" from inside a folder(not removing interior folders): rm -f /path/to/directory/{*,.*}
How to move all files and folders via mv command mv source/{,.}* /destination/folder/here

Openssl

verify if TLS 1.2 is supported openssl s_client -connect google.com:443 -tls1_2
Generate a new private key and Certificate Signing Request - openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key
Generate a self-signed certificate - openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt
Generate a certificate signing request (CSR) for an existing private key - openssl req -out CSR.csr -key privateKey.key -new
Generate a certificate signing request based on an existing certificate - openssl x509 -x509toreq -in certificate.crt -out CSR.csr -signkey privateKey.key
Remove a passphrase from a private key - openssl rsa -in privateKey.pem -out newPrivateKey.pem
Check a Certificate Signing Request (CSR) - openssl req -text -noout -verify -in CSR.csr
Check a private key - openssl rsa -in privateKey.key -check
Check a certificate - openssl x509 -in certificate.crt -text -noout
Check a PKCS#12 file (.pfx or .p12) - openssl pkcs12 -info -in keyStore.p12
Convert a DER file (.crt .cer .der) to PEM - openssl x509 -inform der -in certificate.cer -out certificate.pem
Convert a PEM file to DER - openssl x509 -outform der -in certificate.pem -out certificate.der
Convert a PKCS#12 file (.pfx .p12) containing a private key and certificates to PEM - openssl pkcs12 -in keyStore.pfx -out keyStore.pem -nodes
Convert a PEM certificate file and a private key to PKCS#12 (.pfx .p12) - openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt

Listing Running Services Under SystemD in Linux

systemctl list-units --type=service

Check a public IP

curl http://whatismyip.org/
curl ifconfig.me
curl icanhazip.com

Return the IP of an interface

ifconfig en0 | grep --word-regexp inet | awk '{print $2}'
ip add show eth0 | awk '/inet/ {print $2}' | cut -d/ -f1 | head -1
ip -br a sh eth0 | awk '{ print $3 }' (returns netmask)
ip route show dev eth0 | awk '{print $7}'
hostname -I (return ip only)

Replace all occurrences of string in a directory

Find and replace string - grep -rl "oldstring" ./ | xargs sed -i "" "s/oldstring/newstring/g"

Dig

Check domain with specific NS - dig <domain.com> @<ns-server>
Get NS records for a site - dig <domain.com> ns

Disk checks

Sort disk usage by most first - df -h | tail -n +2 | sort -rk5
Check the size of a top level dicectory - du -h --max-depth=1 /tmp/
Top 50 file sizes - du -ah / | sort -n -r | head -n 50
Show directory sizes (must not be in root directory) - du -sh *
Check disk usage per directory - du -h <dir> | grep '[0-9\.]\+G’
Look for growing directories - watch -n 10 df -ah
Ncurses based disk usage - ncdu -q
Colorized output of du - du -x --max-depth=1|sort -rn|awk -F / -v c=$COLUMNS 'NR==1{t=$1} NR>1{r=int($1/t*c+.5); b="\033[1;31m"; for (i=0; i<r; i++) b=b"#"; printf " %5.2f%% %s\033[0m %s\n", $1/t*100, b, $2}'|tac

Docker

Dokcer resources usage - docker info
know how much space is taken by a particular container docker container ls -s
Know how much spaces is used by Docker Root Dir du -h --max-depth=1 /var/lib/docker
Docker storage usage docker system df
Docker list volumes docker volume ls
Docker list images that are locally stored with the Docker Engine docker image ls
Docker inspect volumes docker volume inspect VOLUME NAME
Remove a group of images - docker images | grep "<none>" | awk '{print $3}' | xargs docker rmi
Remove all untagged containers - docker rm $(docker ps -aq --filter status=exited)
Remove all untagged images - docker rmi $(docker images -q --filter dangling=true)
Remove old (dangling) Docker volumes - docker volume rm $(docker volume ls -qf dangling=true)
Docker remove redundant objects at once docker system prune
Install on Ubuntu - curl -sSL https://get.docker.com/ubuntu/ | sudo sh
Get stats from all containers on a host - docker ps -q | xargs docker stats
Tail last 300 lines of logs for a container - docker logs --tail=300 -f <container_id>
Build an image from the Dockerfile in thecurrent directory and tag the image docker build -t myimage:1.0 .
Pull an image from a registry docker pull myimage:1.0
Retag a local image with a new image name and tag docker tag myimage:1.0 myrepo/myimage:2.0
Push an image to a registry docker push myrepo/myimage:2.0
Run a container from the Alpine version 3.9 image, name the running container “web” and expose port 5000 externally, mapped to port 80 inside the container docker container run --name web -p 5000:80 alpine:3.9
Stop a running container through SIGTERM docker container stop web
Stop a running container through SIGKILL docker container kill web
List the networks docker network ls

Shell Script to Install Docker on Ubuntu

#!/bin/bash
set -e
#Uninstall old versions
sudo apt-get remove docker docker-engine docker.io containerd runc
#Update the apt package index:
sudo apt-get update
#Install packages to allow apt to use a repository over HTTPS:
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
# Add docker's package signing key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# Add repository
sudo add-apt-repository -y \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
# Install latest stable docker stable version
sudo apt-get update
sudo apt-get -y install docker-ce
# Enable & start docker
sudo systemctl enable docker
sudo systemctl start docker
# add current user to the docker group to avoid using sudo when running docker
sudo usermod -a -G docker $USER
 # Output current version
docker -v

Shell Script to Install Docker on Centos

   #!/bin/bash
   #Get Docker Engine - Community for CentOS + docker compose
   set -e
   #Uninstall old versions
   sudo yum remove docker docker-common docker-selinux docker-engine-selinux docker-engine docker-ce
   #Update the packages:
   sudo yum update -y
    #Install needed packages
   sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Configure the docker-ce repo:
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# Install the latest docker-ce
sudo yum install docker-ce
# Enable & start docker
sudo systemctl enable docker.service
sudo systemctl start docker.service
# add current user to the docker group to avoid using sudo when running docker
sudo usermod -a -G docker $(whoami)
# Output current version
docker -v

Shell Script to Install Docker on AWS linux

    #!/bin/bash
    #Get Docker Engine - Community for CentOS + docker compose
    set -e
    #Uninstall old versions
    sudo yum remove docker docker-common docker-selinux docker-engine-selinux docker-engine docker-ce
    #Update the packages:
    sudo yum update -y
    #Install the most recent Docker Community Edition package.
    sudo amazon-linux-extras install docker -y
    # Enable & start docker
    sudo service docker start
    # add current user to the docker group to avoid using sudo when running docker
    #sudo usermod -a -G docker ec2-user
    sudo usermod -a -G docker $(whoami)
    # Output current version
     docker -v

Docker Compose

Shell Script to Install the latest version of docker-compose

#!/bin/bash
# get latest docker compose released tag
COMPOSE_VERSION=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep 'tag_name' | cut -d\" -f4)
sudo curl -L "https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod a+x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
# Output the  version
docker-compose -v

Dockerfile

Dockerizing a simple nodeJs app

FROM node:4.6
WORKDIR /app
ADD ./app
RUN npm install
EXPOSE 3000
CMD npm start

Find

Exlcude directories in find - find /tmp -not $ -path /tmp/dir -prune $ -type p -o -type b

Git

Remove deleted files from repo - git rm $(git ls-files --deleted)
Reset git repo (dangerous) - git reset --hard HEAD
Reset and remove untracked changes in repo - git clean -xdf
Ignore certificates when cloning via HTTPS - git config --global http.sslVerify false
Pull changes and remove stale branches - git pull --prune
Grab the diff of a previous version of a file - git diff HEAD@{1} ../../production.hosts
Grab the diff of a staged change - git diff --cached <file>
Undo a commit to a branch - git reset --soft HEAD~1
View files changed in a commit - git log --stat
Pull latest changes stashing changes first - git pull --autostash
Make an empty commit (good for CI) - git commit --allow-empty -m "Trigger notification"

Jenkins

Setup Jenkins on EC2

#!/bin/bash
sudo yum update -y
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key
sudo yum install java-1.8.0 -y
sudo yum install jenkins -y
sudo service jenkins start

sudo cat
/var/lib/jenkins/secrets/initialAdminPassword

Grep

Look through all files in current dir for word “foo” - grep -R "foo” .
View last ten lines of output - grep -i -C 10 "invalid view source” /var/log/info.log
Display line number of message - grep -n “pattern” <file>

Iptables

Check nat rules for ip redirection - iptables -nvL -t nat

Apache

Turn off “ServerSignature” and “ServerToken” on Apache

Edit httpd.conf or apache2.conf

// Kali, Debian, Ubuntu  Linux Mint
  sudo vi /etc/apache2/apache2.conf
// CentOS, Fedora, RHEL , Arch Linux
 sudo vi /etc/httpd/conf/httpd.conf

Add below settings to the file

   ServerSignature Off
   ServerTokens Prod

Restart Apache

// Kali, Debian, Ubuntu Linux Mint
   sudo service apache2 restart
//Fedora, CentOS/RHEL 7,Arch Linux
 systemctl restart httpd.service

Nginx

Check installed modules - nginx -V
Pretty print installed modules - 2>&1 nginx -V | xargs -n1
Test a configuration without reloading - nginx -t
Stop all nginx processes - nginx -s stop
Start all nginx processes - nginx -s start
Restart all nginx processes - nginx -s restart
Realod nginx configuration (without restarting) - nginx -s reload

Tomcat

Hide tomcat stack traces (showReport) as well as the Server info: Add below lines at the HOST section

<Valve className="org.apache.catalina.valves.ErrorReportValve"
showReport="false"
showServerInfo="false" />

Injecting HTTP Response with the secure header in Tomcat

<filter>
    <filter-name>httpHeaderSecurity</filter-name>
    <filter-class>org.apache.catalina.filters.HttpHeaderSecurityFilter</filter-class>
    <async-supported>true</async-supported>
    <init-param>
      <param-name>antiClickJackingEnabled</param-name>
      <param-value>true</param-value>
    </init-param>
    <init-param>
      <param-name>antiClickJackingOption</param-name>
      <param-value>DENY</param-value>
    </init-param>
    </filter>
    <filter-mapping>
    <filter-name>httpHeaderSecurity</filter-name>
    <url-pattern>/*</url-pattern>
    </filter-mapping>
    <welcome-file-list>
  <welcome-file>index.html</welcome-file>
    </welcome-file-list>

Nmap

Check single port on single host - nmap -p <port> <host/IP>
Intrusive port scan on a single host - nmap -sS <host/IP>
Top ten port on a single host - nmap --top-ports 10 <host/IP>

Password generation

Create hash from password - openssl passwd -crypt <password>
Generate random 8 character password (Ubuntu) - makepasswd -count 1 -minchars 8
Create .passwd file with user and random password - sudo htpasswd -c /etc/nginx/.htpasswd <user>

Removing files

Remove files over 30 days old - find . -mtime +30 | xargs rm -rf
Remove files older than 7 day starting with 'backup' - find . -type f -name "backup*" -mtime +7 -exec rm {} \;

SSH

Generate generic ssh key pair - ssh-keygen -q -t rsa -f ~/.ssh/<name> -N '' -C <name>
AWS PEM key to ssh PUB key : ssh-keygen -y -f eliarms.pem > eliarms.pub

Tail log with colored output

grc tail -f /var/log/filename

Tmux

Kill a window - tmux kill-window -t 0
Kill stuck tmux window - tmux kill-window -t X
Create a new session - tmux new -s <name>
List all sessions - tmux ls

Show process tree of all PIDs - ps auxwf
Show all process info and hierarchy (same as above)- ps -efH
Show orphaned processes for - ps -ef|awk '$3=="1" && /pandora/ { print $2 }'
Show all orphaned processes (could be daemons) - ps -elf | awk '{if ($5 == 1){print $4" "$5" "$15}}'
Show zombie processes - ps aux | grep Z

HSTS

Enable HTTP Strict Transport Security protocol in Tomcat To enable HSTS in Tomcat, follow these steps:

Open the /conf/web.xml file in a text editor.

Uncomment the httpHeaderSecurity filter definition and the section, and then add the hstsMaxAgeSeconds parameter, as shown below.

<filter>
    <filter-name>httpHeaderSecurity</filter-name>
<filter-class>org.apache.catalina.filters.HttpHeaderSecurityFilter</filter-class>
<init-param>
  <param-name>hstsMaxAgeSeconds</param-name>
  <param-value>31536000</param-value>
    </init-param>
    <async-supported>true</async-supported>
    </filter>
    <filter-mapping>
    <filter-name>httpHeaderSecurity</filter-name>
    <url-pattern>/*</url-pattern>
    <dispatcher>REQUEST</dispatcher>
    </filter-mapping>

Save the file
Restart Tomcat

Enable HTTP Strict Transport Security protocol in Apache To enable HSTS in Tomcat, follow these steps:
- Open the /conf/httpd.conf file in a text editor.
- Uncomment the header module: LoadModule headers_module modules/mod_headers.so
- Add a header setting in the VirtualHost section:
```
<VirtualHost www.example.com:80>
Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
</VirtualHost>
```

Enable HTTP Strict Transport Security protocol in IIS To enable HSTS in IIS, do the following:

Add a Strict-Transport-Security header to the web.config file under the IIS installation root directory:

    <system.webServer>
    <httpProtocol>
    <customHeaders>
     <add name="Strict-Transport-Security" value="max-age=31536000"/>
     </customHeaders>
    </httpProtocol>
    </system.webServer>

Restart IIS

Permissions
- Set permissions recursively on a dir (with ACL enabled)
```
setfacl -R -m u:userid:rwx foldername
```

JAVA

Completly remove Openjdk from redhat

 rpm -qa | grep openjdk | xargs  yum -y remove

masipmarch / devops-ninja

DevOps Ninja

About