Giters

henryvps / my-links

A collection of OSS tools and handy links

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

My Links

IT News, Resources, and Community Sites

Table of Contents

IT News

Linux News and Articles

Message Boards and Forums

Asset Management

  • Clusto - Inventory tracking and "cluster management" system
  • Collins - Infrastructure source of truth, created by Tumblr
  • Fusion Inventory - Mulit-lingual
  • GestioIP - IP address management (IPAM), web based, supports discovery
  • GLPI - Also provides license management, software auditing, and ticketing
  • HostDB - New tool to help manage data center inventory
  • IP Plan - IPAM written in PHP
  • IT ITems DataBase (ITDB) - Includes purchase order management, floor plans, and ISO20000-like features
  • Network UPS Tools - Manage power devices from over 100 manufacturers using a single web interface
  • OCS Inventory NG (OCS-NG) - Automated inventory, deployment system, can sync with GLPI
  • openDCIM - Data center infrastructure management
  • phpIPAM - Written in perl. No, Ruby. No, Perl. Probably PHP
  • RackMonkey - No longer under development as of 2013
  • RackTables - Data center asset management, highly recommended
  • racktables-contribs - RackTables user-contributed plugins
  • Ralph - DCIM and CMDB, supports auto-discovery
  • rho - Scans for and gathers information about Linux and Unix servers
  • Snipe IT - Uses Bootstrap, web based, supports mobile
  • Tracmor - Web-based inventory and asset tracking system written in PHP 5 utilizing the Qcodo Framework

Backups

  • Amanda - The classic
  • Attic - Written in Python
  • Backupninja - Centralize way to configure and schedule many different backup utilities
  • BackUp and Restore Program (BURP) - Reduces network traffic and the amount of space required by using librsync
  • BackupPC - Dedupe and a web GUI for restores
  • Bacula - Open source backup tool, lots of downloads so I guess it's good I don't know
  • bup - Uses the git packfile format, supports global dedupe, can use "par2" redundancy
  • Duplicati - Supports encryption and dedupe
  • Duplicity - Encrypted bandwidth-efficient backup using the rsync algorithm
  • FSArchiver - Save the contents of a file system to a compressed archive, if one of the checksums doesn't match the file is lost, not the whole backup
  • lsyncd - Watches a local directory trees event monitor interface (inotify or fsevents) and kicks off an rsync when things change
  • Mondo Rescue - Disaster recovery, supports tapes, disks, network and CD/DVD as backup media, multiple filesystems, LVM, software and hardware RAID
  • Obnam - Snapshots, dedupe, encryption
  • rdiff-backup - Combines the best features of a mirror and an incremental backup in a bandwidth efficient manner
  • Redo Backup and Recovery - Simple bare metal backup and restore
  • Relax and Recover (REAR) - BMR, simple
  • rsnapshot - Uses rsync and hard links, can keep multiple full backups available while using very little disk space
  • SafeKeep - Superb project, uses LVM snapshots
  • SnapRAID - Backup program that also stores RAID parity information

Bootable USB Tools

Cluster Management and Schedulers

  • Aurora - Mesos scheduler for "always on" jobs, similar to Marathon
  • Bamboo - HAProxy auto configuration and auto service discovery for Marathon
  • Chronos - Distributed, fault tolerant cron scheduler that runs on top of Apache Mesos
  • crypt - Store and retrieve encrypted configs from etcd or consul
  • consul - Service discovery and configuration via DNS or HTTP, great for auto-scaling
  • Curator - ZooKeeper client wrapper and rich ZooKeeper framework, by Netflix
  • etcd - Highly-available key value store for shared configuration and service discovery
  • eureka - AWS Service registry for resilient mid-tier load balancing and failover, by Netflix
  • kubernetes - Container cluster manager
  • Marathon - Mesos scheduler for "always on" jobs (eg: web services)
  • Mesos - Cluster manager and scheduler, similar to Google's Borg and Omega projects
  • Mesosphere - Apache Mesos schedulers for almost anything
  • Myriad - YARN framework/scheduler
  • Serf - Decentralized solution for service discovery and orchestration that is lightweight, highly available, and fault tolerant
  • SmartStack - Automated service discovery and registration framework, includes Nerve and Synapse
  • vulcand - HTTP proxy that uses etcd as a configuration backend
  • Zookeeper - Distributed configuration service, synchronization service, and naming registry

Configuration Management and Orchestration

  • Ansible - CM and orchestration, also can do provisioning
  • Ansible Galaxy - Community site for finding, reusing, and sharing Ansible content
  • Awesome Chef - A curated list of amazingly awesome Chef resources
  • BOSH - Cloud orchestration, supports Docker containers too
  • Capistrano - Run scripts/commands and push software using SSH, a RubyGem, uses a Rake DSL
  • Chef - Well known configuration management tool
  • chef-book - Open source book about Chef
  • chef-handlers - Log failures to IRC and send metrics to Graphite (including deploys)
  • ChefScripts - Quickly parse the Chef run log to get the resources updated recently, in chronological order
  • Chef Development Kit (ChefDK) - Includes Berkshelf, Test Kitchen, ChefSpec, Foodcritic, Knife, Chef Zero, Ohai, and more
  • Chef is Easy - Chef tutorial
  • Chef Solo - Allows using cookbooks on nodes without access to a Chef server, designed for testing
  • Chef Supermarket - Massive collection of cookbooks
  • Chef Wiki - The official Opscode wiki
  • Cooking Infrastructure by Chef - Free Chef ebook
  • ElasticBox - Create your own catalog of applications and infrastructure components then build, deploy, and go
  • Fabric - Python library, uses SSH
  • Getting Started with Chef - Chef tutorial
  • grocery-delivery - Utility for managing cookbook uploads to distributed Chef backends
  • Juju - Ubuntu focused but works on RHEL / CentOS now
  • knife-backup - Knife plugin to help backup and restore a Chef server
  • Knife Community Plugins -
  • Learn Chef - Chef tutorial by Opscode
  • mcollective - Orchestration, can be used independently of Puppet
  • motherbrain - Use knife to spin up one node? Nah, use this to create a whole environment
  • omnibus - Create full-stack installers for your project, Chef
  • Open Linux Management Infrastructure (OpenLMI) - Manage, monitor, and configure servers via API calls (instead of SSH), included in RHEL 7
  • Opscode - Chef / Opscode on Github, tons of great tools
  • Puppet - Well known CM tool
  • puppet-lint - Check that your Puppet manifest conform to the style guide
  • Puppet Forge - Community repository of Puppet modules
  • puppet-module-skeleton - Skeleton modules for Puppet
  • Really Awesome New Cisco config Differ (RANCID) - Designed to monitor network hardware
  • Rudder - Automation and compliance, uses CFengine
  • Salt - Cloud orchestration, server provisioning, and configuration management
  • ServerSpec - Write RSpec tests for checking that your servers are configured correctly
  • spiceweasel - Quickly automates the creation of whole Chef environments (batch loading)
  • Supermarket - Community cookbook repository for Chef, can be hosted internally
  • Using Chef - Chef weekly newsletter
  • VirtKick - Simple panel for orchestrating VMs and containers
  • zero-user - Zero hassle Linux user management, requires Ansible

Containers

  • cadvisor - Analyzes resource usage and performance characteristics of running containers
  • Control Center - App management and orchestration, can manage Docker applications
  • crane - Docker orchestration
  • Docker - Designed to applications in a container, but not a whole OS instance
  • Docker-Registry - Source code to Docker Registry, allows you to create private registries
  • Docker Nano - The smallest possible Docker images
  • dockerfiles - Great little collection of pre-made Docker files
  • Docker Hub Registry - Official Docker images for many projects
  • Dokku - Docker powered mini-Heroku in around 100 lines of Bash
  • Dokku Alternative - Dokku fork, supports plugins
  • Ferry - Big data development environment on your local machine using Docker
  • fig - Fast, isolated development environments using Docker
  • Flocker - Docker container and data management
  • geard - CLI tool for installing Docker images as containers onto a systemd-enabled Linux system
  • libcontainer - Used by Docker 1.0+ so it doesn't have to use LXC/libvirt/systemd-nspawn
  • libswarm - Toolkit to compose network services, by Docker
  • Linux Containers (LXC) - Runs an OS instance (sshd, cron, daemons, init, etc) within a container
  • lmctfy - Open source version of Google’s container stack, which provides Linux application containers
  • Longshoreman - Heroku-like CLI tool for automated Docker deployments
  • nsenter - Attach to a Docker container
  • Panamax - Containerized app creator with an open-source app marketplace hosted in GitHub
  • pipework - SDN for Linux Containers
  • ShutIt - Manage your build process in a way that is both structured and flexible
  • Weave - Virtual network that connects Docker containers deployed across multiple hosts

File Sync and Sharing

  • BitTorrent Sync (btsync) - Sync on the local network or over the internet -- closed source
  • CloudBerry Explorer - Windows client for accessing AWS S3 buckets -- freeware, closed source
  • Cloud Explorer - Access storage on multiple clouds, Windows -- freeware, closed source
  • ownCloud - The most popular file sync and share program
  • Pulse - Uses an ID rather than an IP address, share your ID with friends and go. Formerly Syncthing
  • Pydio - Formerly AjaXplorer, AGPL license
  • Seafile - Also offers a paid professional edition with more features
  • SparkleShare - Uses git under the hood
  • Syncany - Supports tons of different protocols for the transfer
  • Unison - Multiplatform, feature-rich file sync

Git Tutorials

High Availability Clustering Tools

High Performance Computing Tools

HTTP Load Generation Tools

  • Gatling - HTTP, JMS, and WebSocket load generator
  • FunkLoad - Another load generator
  • httperf - HTTP load generator, originally created by HP, now open source
  • httping - Simple program that "pings" a URL and shows response time
  • JChav - Visualize JMeter output
  • JMeter - Supports many different services and protocols, written in Java
  • Locust - Written in Python, allows you to define user behavior
  • Siege - CLI based, supports basic authentication, cookies, and HTTPS
  • Tsung - Used to stress HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP and Jabber/XMPP servers
  • Web Performance Today - News site dedicated to web / HTTP performance
  • wrk - Multi-threaded CLI tool

IO Load Generation Tools

Logging and Log Management

  • Adiscon LogAnalyzer - Slick web interface for syslog messages
  • Bigdesk - Live charts and statistics for elasticsearch cluster
  • ElasticDump - Import / export tools for Elasticsearch
  • Elasticsearch - Component of the ELK stack, used for super fast searches
  • elasticsearch-river-email - E-mail river plugin for Elasticsearch
  • Elasticsearch Copier (escp) - Tool to copy elasticsearch indexes, great for copying from prod to dev
  • Elasticsearch Exporter - Small script to export data from one Elasticsearch cluster into another
  • FileSystem River (fsriver) - Elasticsearch river plugin helps to index documents from your local file system and using SSH
  • Fluentd - Unified logging layer
  • Flume - Distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data
  • GoAccess - Real-time web log analyzer and interactive viewer that runs in a terminal
  • Graylog2 - Log capture and analysis
  • InfluxDB - Distributed time series database with no external dependencies
  • kairosdb - Time series DB written on top of Cassandra
  • Kibana - Visualization component of the ELK stack
  • Logplex - Distributed syslog log router, uses Redis
  • logstalgia - Web site access log visualization tool, aka Apache Pong
  • logstash - Collect, parse, and store logs. A component of the popular ELK stack
  • logstash-forwarder - Formerly Lumberjack, forwards logs to centralized server(s), very lightweight
  • multilog - Reads a sequence of lines from stdin and appends selected lines to any number of logs
  • NXLOG - Universal log collector and forwarder, supports many formats/platforms/sources including Windows
  • OpenTSDB - Store and serve massive amounts of time series data without losing granularity
  • Piwik - Web analytics platform with a killer UI
  • Webalizer - Web and FTP log analyzer

Monitoring and Alerting

  • Adagios - Web based Nagios configuration interface
  • Awesome Dashboard - A collection of dashboards
  • Bosun - By Stack Exchange
  • Cabot - Self-hosted, easily-deployable monitoring and alerts service - like a lightweight PagerDuty
  • Cacti - Web-based network monitoring and graphing tool designed as a front-end to RRDtool
  • check_mk - Extensions / plugins for Nagios
  • collectd - Collects system performance statistics
  • collectd Related Sites - Great tools that integrate with collectd
  • Dashing - Dashboard framework
  • Dashing Widgets - Massive collection of widgets for Dashing
  • Freeboard - Ridiculously simple dashboards for your devices
  • Fully Automated Nagios - Easy to install, pre-packaged Nagios that includes the most-used tools and plugins
  • Ganglia - Monitoring software that's focused on HPC / distributed clusters
  • Grafana - Modern dashboard for Graphite
  • Graphite - Store numeric time-series data and render graphs of the data on demand
  • Graphite-Sankey - Turns Graphite targets into a D3.js Sankey visualization
  • Graphite Tools - Tools that work with Graphite
  • Icinga - Nagios fork, updated frequently
  • jmxtrans - Connector between speaking to a JVM via JMX on one end and whatever logging / monitoring / graphing package
  • LibreNMS - GPL fork of Observium
  • linux-dash - A simple web dashboard to monitor your Linux server
  • Monit - Includes tools to automatically take action when certain conditions are met
  • Monitorix - So lightweight that it can be used in mobile devices, aka Mikaku
  • Multi Router Traffic Grapher (MRTG) - Still being updated!
  • Naemon - Modular Nagios fork
  • Nagdash - Dashboard / NOC screen for Nagios
  • Nagios - One of the most widely used OSS monitoring programs
  • Nagiosgraph - Another visualization tool for Nagios data
  • NagVis - Visualization suite for Nagios
  • NeDi - Network discovery and monitoring
  • Network Weathermap - Network visualization tool, create a "weather map" just like big ISPs
  • OpenNMS - Network monitoring, also supports configuration/asset management
  • Open Monitoring Distribution (OMD) - Nagios bundled with the most popular plugins and tools
  • OpsView Core - GPL version of OpsView, not as feature-rich as the commercial product
  • php-weathermap-graphite - Allows Network Weathermap to source data from Graphite
  • pmacct - IP and network traffic accounting / monitoring
  • PNP4Nagios - Nagios visualization tool
  • riemann - Aggregates events from your servers and applications with a powerful stream processing language.
  • RRDtool - High performance data logging and graphing system for time series data
  • Sensu - Open source monitoring framework, cloud-focused
  • Seyren - Alerting dashboard for Graphite
  • Shinken - Compatible with Nagios
  • Skyline - Designed to be used wherever there are a large quantity of high-resolution timeseries which need constant monitoring
  • StatsD - Monitoring in node.js
  • Tattle - Self service alerting and dashboard frontend for Graphite and Ganglia
  • Thruk - Web based, can create SLA reports, has a mobile client
  • Uchiwa - Dashboard for Sensu
  • Xymon - Fork of Big Brother
  • Zabbix - Stores monitoring data in a DB, has agents for almost every OS
  • Zenoss - Supports Nagios plug-in format

Network Load Generation Tools

  • Flowgrind - Distributed TCP traffic generator
  • iperf3 - Supports tuning of various parameters related to timing, protocols, and buffers
  • netperf - The web page looks ancient but the software's still being updated!
  • netperf-wrapper - Python wrapper to run multiple simultaneous netperf instances and aggregate the results
  • ostinato - "WireShark in reverse"

Networking Tools

  • Argus - Generates network activity reports
  • bbFTP - Uses it's own transfer protocol, optimized for large file transfers
  • BIRD - (Almost) fully functional IP routing daemon for Linux, supports tons of standard routing protocols
  • Charles Web Debugging Proxy - Highly recommended, includes support for decoding HTTPS traffic and throttling -- commercial software
  • CoDel - Network scheduling algorithm designed to overcome bufferbloat
  • Comcast - Simulate crappy network connections
  • DNSsec - Reddit posters in r/netsec explain DNSSec
  • firewalld - Replaces iptables, included in RHEL/CentOS 7
  • ipcalc - CLI tool to calculate subnets, netmasks, IP ranges, broadcast addresses, and more
  • GridFTP - Supports parallel streams
  • lftp - Supports many protocols (FTPS, HTTPS, SFTP), scheduling, bandwidth throttling, scripting, and more
  • localtunnel - Share a web service on your local development machine without messing with DNS and firewall settings
  • mitmproxy - Intercept, modify, replay and save HTTP/S traffic, edit flows on the fly
  • moloch - Large scale IPv4 full PCAP capturing, indexing and database system
  • mtr (My traceroute) - Combines ping and traceroute into a single program
  • namebench - Hunts down the fastest DNS servers for your computer to use
  • ncat - Improved netcat, written by the Nmap team
  • netbrake - Limit the bandwidth used by a process
  • nfdump - Captures network flows
  • NfSen - GUI for nfdump
  • ngrep - Network grep
  • nping - Create custom network packets
  • ntopng - New version of the popular ntop tool
  • Paris Traceroute - Shows proper network topology when load balancers and load-balanced routers are used
  • PageKite - Makes web servers or SSH publicly available on any server
  • prettyping.sh - bash wrapper for ping to create pretty output
  • Server Name Indication (SNI) - Think vhosts for SSL
  • SMB Multichannel - Multiple SMB streams over a single TCP connection, killer feature
  • socat - netcat on steroids, supports serial devices
  • sslh - Protocol multiplexer, forwards packets based on the first packet
  • Stanford Linear Accelerator Center - Network Monitoring Tools - MASSIVE list of network monitoring tools
  • tc - Built-in Linux kernel traffic control
  • tcpflow - TCP demultiplexer, each flow is stored in it's own file
  • TCPWatch - TCP monitoring and logging tool
  • trickle - Userspace bandwidth shaper
  • WireShark - The classic network analyzer
  • WireShark Tools - Superb list of network tools from the WireShark wiki
  • ZeroTier - Cloud / provider-agnostic private network backplanes

Packages and Repositories

  • apt-dater - ncurses frontend for package management via SSH, also supports yum
  • aptly - Debian-centric, includes repo support and the ability to take snapshots for easy rollback
  • CentOS Errata for Spacewalk (CEFS) - Import errata information from CentOS-announce into Spacewalk
  • CentOS Repositories - Large list of both official and unofficial CentOS software repositories
  • CentOS Software Collections (SCL) - Use multiple versions of software on a system without disturbing the system default version
  • Copr - Automatic build system providing a package repository as its output
  • DNF - Package management system that will replace Yum in upcoming Fedora releases
  • ELRepo - Repo that focuses on hardware related packages, supports RHEL and CentOS
  • Extra Packages for Enterprise Linux (EPEL) - Supports CentOS, RHEL, Scientific Linux, and Oracle Linux
  • Extra Packaging Guidelines and Policies for EPEL - Packaging guidelines, great even if not creating EPEL stuff
  • Fedora Packaging Guidelines - Excellent information that can be applied to CentOS & RHEL
  • Fucking Package Management (fpm) - Build packages for multiple platforms (deb, rpm, etc) with great ease and sanity
  • IUS SafeRepo Initiative - Create "safe" 3rd party repos
  • Koji - Software that builds packages for Fedora, can be used for other stuff too, uses mock
  • Mock - Build packages in a simple chroot so you don't blow up your system
  • mrepo - Yum & APT repository mirroring, formerly Yam
  • OStree - Tool for managing bootable, immutable, versioned filesystem trees (not really a package manager but...)
  • Package Repository Manager (PRM) - OS-independent package repository tool
  • Pulp - Next generation repository management, a component of Red Hat Satellite 6
  • Red Hat Software Collections (SCL) - Use multiple versions of software on a system without disturbing the system default version
  • reposync - Synchronize yum repositories to a local directory
  • rpm-ostree - Take RPM packages and commit them to an OSTree repository
  • RPM Fusion - Provides software that the Fedora Project or Red Hat doesn't want to ship
  • Spacewalk - Provides package management (among other things)
  • yum-presto - yum plugin that provides support for downloading package deltas instead of whole packages
  • yum-security - Plugin to only install security updates

Password Management

  • 1Password - Password manager for OS X, Windows, iOS, and Android -- closed source
  • ccrypt - Replacement for the standard Unix "crypt" utility
  • Diceware - Neat passphrase generator
  • GNU Privacy Guard (GPG) - The original! Store em in text files and encrypt
  • GRC Password Generator - Generates 63 bit passwords, web based
  • KeePass - Lightweight, easy to use password manager, runs on most OSes, requires MONO for non-Windows systems
  • Keepass2Android - Compatible with KeePass 2.x, requires MONO
  • KeePassDroid - Compatible with KeePass 1.x
  • KeePassX - Alternative to KeePass, requires QT
  • kpcli - Command line interface for KeePass
  • LastPass - Fremium, can sync to other devices -- closed source
  • lastpass-cli - CLI for LastPass
  • msktutil - Keytab client for a Microsoft Active Directory environment
  • One-Time Secret - Create links that self-destruct after a single viewing, great for sharing passwords
  • pass - Uses GPG at it's core, supports tracking password changes in git
  • Password Gorilla - Written in Tcl/Tk
  • Password Pusher - Links to passwords expire after a certain number of views and/or time has passed, RoR app
  • Password Safe - Designed by Bruce Schneier himself
  • Pleasant Password Server - Multi-user management tool for KeePass, great for businesses -- commercial software
  • pwgen - Password generated, included in most Linux distros
  • RatticDB - Web-based password management, includes an API, great for teams or companies
  • scrypt - Super secure against hardware brute-force attacks, is also a part of/module for most programming languages
  • TeamPass - Collaborative password management using the LAMP stack
  • webpasswordsafe - Very powerful, designed "for the enterprise"
  • zxcvbn - Password strength estimator, written by Dropbox, operates in a browser window

Performance Monitoring

  • apachetop - top for Apache
  • atop - Supports both real-time and historical monitoring
  • bandwidthd - Monitors and graphs bandwidth usage
  • blktrace - Article on blktrace, includes tldr setup steps in the comments
  • bmon - Console based network monitor
  • bwm-ng - Small and simple console-based live network and disk I/O bandwidth monitor
  • collectl - sar on steroids
  • Conky - Lightweight system monitor for X windows
  • Diamond - Python daemon that collects system metrics and publishes them to Graphite (or similar), has an API
  • dim_STAT - Collects almost everything and stores it in a MySQL database, produces reports too
  • dstat - Versatile replacement for vmstat, iostat, netstat and ifstat
  • ESnet Fasterdata Knowledge Base - Provides proven, operationally sound methods for troubleshooting and solving performance issues
  • findswappers - perl script that identifies swapped processes
  • Glances - Real-time monitoring, written in curses and Python
  • htop - top replacement, has a few additional features
  • iftop - top for network interfaces
  • ioping - Monitor I/O latency in real time
  • iotop - top for I/O requests, displays information on a per-process basis
  • iptraf - A top'esque program for network traffic, has multiple display modes, written in ncurses
  • ksar - Creates pretty graphs from sar output
  • Linux Performance Observability Tools - Awesome graphic that shows you which tool to use, by Brendan Gregg
  • mem_logger.sh - Monitor a processes' memory usage over time
  • Munin - Historical performance monitoring to help determine when you server became "slow"
  • nbwmon - Ncurses bandwidth monitor
  • netatop - Kernel module for atop to watch and report on network packets
  • NetHogs - Displays per-process bandwidth usage
  • ngxtop - top for nginx
  • nicstat - vmstat for network interfaces
  • nload - Console application that monitors network traffic and bandwidth usage in real time, neat ASCII graphs
  • nmon - Provides both real-time and historical monitoring
  • PageSpeed Module - Reduce page load time without modify your existing content or workflow
  • perf-tools - By Brendan Gregg, uses perf and ftrace, includes iosnoop, KILLER TOOLS
  • Performance Co-Pilot (PCP) - Included in RHEL / CentOS 7
  • pidstat - vmstat type output for CPU, disk I/O, page faults, and more on a per-process basis
  • pmap - Shell scripts for tracking memory usage using "pmap"
  • PowerTOP - Real-time power consumption on a per-process & per-thread basis
  • ps_mem - Accurately reports core memory usage for a process
  • recap - Collects info from various standard utilities (free, sar, vmstat, etc) at specified intervals
  • slabtop - Tutorial on how to use slabtop, useful when you have no idea on how your RAM is being used
  • smem - Reports memory usage based on proportional set size (PSS) instead of the usual resident set size (RSS)
  • sysdig - "Think of it as strace + tcpdump + lsof + awesome sauce.", recommended
  • SmokePing - Network latency visualizer, written by the MRTG and RRDtool guy
  • vnStat - Console based traffic monitor, supports statistic collecting
  • wtop - Another top for Apache

Provisioning and Lifecycle Management

  • cloud-init - Configures settings the first time a system spins up (SSH keys, hostname, almost anything else)
  • Cobbler - Provisioning system used in Red Hat's Satellite
  • Crowbar - Multi-purpose provisioning tool, originally created by Dell
  • edeploy - provision and update systems using trees of files instead of packages or VM images
  • Foreman - Provisioning and life cycle management
  • gPXE - GPL'ed version of PXE
  • iPXE - GPL'ed version of PXE, replacement for gPXE
  • Katello - Candlepin + Pulp + Foreman, the core of Red Hat Satellite 6
  • Kickstart - The classic Red Hat tool
  • Metal as a Service (MaaS) - Ubuntu provisioning system
  • Packer - Automates VM creation for multiple platforms (VMware, AWS, etc), can use a "golden image" approach
  • Razor - Puppet provisioning software that handles bare metal hardware and virtual server provisioning
  • Sandstorm.io - Download, install, and run apps in a container
  • Terraform - Infrastructure provisioning using existing tools
  • Vagrant - "Hypervisor wrapper" that leverages CM software to quickly spin up environments
  • veewee - Automate all the steps for building Vagrant base boxes

Python Tutorials

RDBMS Load Generation and Performance Tools

Regular Expressions

Security Tools

  • Advanced Intrusion Detection Environment (AIDE) - File integrity checker, alternative to Tripwire
  • Angry IP Scanner - Supports plugins, also gathers NetBIOS information
  • Automatic Server Hardening - Scripts and cookbooks/manifests for Puppet and Chef
  • BlackArch Linux - Penetration testing Linux distro, based off of Arch Linux
  • BoringSSL - Google's fork of OpenSSL, does not guarantee API and ABI compatibility
  • Brakeman - Static analysis security vulnerability scanner for Ruby on Rails applications
  • bro - Framework for network analysis and security monitoring
  • chkrootkit - Rootkit checker, best used from a live CD
  • cipherscan - Find out which SSL ciphersuites are supported by a target
  • CIS-CAT Benchmark Tool - Center for Internet Security's benchmark tool, evaluates systems based on their hardening recommendations
  • CIS Linux Benchmarks - Linux OS hardening guides, superb!
  • Computer Aided INvestigative Environment (CAINE) - Live CD that focuses on forensics
  • ConfigServer Security and Firewall (CSF) - Stateful Packet Inspection (SPI) firewall, Login/Intrusion Detection and Security application for Linux servers
  • Cryptkeeper - System tray applet that manages EncFS encrypted folders
  • dnscrypt-wrapper - Add dnscrypt support to any resolver
  • diskscrub - Supports many disk-wiping standards including military / government grade wipes
  • duraconf - A collection of hardened configuration files for SSL/TLS services by Jacob Appelbaum
  • EncFS - Encrypted file system in user space via FUSE
  • fail2ban - Watches log files to ban IPs based on rules (too many failed logins, exploit attempts, brute force attacks, etc)
  • fail2ban Dashboard - Written with Flask
  • Forensics Wiki - Digital forensics wiki
  • FTimes - System baselining and evidence collection tool
  • fwknop (FireWall KNock OPerator) - Single Packet Authorization (SPA) aka next-generation port knocking
  • HoneyDrive - Honeypot appliance
  • hping - Create custom TCP/IP packets, very flexible, see also hping3
  • HTTP Strict Transport Secuirity (HSTS) - Forces browsers to interact with a site by only using HTTPS
  • Information Security Cheat Sheets and Checklists - An assortment of IS checklists and cheat sheets
  • Kali Linux - Penetration testing Linux distro
  • Kippo - SSH honeypot
  • knockd - Port knocking server for Linux
  • LibreSSL - OpenBSD's fork of OpenSSL, Linux compatible
  • Lynis - Auditing and hardening tool, supports most Unix-like operating systems
  • Metasploit - Classic exploit framework
  • mtree - Can be used as a simple IDS
  • Network Security Toolkit (NST) - Live CD that includes most tools in insecure.org's top 125 tools list
  • nmap - Classic port scanner
  • nmap Scripting Engine (NSE) - Tons of scripts for nmap
  • nogotofail - Spot and fix weak TLS/SSL connections and sensitive cleartext traffic
  • OpenSCAP - Standardized approach to maintaining the security of systems, evaluates machine compliance with a given profile
  • OpenVAS - Vulnerability scanner, forked from the now closed-source Nessus scanner
  • OpenVPN - Well known open source VPN
  • Open Source Tripwire - File integrity checker and monitor, replacement for the now closed-source Tripwire
  • Open Web Application Security Project (OWASP) - Focus on application security
  • Oracle Database Attacking Tool (ODAT) - Open source penetration testing tool that tests the security of Oracle databases remotely
  • OSSEC - Host based intrusion detection system (HIDS), supports most Unix-like OSes
  • pfsense - The best firewall software, supports appliances and Live CDs
  • pyCAmanager - GUI for managing your certificate authority, written in Python
  • Qualys SSL Server Test - Evaluates and provides recommendations for the SSL settings of any web site
  • r/netsec Tools List - r/netsec creates a list of security related tools
  • Red October - Go server for two-man rule style file encryption and decryption
  • Rootkit Hunter - Compares hashes of important files with known good hashes that are stored in online databases
  • Samhain - HIDS, file integrity checker, rootkit detection, log file monitoring, and more
  • ScoutBot - Automated network scanner for Raspberry Pi
  • Security Engineering - Free online book
  • Security Onion - Linux distro for IDS, NSM, and log management
  • Snoopy Logger - Logs commands that are executed and saves the information to syslog
  • Snort - The classic network intrusion prevention system (NIPS)
  • SoftEther - Maybe the best VPN software out there
  • SSH Guard - Think fail2ban for SSH
  • sslconfig - CloudFlare's Internet facing SSL cipher configuration
  • sslsniff - A tool for automated MITM attacks on SSL connections
  • SSLsplit - Transparent and scalable SSL/TLS interception
  • sslyze - Fast and full-featured SSL scanner, written in Python
  • sudosh2 - Login shell that records all keystrokes and output, supports session playback
  • tinc - Simple, multi-platform VPN
  • Tripwire - File integrity checker and monitor, open source version
  • VirusTotal - Allows you to upload a file and have it scanned by tons of virus scanners
  • ZMap - Scan the entire Internet (or a specified IPv4 range)
  • zuluCrypt - Simple CLI and GUI frontend to cryptsetup and tcplay

Shell Scripting

SSH Tools

  • autossh - Automatically restart SSH sessions that stop passing traffic
  • Balancing Load Across Systems (Ballast) - Balance user load across SSH servers based on configured policies
  • Bitvise SSH Client - Feature-rich SSH & SFTP client for Windows, free for individual use -- closed source
  • Byobu - An enhanced version of the "screen" utility
  • cloud-ssh - SSH wrapper that supports hostname tab-completion based on name, tag, or security group
  • ClusterSSH - Make a change on many servers at the same time
  • Corkscrew - Tunnel SSH through HTTP proxies
  • csshX - Cluster SSH for OS X
  • Dancer's Shell (dsh) - Wrapper for executing multiple remote shell (rsh/remsh/ssh) commands
  • Guacamole - Clientless (in-browser) remote desktop gateway, supports VNC, RDP, and SSH
  • HPN-SSH - High performance ssh/scp (auth handshake is encrypted, data transfer is not)
  • KeyBox - Web-based SSH console that executes commands on multiple shells
  • Keychain - frontend to ssh-agent and ssh-add, reduces the number of times you have to type in your passphrase
  • KiTTY - PuTTY fork with additional features
  • Mass Parallel SSH (mpssh) - Simple parallel SSH
  • Match - Creates a conditional block, great for controlling actions on a per-user and/or per-host basis
  • MobaXterm - Tabbed SSH, VNC, and RDP client for Windows, free for personal use -- closed source
  • Mobile Shell (Mosh) - Remote shell that supports roaming (client IP address changes) and intermittent connectivity
  • mRemoteNG - Open source, tabbed, multi-protocol, remote connections manager
  • MTPuTTY (Multi-Tabbed PuTTY) - Multiple PuTTY sessions in a single window
  • MUltihost SSH Wrapper (Mussh) - SSH wrapper
  • NoVNC - Client-less VNC in a web browser, uses HTML5 and WebSockets
  • parallel-ssh (pssh) - Parallel version of OpenSSH tools, comes with prsync too
  • Parallel Distributed Shell (pdsh) - Kick off many SSH sessions in parallel
  • paramiko - SSH2 protocol library for Python, provides both client and server
  • Parcel - High performance scp replacement, WARNING: uses UDP
  • PuTTY - The classic SSH client for Windows
  • rssh - Restricted shell for SSH, only allows scp, sftp, rdist, rsync, and cvs
  • scponly - Shell that only allows incoming SCP, supports chroot
  • SSH Power Tool (sshpt) - Execute commands and upload files to many servers simultaneously via SSH without using pre-shared keys
  • SSHFS - Mount remote file systems using a SSH tunnel
  • sshrc - Sources the ~/.sshrc on your local computer after logging in remotely
  • sshuttle - Transparent proxy server that works as a poor man's VPN, forwards over SSH, supports DNS tunneling
  • SuperPutty - Allows the PuTTY SSH client to be opened in tabs
  • tmux - An alternative to GNU screen
  • tmux-cssh - Cluster SSH for tmux
  • WinSCP - The well-known SCP client for Windows
  • Xming - Windows Xwindows server, supports PuTTY

Storage Tools

  • ATA Secure Erase - Send a signal to an ATA drive to perform a hardware-based erase
  • bcache - Block layer cache, allows SSDs to be used as caches for slower drivers
  • EnhanceIO - SSD caching, in the mainline Linux kernel, a fork of Facebook's Flashcache project
  • Flash-Friendly File System (F2FS) - File system designed specifically for SSDs
  • fs-cache - NFS client caching
  • IPOS Calculator - Created by Marek Wolynko
  • Linux-IO Target (LIO) - Create and share iSCSI, FC, FCoE, and other storage targets on Linux
  • Linux Journal - Linux Swap Space - Superb article about Linux swap, includes some tuning parameters
  • lvcache - Wrapper to make lvmcache easier
  • OpenDedupe - Open source dedupe software, works on both block storage and file systems
  • Parted Magic - Resize, grow, shrink, clone, recovery, wiping, benchmarking, and more. Supports Linux and Windows file systems.
  • RAID LVMs - As of RHEL 6.3 LVM supports software RAID without using mdraid
  • Red Hat LVM RAID Calculator - Login required
  • snapper - CLI tool to manage Btrfs snapshots, snapshot timelines, and more
  • System Storage Manager (SSM) - Generic CLI for managing all types of storage (DM, LVM, multipath), added in RHEL 7
  • zram - A compressed, in-memory block device to reduce paging
  • zswap - A compressed, write-back cache cache for swapped pages

Tracing Debugging Profiling

Two Factor Authentication

  • Authy - Provides both personal and enterprise versions
  • Barada Aint Respecting Any Deceptive Adversaries (BARADA) - PAM module for TFA support
  • Crypto Stick - OTP and more -- commercial products
  • Duo Security - Cloud-based, supports many methods of authentication (hard token, smartphone application, etc) -- commercial product
  • FreeOTP - Open source fork of Google Authenticator for iOS and Android
  • Google Authenticator - Two factor authentication; supports Android, iOS, and even has a PAM module -- closed source
  • Two Factor Auth (2FA) - Massive list of web sites that support and don't support two factor authentication
  • YubiKey - Acts as a USB keyboard to automatically input a strong, one time password -- commercial product
  • Yubico PAM - PAM module for use with YubiKey devices

Virtualization

Misc Tools and Utilities

  • ack - grep-like tool designed to search source code
  • Adminer - GUI for database management in a single PHP
  • appscale - Deploy and host Google App Engine applications almost anywhere
  • AutoHotKey - Automate input (desktop and web forms, data entry, keybinds, etc)
  • autojump - Small database of directories that you visited in the past, used to quickly navigate complex directory structures
  • Awesome Node.js - A curated list of Node.js packages
  • Awesome PHP - Curated list of PHP libraries and resources
  • Awesome SQLAlchemy - A curated list of awesome tools for SQLAlchemy
  • Awesome Sysadmin - A curated list of open source resources
  • blockdiag - Generate simple block/sequence/activity/network diagrams from text files
  • boto - Python interface to AWS
  • Checkpoint/Restore in Userspace (CRIU) - Freeze a process, save it to disk, then resume it later
  • comm - Display lines that two files have in common (eg: the opposite of diff)
  • conserver - Virtual console server with many features
  • curlish - Improved curl, includes cookie management and OAuth support
  • cwrap - Wrappers for creating test scenarios and faking behavior, super useful
  • Cygwin - GNU shell and tools for Windows
  • devcheatsheet.com - Cheat sheets and quick references for almost every programming language
  • disown - Disowns a processes from the shell, useful if you forgot to use screen or nohup
  • dotfiles - Store your .bashrc and others on GitHub
  • dsync - No software yet but a hell of a theory
  • eatmemory - Allocates X amount of memory, great for testing
  • etckeeper - Stores /etc in version control
  • Etherpad - Enter, save, and share text/code in a web browser
  • f.lux - Changes your screen from blue light to yellow light when the sun sets to tell your brain it's night time -- closed source
  • FastGlacier - Amazon Glacier client for Windows -- closed source
  • FileUtils - File utilities for distributed computing
  • Filewatcher - Written in Ruby, monitors changes in the filesystem by polling
  • ForgeRock - Enterprise identification management and access (IAM)
  • FreeIPA - Identity, policy, and audit suite, think Active Directory for Linux
  • gcalcli - CLI for Google Calendar
  • Geocities-izer - Make any web page look like a Geocities page
  • Gource - Software version control visualization tool
  • grabmem - Allocates memory until the machine runs out or you quit it with ctrl-c
  • Gravit - Web-based vector image editor / design tool, source code is available on GitHub
  • gt5 - A "du browser", handy for finding large files
  • HasteBin - Open source pastebin alternative for sharing code, can be installed locally / on-site
  • HeidiSQL - GUI SQL DB browser and editor
  • httpie - curl replacement with many new features
  • incrond - Fires off a job when a file system event happens
  • Inkscape - Open source vector image editor
  • IRCcat - Sent log messages and alerts to an IRC channel, can also use IRC to send commands
  • ISO 8601 - ISO standard for dates, the only way to fly
  • JSON Formatter - Automatically format JSON code, 100% client side
  • Inconsolation - MASSIVE list of lightweight and (legit) obscure tools
  • inotify - Watch file(s) for changes, take an action when the change occurs
  • inxi - "a full featured system information script" (hardware info, etc)
  • Keybase - Upload your public key and find other people's public key via their social media user name(s)
  • LargeFile - vim plugin that automatically disables certain things so you can edit large files faster
  • lz4 - Crazy fast compression algorithm
  • maim - Linux desktop screenshot tool
  • Mergeatron - Kicks off Jenkins when it sees a PR, gives a thumbs up/down reply
  • mockaroo - Generate up to 100,000 rows of realistic test data in CSV, JSON, SQL, and Excel formats
  • moreutils - "Collection of the unix tools that nobody thought to write long ago when unix was young"
  • multicast remote sync (mrsync) - Transfers from a master to many remote machines using Unix multicast sockets
  • multitail - Monitor multiple log files in a single terminal window
  • ncdu - ncurses version of "du"
  • Netflix OSS - Netflix OSS tools on GitHub
  • Notepad++ - Killer GPL'ed text editor for Windows
  • observr - Monitor a directory and kick off an action when a file in the directory changes
  • OpenResty - Turn nginx into a non-blocking API server
  • parallel - Execute jobs in parallel using one or more computers
  • percol - Interactive grep tool
  • pv - Shows the progress of data as it flows through a pipe
  • Rackerlabs - Rackspace's OSS tools, hosted on GitHub
  • rcron - cron redundancy and failover, ensures a job will only run on the "active" machine
  • Repose - REST proxy
  • Request Tracker - Bug tracking, help desk ticketing, customer service, workflow processes, change management and more
  • reset - Reset your terminal after you accidentally cat a binary file
  • Rundeck - Easy automation
  • ScreenCloud - Take a screenshot and either upload it to their site to share, or save it locally
  • script - Standard utility to record commands, input, and output
  • Silver Searcher, The - Source code searching tool, similar to "ack" but faster
  • Simple Event Correlator (SEC) - Umm, no idea where to put this one, it's super multi-purpose
  • Snappy - Compression that aims for very high speed and resonable compression, by Google
  • SQL Fiddle - Write and test SQL
  • Sticky Notes - On-site / local pastebin
  • Sublime Text - Ok it's closed source and costs money but it's probably the best one out there
  • Synergy - Share a single keyboard and mouse with multiple physical computers
  • sysadmin-util - A collection of small but useful utilities
  • tac - Reverse "cat" (eg: last line displayed first)
  • tengine - A distribution of Nginx with some advanced features
  • TermRecord - Record and playback terminal sessions
  • Tools of the Trade - Massive list of SaaS, via Hacker News
  • tup - Super fast and efficient build system
  • UDR - UDT wrapper for rsync that improves throughput of large data transfers over long distances
  • Unix Rosetta Stone - The one and only resource for a mixed Linux/Unix environment
  • Varnish - Caching HTTP accelerator
  • watch - Run a command repeatedly at a specified interval and display the output
  • watch-make - Monitor Makefile dependencies and kick off make when one changes
  • watchman - Watch files and take action when they change (eg: kick off the CI system)
  • xCAT - Complete management solution for distributed systems
  • Your Own URL Shortener (YOURLS) - Lets you run your own URL shortener a'la TinyURL or bit.ly
  • watchr - Monitor a directory tree and kick off a script when a file changes, written in Ruby
  • Webjob - Downloads a program or script from a remote WebJob server, executes it, and sends back output to the server
  • zgrep - grep the contents of a gzip'ed file without uncompressing and piping
  • zopfli - Much slower than zlib but (almost) always compresses better

About

A collection of OSS tools and handy links