A guide covering Digital Forensics the applications, libraries and tools that will make you a better and more efficient with Digital Forensics development.
Note: You can easily convert this markdown file to a PDF in VSCode using this handy extension Markdown PDF.
Digital Forensics is the process of recovering and preserving material found on digital devices during the course of criminal investigations. Digital forensics tools include hardware and software tools used by law enforcement to collect and preserve digital evidence and support or refute hypotheses before courts.
Computer Forensics is the process of examining digital media in a forensic-like manner with the goal of identifying, preserving, recovering, analyzing and presenting facts and opinions about the digital information.
Mobile device forensics is the science of recovering digital evidence from a mobile device under forensically sound conditions using accepted methods. Mobile device forensics is an evolving specialty in the field of digital forensics.
Network forensics is a science that centers on the discovery and retrieval of information surrounding a cybercrime within a networked environment. Common forensic activities include the capture, recording and analysis of events that occurred on a network in order to establish the source of cyberattacks.
Database forensics is the process of interrogating a failed database and trying to reconstruct the metadata and page information from within a data set, whereas database recovery implies some kind of restorative process that will enable the database to become viable enough to be put back into a production environment, or healthy enough to provide a backup that can be used in a database restore.
-
OSSTMM (Open Source Security Testing Methodology Manual) PDF
-
NIST Technical Guide to Information Security Testing and Assessment (PDF)
-
Open Source Security Foundation (OpenSSF) npm Best Practices Guide
-
Open Source Security Foundation (OpenSSF) Best Practices for Open Source Developers
-
Open Source Security Foundation (OpenSSF) Identifying Security Threats in Open Source Projects
-
Learn Computer Forensics with Online Courses and Lessons | edX
-
Computer Forensics Courese Learning Path - Infosec Institute
-
National Computer Forensics Institute(NCFI) Training Courses
-
Mile2's Certified Digital Forensics Examiner training course
-
Cyber Security Training, Certifications, Degrees and Resources | SANS Institute
Autopsy® is a digital forensics platform and graphical interface to The Sleuth Kit® and other digital forensics tools. It is used by law enforcement, military, and corporate examiners to investigate what happened on a computer. You can even use it to recover photos from your camera's memory card.
The Sleuth Kit® (TSK) is a library and collection of command line tools that allow you to investigate disk images. The core functionality of TSK allows you to analyze volume and file system data. The library can be incorporated into larger digital forensics tools and the command line tools can be directly used to find evidence.
PTK Forensics is a computer forensic framework for the command line tools in the SleuthKit plus much more software modules.
DFF (Digital Forensics Framework) is a free and Open Source computer forensics software built on top of a dedicated Application Programming Interface (API). It can be used both by professional and non-expert people in order to quickly and easily collect, preserve and reveal digital evidences without compromising systems and data.
Mobile Device Investigator® is a security tool that powers rapid investigations of iOS and Android devices by connecting a suspect device via USB port to perform logical acquisitions.
Digital Evidence Investigator® is a digital forensic tool for Windows, Linux, and macOS (including T2 and M1 chips). DEI collects digital evidence and presents it in a timeline view to tie the user to files and artifacts.
Digital Evidence Investigator® PRO is a tool that includes Windows, Linux and macOS (including T2 and M1 chips) computer forensic capabilities of Digital Evidence Investigator® and Mobile Device Investigator® iOS/Android capabilities in a single license.
Guymager is a free forensic imager for media acquisition. Its main features are: Easy user interface in different languages. Really fast, due to multi-threaded, pipelined design and multi-threaded data compression. Generates flat (dd), EWF (E01) and AFF images, supports disk cloning. Free of charges, completely open source.
X-Ways Forensics is a commercial digital forensics platform for Windows.
X-Ways Investigator is a reduced, simplified version of X-Ways Forensics for police investigators, lawyers, and auditors.
WinHex is a Hex editor, disk editor, and RAM editor. Computer forensics, data recovery, and IT security tool.
F-Response is a remote network drive analysis capability, remote RAM access, and cloud storage access.
AccessData Forensics Toolkit (FTK®) is built for speed, stability and ease of use. It provides comprehensive processing and indexing up front, so filtering and searching is faster than with any other product. This means you can zero in on the relevant evidence quickly, dramatically increasing your analysis speed.
OpenText™ EnCase™ is a commercial forensics platform. It offers support for evidence collection from over twenty-five different types of devices, including desktops, mobile devices and GPS. Within the tool, a forensic investigator can inspect the collected data and generate a wide range of reports based upon predefined templates.
Redline® is FireEye's premier free endpoint security tool, provides host investigative capabilities to users to find signs of malicious activity through memory and file analysis and the development of a threat assessment profile. It collects information about running processes on a host, drivers from memory and gathers other data like meta data, registry data, tasks, services, network information and internet history to build a proper report.
Paraben's Electronic Evidence Examiner—E3 is a comprehensive digital forensic platform designed to handle more data, more efficiently while adhering to Paraben's paradigm of specialized focus of the entire forensic exam process. Paraben has capabilities in:
- Desktop forensics
- Email forensics
- Smartphone analysis
- Cloud analysis
- IoT forensics
- Triage and visualization
Bulk Extractor is a program that extracts features such as email addresses, credit card numbers, URLs, and other types of information from digital evidence files. It is a useful forensic investigation tool for many tasks such as malware and intrusion investigations, identity investigations and cyber investigations, as well as analyzing imagery and pass-word cracking.
Registry Recon is a powerful computer forensics tool developed by Arsenal Recon. The tool is used to extract, recover, and parse registry data from Windows systems. The process of manually scouring Windows Registry files proves to be extremely time consuming and leaves gaping holes in the ability to recover critical information.
Volatility is the memory forensics framework. It is used for incident response and malware analysis. With this tool, you can extract information from running processes, network sockets, network connection, DLLs and registry hives. It also has support for extracting information from Windows crash dump files and hibernation files.
WindowsSCOPE is a commercial memory forensics and reverse engineering tool used for analyzing volatile memory. It is basically used for reverse engineering of malware. It provides the ability to analyze the Windows kernel, drivers, DLLs and virtual and physical memory.
Wireshark is the most widely used network traffic analysis tool in existence. It has the ability to capture live traffic or ingest a saved capture file.
Network Miner is an open source Network Forensic Analysis Tool (NFAT) for Windows (also Linux, macOS X , and FreeBSD). NetworkMiner can be used as a passive network sniffer/packet capturing tool in order to detect operating systems, sessions, hostnames, and open ports without putting any traffic on the network.
Xplico is an open-source network forensic analysis tool. It is used to extract useful data from applications which use Internet and network protocols. It supports most of the popular protocols including HTTP, IMAP, POP, SMTP, SIP, TCP, UDP, TCP and others. Output data of the tool is stored in an SQLite database or MySQL database. It also supports both IPv4 and IPv6.
Oxygen Forensic Detective is a forensic's tool that focuses on mobile devices but is capable of extracting data from a number of different platforms, including mobile, IoT, cloud services, drones, media cards, backups and desktop platforms. It uses physical methods to bypass device security (such as screen lock) and collects authentication data for a number of different mobile applications.
XRY is a collection of different commercial tools for mobile device forensics. XRY Logical is a suite of tools designed to interface with the mobile device operating system and extract the desired data. XRY Physical, on the other hand, uses physical recovery techniques to bypass the operating system, enabling analysis of locked devices.
SIFT Workstation is another open-source Linux virtual machine that aggregates free digital forensics tools. This platform was developed by the SANS Institute and its use is taught in a number of their courses.
HashKeeper is a central database repository of Forensic Intelligence donated by various sources, usually obtained by law enforcement during the course of forensic investigations of suspect systems.
Forensic Explorer Command Line (FEX CLI) is a forensic data processing engine used for computer forensics and electronic discovery. The FEX CLI can be run on a single workstation to an enterprise level virtual environment spawning multiple simultaneous processing instances.
FEX Memory Imager (FEX Memory) is a free imaging tool designed to capture the physical Random Access Memory (RAM) of a suspect’s running computer. This allows investigators to recover and analyze valuable artifacts found only in memory.
FEX Imager™ is a free forensic imaging program that will acquire or hash a bit-level forensic image with full MD5, SHA1, SHA256 hash authentication. It can acquire a physical drive, logical drive, folders and files, remote devices (using servlet), or re-acquire a forensic image.
Forensic Explorer™ is a flexible and easy to use GUI with advanced sort, filter, keyword search, data recovery and script technology. It can quickly process large volumes of data, automate complex investigation tasks, produce detailed reports and increase productivity.
Rehex is a cross-platform (Windows, Linux, Mac) hex editor for reverse engineering, and everything else.
DIRTY is a Augmenting Decompiler Output with Learned Variable Names and Types developed by the Socio-Technical Research Using Data Excavation Lab, at Carnegie Mellon University.
HVM (Hardware Virtual Machine) is a virtualization type that provides the ability to run an operating system directly on top of a virtual machine without any modification, as if it were run on the bare-metal hardware.
PV(ParaVirtualization) is an efficient and lightweight virtualization technique introduced by the Xen Project team, later adopted by other virtualization solutions. PV does not require virtualization extensions from the host CPU and thus enables virtualization on hardware architectures that do not support Hardware-assisted virtualization.
Virtualization-based Security (VBS) is a hardware virtualization feature to create and isolate a secure region of memory from the normal operating system.
Hypervisor-Enforced Code Integrity (HVCI) is a mechanism whereby a hypervisor, such as Hyper-V, uses hardware virtualization to protect kernel-mode processes against the injection and execution of malicious or unverified code. Code integrity validation is performed in a secure environment that is resistant to attack from malicious software, and page permissions for kernel mode are set and maintained by the hypervisor.
KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko.
QEMU is a fast processor emulator using a portable dynamic translator. QEMU emulates a full system, including a processor and various peripherals. It can be used to launch a different Operating System without rebooting the PC or to debug system code.
Hyper-V enables running virtualized computer systems on top of a physical host. These virtualized systems can be used and managed just as if they were physical computer systems, however they exist in virtualized and isolated environment. Special software called a hypervisor manages access between the virtual systems and the physical hardware resources. Virtualization enables quick deployment of computer systems, a way to quickly restore systems to a previously known good state, and the ability to migrate systems between physical hosts.
VirtManager is a graphical tool for managing virtual machines via libvirt. Most usage is with QEMU/KVM virtual machines, but Xen and libvirt LXC containers are well supported. Common operations for any libvirt driver should work.
oVirt is an open-source distributed virtualization solution, designed to manage your entire enterprise infrastructure. oVirt uses the trusted KVM hypervisor and is built upon several other community projects, including libvirt, Gluster, PatternFly, and Ansible.Founded by Red Hat as a community project on which Red Hat Enterprise Virtualization is based allowing for centralized management of virtual machines, compute, storage and networking resources, from an easy-to-use web-based front-end with platform independent access.
HyperKit is a toolkit for embedding hypervisor capabilities in your application. It includes a complete hypervisor, based on xhyve/bhyve, which is optimized for lightweight virtual machines and container deployment. It is designed to be interfaced with higher-level components such as the VPNKit and DataKit. HyperKit currently only supports macOS using the Hypervisor.framework making it a core component of Docker Desktop for Mac.
Intel® Graphics Virtualization Technology (Intel® GVT) is a full GPU virtualization solution with mediated pass-through, starting from 4th generation Intel Core (TM) processors with Intel processor graphics(Broadwell and newer). It can be used to virtualize the GPU for multiple guest virtual machines, effectively providing near-native graphics performance in the virtual machine and still letting your host use the virtualized GPU normally.
Apple Hypervisor is a frameowrk that builds virtualization solutions on top of a lightweight hypervisor, without third-party kernel extensions. Hypervisor provides C APIs so you can interact with virtualization technologies in user space, without writing kernel extensions (KEXTs). As a result, the apps you create using this framework are suitable for distribution on the Mac App Store.
Apple Virtualization Framework is a framework that provides high-level APIs for creating and managing virtual machines on Apple silicon and Intel-based Mac computers. This framework is used to boot and run a Linux-based operating system in a custom environment that you define. It also supports the Virtio specification, which defines standard interfaces for many device types, including network, socket, serial port, storage, entropy, and memory-balloon devices.
Apple Paravirtualized Graphics Framework is a framework that implements hardware-accelerated graphics for macOS running in a virtual machine, hereafter known as the guest. The operating system provides a graphics driver that runs inside the guest, communicating with the framework in the host operating system to take advantage of Metal-accelerated graphics.
Cloud Hypervisor is an open source Virtual Machine Monitor (VMM) that runs on top of KVM. The project focuses on exclusively running modern, cloud workloads, on top of a limited set of hardware architectures and platforms. Cloud workloads refers to those that are usually run by customers inside a cloud provider. Cloud Hypervisor is implemented in Rust and is based on the rust-vmm crates.
VMware vSphere Hypervisor is a bare-metal hypervisor that virtualizes servers; allowing you to consolidate your applications while saving time and money managing your IT infrastructure.
Xen is focused on advancing virtualization in a number of different commercial and open source applications, including server virtualization, Infrastructure as a Services (IaaS), desktop virtualization, security applications, embedded and hardware appliances, and automotive/aviation.
Ganeti is a virtual machine cluster management tool built on top of existing virtualization technologies such as Xen or KVM and other open source software. Once installed, the tool assumes management of the virtual instances (Xen DomU).
Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.
Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the "works on my machine" excuse a relic of the past. It provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.
Parallels Desktop is a Desktop Hypervisor that delivers the fastest, easiest and most powerful application for running Windows/Linux on Mac (including the new Apple M1 chip) and ChromeOS.
VMware Fusion is a Desktop Hypervisor that deliver desktop and ‘server’ virtual machines, containers and Kubernetes clusters to developers, and IT professionals on the Mac.
VMware Workstation is a hosted hypervisor that runs on x64 versions of Windows and Linux operating systems; it enables users to set up virtual machines on a single physical machine, and use them simultaneously along with the actual machine.
GlusterFS is a free and open source scalable network filesystem. Gluster is a scalable network filesystem. Using common off-the-shelf hardware, you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks.
Ceph is a software-defined storage solution designed to address the object, block, and file storage needs of data centers adopting open source as the new norm for high-growth block storage, object stores and data lakes. Ceph provides enterprise scalable storage while keeping CAPEX and OPEX costs in line with underlying bulk commodity disk prices.
Hadoop Distributed File System (HDFS) is a distributed file system that handles large data sets running on commodity hardware. It is used to scale a single Apache Hadoop cluster to hundreds (and even thousands) of nodes. HDFS is one of the major components of Apache Hadoop, the others being MapReduce and YARN.
ZFS is an enterprise-ready open source file system and volume manager with unprecedented flexibility and an uncompromising commitment to data integrity.
OpenZFS is an open-source storage platform. It includes the functionality of both traditional file systems and volume manager. It has many advanced features including:
- Protection against data corruption.
- Integrity checking for both data and metadata.
- Continuous integrity verification and automatic "self-healing" repair.
Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing on fault tolerance, repair and easy administration. Its main features and benefits are:
- Snapshots which do not make the full copy of files
- RAID - support for software-based RAID 0, RAID 1, RAID 10
- Self-healing - checksums for data and metadata, automatic detection of silent data corruptions
Squashfs is a compressed read-only filesystem for Linux. It uses zlib, lz4, lzo, or xz compression to compress files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimize data overhead.
Apple File System (APFS) is the default file system for Mac computers using macOS 10.13 or later, features strong encryption, space sharing, snapshots, fast directory sizing, and improved file system fundamentals.
NTFS(New Technology File System) is the primary file system for recent versions of Windows and Windows Server—provides a full set of features including security descriptors, encryption, disk quotas, and rich metadata, and can be used with Cluster Shared Volumes (CSV) to provide continuously available volumes that can be accessed simultaneously from multiple nodes of a failover cluster.
exFAT(Extended File Allocation Table ) is the file system that was the successor to FAT32 in the FAT family of file systems. It was optimized for flash memory such as USB flash drives and SD cards.
-
OSSTMM (Open Source Security Testing Methodology Manual) PDF
-
NIST Technical Guide to Information Security Testing and Assessment (PDF)
Common Criteria for Information Technology Security Evaluation (CC) is an international standard (ISO / IEC 15408) for computer security. It allows an objective evaluation to validate that a particular product satisfies a defined set of security requirements.
ISO 22301 is the international standard that provides a best-practice framework for implementing an optimised BCMS (business continuity management system).
ISO27001 is the international standard that describes the requirements for an ISMS (information security management system). The framework is designed to help organizations manage their security practices in one place, consistently and cost-effectively.
ISO 27701 specifies the requirements for a PIMS (privacy information management system) based on the requirements of ISO 27001. It is extended by a set of privacy-specific requirements, control objectives and controls. Companies that have implemented ISO 27001 will be able to use ISO 27701 to extend their security efforts to cover privacy management.
SOC 2 is an auditing procedure that ensures your service providers securely manage your data to protect the interests of your comapny/organization and the privacy of their clients.
NIST CSF is a voluntary framework primarily intended for critical infrastructure organizations to manage and mitigate cybersecurity risk based on existing best practice.
EU GDPR (General Data Protection Regulation) is a privacy and data protection law that supersedes existing national data protection laws across the EU, bringing uniformity by introducing just one main data protection law for companies/organizations to comply with.
CCPA (California Consumer Privacy Act) is a data privacy law that took effect on January 1, 2020 in the State of California. It applies to businesses that collect California residents’ personal information, and its privacy requirements are similar to those of the EU’s GDPR (General Data Protection Regulation).
Payment Card Industry (PCI) Data Security Standards (DSS) is a global information security standard designed to prevent fraud through increased control of credit card data.
Landlock LSM(Linux Security Module) is a framework to create scoped access-control (sandboxing). Landlock is designed to be usable by unprivileged processes while following the system security policy enforced by other access control mechanisms (DAC, LSM, etc.).
Secure boot is a security standard developed by members of the PC industry to help make sure that a device boots(Unified Extensible Firmware Interface (UEFI) BIOS) using only software(such as bootloaders, OS, UEFI drivers, and utilities) that is trusted by the Original Equipment Manufacturer (OEM).
How Encryption Keys work
-
Symmetric is a data encryption method whereby the same private key is used to encode and decode information.
-
Asymmetric is a data encryption method that allows users to encrypt information using shared keys. For example, if you need to send a message across the internet, but you don't want anyone but the intended recipient to see what you've written.
Types of Encryption
-
Triple DES (Triple Data Encryption Algorithm) is a symmetric-key block cipher, which applies the DES cipher algorithm three times to each data block(contains 64 bits of data).
-
AES (Advanced Encryption Standard) is an algorithm that encrypts and decrypts data in blocks of 128 bits. It can do this using 128-bit, 192-bit, or 256-bit keys.
-
RSA (Rivest–Shamir–Adleman) is a type of public-key cryptography used for secure data transmission of e-mail and other digital transactions over the Internet.
-
Twofish is a symmetric key block cipher with a block size of 128 bits and key sizes up to 256 bits. It is an advanced version of Blowfish encryption.
-
Format Preserving Encryption (FPE) is a valid encryption algorithm to be used for compliance with NIST standards. It is mostly used in on-premise encryption and tokenization solutions.
Application Level Encryption
-
Hashes is a function that converts an input of letters and numbers into an encrypted output of a fixed length. For example, algorithms such as MD5 (Message Digest 5) or SHA (Secure Hash Algorithm).
-
Digital Certificates is a file that verifies the identity of a device or user and enables encrypted connections. A digital signature is a hashing approach that uses a numeric string to provide authenticity and validate identity. Digital certificates are typically issued by a certificate authority (CA), which is a trusted third-party entity that issues digital certificates for use by other parties.
Netdata is high-fidelity infrastructure monitoring and troubleshooting, real-time monitoring Agent collects thousands of metrics from systems, hardware, containers, and applications with zero configuration. It runs permanently on all your physical/virtual servers, containers, cloud deployments, and edge/IoT devices, and is perfectly safe to install on your systems mid-incident without any preparation.
Acra is a single database security suite with 9 strong security controls: application level encryption, searchable encryption, data masking, data tokenization, secure authentication, data leakage prevention, database request firewall, cryptographically signed audit logging, security events automation. It is designed to cover the most important data security requirements with SQL and NoSQL databases and distributed apps in a fast, convenient, and reliable way.
OWASP Nettacker is a project created to automate information gathering, vulnerability scanning and eventually generating a report for networks, including services, bugs, vulnerabilities, misconfigurations, and other information. This software will utilize TCP SYN, ACK, ICMP, and many other protocols in order to detect and bypass Firewall/IDS/IPS devices.
OpenSCAP is U.S. standard maintained by National Institute of Standards and Technology (NIST). It provides multiple tools to assist administrators and auditors with assessment, measurement, and enforcement of security baselines. OpenSCAP maintains great flexibility and interoperability by reducing the costs of performing security audits. Whether you want to evaluate DISA STIGs, NIST‘s USGCB, or Red Hat’s Security Response Team’s content, all are supported by OpenSCAP.
OWASP is an online community, produces freely-available articles, methodologies, documentation, tools, and technologies in the field of web application security.
Open Vulnerability and Assessment Language is a community effort to standardize how to assess and report upon the machine state of computer systems. OVAL includes a language to encode system details, and community repositories of content. Tools and services that use OVAL provide enterprises with accurate, consistent, and actionable information to improve their security.
Terrascan is a static code analyzer for Infrastructure as Code to mitigate risk before provisioning cloud native infrastructure.
Sliver is an open source cross-platform adversary emulation/red team framework, it can be used by organizations of all sizes to perform security testing. Sliver's implants support C2 over Mutual TLS (mTLS), WireGuard, HTTP(S), and DNS and are dynamically compiled with per-binary asymmetric encryption keys.
Payloads All The Things is a list of useful payloads and bypass for Web Application Security and Pentest/CTF.
TheHive is a scalable 3-in-1 open source and free Security Incident Response Platform designed to make life easier for SOCs, CSIRTs, CERTs and any information security practitioner dealing with security incidents that need to be investigated and acted upon swiftly. It is the perfect companion to MISP.
Cortex is a Powerful Observable Analysis and Active Response Engine. This solves a common problem frequently encountered by SOCs, CSIRTs and security researchers in the course of threat intelligence, digital forensics and incident response.
MITRE ATT&CK® is a global knowledge base of adversary tactics and techniques based on real-world security observations. It is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community.
CALDERA™ is a cyber security platform designed to easily automate adversary emulation, assist manual red-teams, and automate incident response.
Atomic Red Team™ is a library of tests mapped to the MITRE ATT&CK® framework. Security teams can use Atomic Red Team to quickly, portably, and reproducibly test their environments.
OpenCTI is an open source platform allowing organizations to manage their cyber threat intelligence knowledge and observables. It has been created in order to structure, store, organize and visualize technical and non-technical information about cyber threats.
Amass is an OWASP Project that performs network mapping of attack surfaces and external asset discovery using open source information gathering and active reconnaissance techniques.
LGTM is a tol that finds and prevents zero-days and other critical bugs, with customizable alerts and automated code review.
Semgrep is a code scanning at ludicrous speed. It find bugs and reachable dependency vulnerabilities.
Socket Security is a tool that protects your app from malicious open source dependencies.
Snyk is a tool that find, fix (and prevent!) known vulnerabilities in your code.
GitProtect.io is a free Backup for GitHub that does automatic, daily repo and metadata backup - no maintenance needed: fast restore, DR, AWS, and S3 cloud storage support.
Mend Bolt isa that detects open source vulnerabilities in real time with suggested fixes for quick remediation.
Rewind Backups for GitHub (Formerly BackHub) is a tool that does daily, automatic backups of your repos & metadata. Restore your backups with metadata in seconds + Sync to your S3 or Azure.
Renovate is a tool that keeps dependencies up-to-date with automated Pull Requests.
GuardRails provides continuous security feedback for modern development teams.
Dnsmasq is a tool that provides network infrastructure for small networks: DNS, DHCP, router advertisement and network boot. It is designed to be lightweight and have a small footprint, suitable for resource constrained routers and firewalls. It has also been widely used for tethering on smartphones and portable hotspots, and to support virtual networking in virtualisation frameworks. Supported platforms include Linux (with glibc and uclibc), Android, BSD, and MacOS.
IVRE is a network recon framework. That let's you build your own, self-hosted and fully-controlled alternatives to Shodan, ZoomEye, Censys, and GreyNoise. IVRE can run your Passive DNS service, collect and analyse network intelligence from your sensors, and much more.
MISP is an open source software solution for collecting, storing, distributing and sharing cyber security indicators and threats about cyber security incidents analysis and malware analysis. MISP is designed by and for incident analysts, security and ICT professionals or malware reversers to support their day-to-day operations to share structured information efficiently.
Rapid7 Nexpose is a vulnerability scanner which aims to support the entire vulnerability management lifecycle, including discovery, detection, verification, risk classification, impact analysis, reporting and mitigation. It integrates with Rapid7's Metasploit for vulnerability exploitation.
Nikto is an Open Source web server scanner which performs comprehensive tests against web servers for multiple items, including over 6400 potentially dangerous files/CGIs, checks for outdated versions of over 1200 servers, and version specific problems on over 270 servers.
Scapy is a powerful interactive packet manipulation tool, packet generator, network scanner, network discovery tool, and packet sniffer.
OSSEC HIDS(Host Intrusion Detection System) is an open source security tool that performs log analysis, integrity checking, rootkit detection, time-based alerting and active response. In addition to its IDS functionality, it is commonly used as a SEM/SIM solution.
OpenMPTCProuter is a tool that uses MultiPath TCP (MPTCP) to really aggregate multiple Internet connections and OpenWrt.
Scrummage is an OSINT tool that centralises search functionality from a bounty of powerful, publicly-available, third-party, OSINT websites.
Sandstorm is a self-hostable web productivity suite. It's implemented as a security-hardened web app package manager.
mkcert is a simple zero-config tool to make locally trusted development certificates with any names you'd like.
Age is a simple, modern and secure encryption tool (and Go library) with small explicit keys, no config options, and UNIX-style composability.
SOPS is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.
Universal Radio Hacker (URH) is a complete suite for wireless protocol investigation with native support for many common Software Defined Radios. URH allows easy demodulation of signals combined with an automatic detection of modulation parameters making it a breeze to identify the bits and bytes that fly over the air.
Cloudflare Tunnel client is a tunneling daemon that proxies traffic from the Cloudflare network to your origins. This daemon sits between Cloudflare network and your origin (e.g. a webserver). Cloudflare attracts client requests and sends them to you via this daemon, without requiring you to poke holes on your firewall --- your origin can remain as closed as possible.
Cloudflare WARP client is a tool that allows you to protect corporate devices by securely and privately sending traffic from those devices to Cloudflare’s edge, where Cloudflare Gateway can apply advanced web filtering. It also makes it possible to apply advanced Zero Trust policies that check for a device’s health before it connects to corporate applications.
Bettercap is a powerful, easily extensible and portable framework written in Go which aims to offer to security researchers, red teamers and reverse engineers an easy to use, all-in-one solution with all the features they might possibly need for performing reconnaissance and attacking WiFi networks, Bluetooth Low Energy devices, wireless HID devices and Ethernet networks.
Wifiphisher is a rogue Access Point framework for conducting red team engagements or Wi-Fi security testing. Using Wifiphisher, penetration testers can easily achieve a man-in-the-middle position against wireless clients by performing targeted Wi-Fi association attacks.
Attack Surface Analyzer is a Microsoft developed open source security tool that analyzes the attack surface of a target system and reports on potential security vulnerabilities introduced during the installation of software or system misconfiguration.
CrowdSec is an open-source and collaborative security stack leveraging the crowd power to generate a global CTI database to protect the user network. It will analyze behaviors, respond to attacks & share signals across the community.
Crowdsec Firewall Bouncer is a tool that will fetch new and old decisions from a CrowdSec API to add them in a blocklist used by supported firewalls.
Supported firewalls:
- iptables (IPv4 heavy_check_mark / IPv6 heavy_check_mark )
- nftables (IPv4 heavy_check_mark / IPv6 heavy_check_mark )
- ipset only (IPv4 heavy_check_mark / IPv6 heavy_check_mark )
- pf (IPV4 heavy_check_mark / IPV6 heavy_check_mark )
Pulse is a powerful logging system for Apple Platforms builtin in SwiftUI. It allows you to record and inspect logs and URLSession network requests right from your iOS app. Shared logs and view them in Pulse Pro or use remote logging to see them in real-time. Logs are stored locally and never leave your devices.
tshark.dev is your complete guide to working with packet captures on the command-line.
Nebula is a scalable overlay networking tool with a focus on performance, simplicity and security. It lets you seamlessly connect computers anywhere in the world. Nebula is portable, and runs on Linux, OSX, Windows, iOS, and Android. It can be used to connect a small number of computers, but is also able to connect tens of thousands of computers.
Parca is a tool for continuous profiling for analysis of CPU and memory usage, down to the line number and throughout time. Saving infrastructure cost, improving performance, and increasing reliability.
DeepFlow is a highly automated observability platform for cloud-native developers. Using new technologies such as eBPF, WASM, and OpenTelemetry, DeepFlow innovatively implements core mechanisms such as AutoTracing, AutoMetrics, AutoTagging, and SmartEncoding, which greatly avoids code instrumentation and significantly reduces the resource overhead of back-end data warehouses.
Hetty is an HTTP toolkit for security research. It aims to become an open source alternative to commercial software like Burp Suite Pro, with powerful features tailored to the needs of the infosec and bug bounty community.
LGTM is a tol that finds and prevents zero-days and other critical bugs, with customizable alerts and automated code review.
Intel Owl is an Open Source Intelligence, or OSINT solution to get threat intelligence data about a specific file, an IP or a domain from a single API at scale. It integrates a number of analyzers available online and a lot of cutting-edge malware analysis tools.
Deepfence ThreatMapper is a runtime tool that hunts for vulnerabilities in your cloud native production platforms(Linux, K8s, AWS Fargate and more.), and ranks these vulnerabilities based on their risk-of-exploit.
Dockle is a Container Image Linter for Security and helping build the Best-Practice Docker Image.
SpiceDB is an open source database system for managing security-critical application permissions inspired by Google's Zanzibar paper.
Trivy is a comprehensive security scanner for vulnerabilities in container images, file systems, and Git repositories, as well as for configuration issues and hard-coded secrets.
Lynis is a security auditing tool for Linux, macOS, and UNIX-based systems. Assists with compliance testing (HIPAA/ISO27001/PCI DSS) and system hardening. Agentless, and installation optional.
RustScan is a Modern Port Scanner.
gosec is a Golang Security Checker that inspects source code for security problems by scanning the Go AST.
Prowler is an Open Source security tool to perform AWS security best practices assessments, audits, incident response, continuous monitoring, hardening and forensics readiness. It contains more than 240 controls covering CIS, PCI-DSS, ISO27001, GDPR, HIPAA, FFIEC, SOC2, AWS FTR, ENS and custom security frameworks.
eNgine is an automated reconnaissance framework for web applications with a focus on highly configurable streamlined recon process via Engines, recon data correlation and organization, continuous monitoring, backed by a database, and simple yet intuitive UI.
Osmedeus is a Workflow Engine for Offensive Security. It was designed to build a foundation with the capability and flexibility that allows you to build your own reconnaissance system and run it on a large number of targets.
IDA Pro(Interactive DisAssembler Professional) is a programmable and multi-processor disassembler combined with a local/remote debugger and along with a complete plugin programming environment. It's a great tool for testing and discovering security vulnerabilities.
Ghidra is a software reverse engineering (SRE) framework developed by NSA's Research Directorate for NSA's cybersecurity mission. It helps analyze any malicious code and malware like viruses, and can give cybersecurity professionals a better understanding of potential vulnerabilities in their networks and systems.
DataWave is an ingest/query framework that leverages Apache Accumulo to provide fast, secure data access.
Emissary is a P2P based data-driven workflow engine that runs in a heterogeneous possibly widely dispersed, multi-tiered P2P network of compute resources. Workflow itineraries are not pre-planned as in conventional workflow engines, but are discovered as more information is discovered about the data.
MADCert is a cross-platform tool that consists of a certificate generator, a file system certificate manager, and a command line interface for the purposes of testing.
BLESS(Bastion's Lambda Ephemeral SSH Service) is an SSH Certificate Authority that runs as an AWS Lambda function and is used to sign SSH public keys.
Zuul is an L7 application gateway that provides capabilities for dynamic routing, monitoring, resiliency, security, and more.
Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures. It is fully integrated with Spinnaker, the continuous delivery platform. Chaos Monkey will work with any backend that Spinnaker supports (AWS, Google Compute Engine, Azure, Kubernetes, Cloud Foundry).
Priam is a tool/process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra.
Vector is an on-host performance monitoring framework which exposes hand picked high resolution metrics to every engineer’s browser.
Control Groups(Cgroups) is a Linux kernel feature that allows you to allocate resources such as CPU time, system memory, network bandwidth, or any combination of these resources for user-defined groups of tasks (processes) running on a system.
Libgcrypt is a general purpose cryptographic library originally based on code from GnuPG.
Aircrack-ng is a network software suite consisting of a detector, packet sniffer, WEP and WPA/WPA2-PSK cracker and analysis tool for 802.11 wireless LANs. It works with any wireless network interface controller whose driver supports raw monitoring mode and can sniff 802.11a, 802.11b and 802.11g traffic.
Burp Suite is a leading range of cybersecurity tools.
Cilium uses eBPF to accelerate getting data in and out of L7 proxies such as Envoy, enabling efficient visibility into API protocols like HTTP, gRPC, and Kafka.
Hubble is a Network, Service & Security Observability for Kubernetes using eBPF.
Istio is an open platform to connect, manage, and secure microservices. Istio's control plane provides an abstraction layer over the underlying cluster management platform, such as Kubernetes and Mesos.
Certgen is a convenience tool to generate and store certificates for Hubble Relay mTLS.
syzkaller is an unsupervised, coverage-guided kernel fuzzer.
SchedViz is a tool for gathering and visualizing kernel scheduling traces on Linux machines.
oss-fuzz aims to make common open source software more secure and stable by combining modern fuzzing techniques with scalable, distributed execution.
OSSEC is a free, open-source host-based intrusion detection system. It performs log analysis, integrity checking, Windows registry monitoring, rootkit detection, time-based alerting, and active response.
Metasploit Project is a computer security project that provides information about security vulnerabilities and aids in penetration testing and IDS signature development.
Wfuzz was created to facilitate the task in web applications assessments and it is based on a simple concept: it replaces any reference to the FUZZ keyword by the value of a given payload.
Nmap is a security scanner used to discover hosts and services on a computer network, thus building a "map" of the network.
Patchwork is a web-based patch tracking system designed to facilitate the contribution and management of contributions to an open-source project.
pfSense is a free and open source firewall and router that also features unified threat management, load balancing, multi WAN, and more.
Snort is an open-source, free and lightweight network intrusion detection system (NIDS) software for Linux and Windows to detect emerging threats.
Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education.
Tink is a multi-language, cross-platform, open source library that provides cryptographic APIs that are secure, easy to use correctly, and harder to misuse.
Diamond Model of Intrusion Analysis is a model to describe cyber attacks. It contains 4 parts - adversary, infrastructure, capability, and target.
Diamond Model of Intrusion Analysis security model
Cyber Kill Chain framework is part of the Intelligence Driven Defense model for identification and prevention of cyber intrusions activity. The model identifies what the adversaries must complete in order to achieve their objective.
Cyber Kill Chain security Model
MITRE ATT&CK is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. The ATT&CK knowledge base is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community.
MITRE ATT&CK security model
ISO/IEC 27005 InfoSec Risk Management is an international standard published by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) providing good practice guidance on managing risks to information.
ISO 27005 InfoSec Risk Management
A threat intelligence platform is a solution that collects and processes threat data from multiple sources. It provides security teams with detailed information about threats like known malware. The SOAR platform can use the information from the threat intelligence platform to guide the strategy and resolution needed against critical threats.
Anomali ThreatStream is a tool that automates the threat intelligence collection and management lifecycle to speed detection, streamline investigations and increase analyst productivity.
IBM X-Force Exchange is a threat intelligence sharing platform that you can use to research security threats, to aggregate intelligence, and to collaborate with peers. Logged in users have integrated access to all the functionality of the site: searching, commenting, Collections and sharing.
LookingGlass scoutTHREAT is a threat intelligence tol that offers automated ingestion and analysis of structured and unstructured threat intelligence, enabling your analysts to collaboratively develop and refine threat actor models by uncovering adversarial capabilities and motivations, tracking relevant reporting to highlight adversaries attacking sectors you care about, and mapping tactics, techniques, and procedures to personas and threat actors.
Recorded Future Intelligence Cloud is a threat intelligence platform that uniquely combines persistent data collection, large-scale graph analysis, and the analytical acumen of Recorded Future's global research team to provide the most complete coverage of intelligence across adversaries, their infrastructure, and the organizations they target, empowering business and security leaders to act with speed and confidence.
ThreatConnect is a threat intelligence platform that has a vision for security that encompasses the most critical elements - risk, threat, and response.
SOAR solutions work by prioritizing and standardizing incident response activities so that security teams can collaborate on investigating and managing incidents. Workflows that can be handled through automation go through standardized response processes defined in playbooks.
SOAR platforms vary depending on vendor, but all of them should include these key features:
- Orchestration: A SOAR solution can facilitate the connection between security and productivity tools, such as firewalls and intrusion detection tools.
- Automation: A SOAR solution can automate standard cybersecurity workflows, such as the identification of security alerts and possible intrusions.
- Response: A SOAR platform can work with both automated and manual processes to support a timely response to security threats.
- Integration: A SOAR platform can work with a variety of complementary security products to support the organization’s overall security posture.
- Playbooks and automation: SOAR helps security teams use collected data to streamline operations through security automation and the use of playbooks.
- Threat prioritization: SOAR helps security teams prioritize and group alerts for more efficient threat detection and investigation.
- Reporting and analysis: SOAR platforms can generate reports to help security teams identify trends in their organization.
- Security dashboard: SOAR platforms can serve as a central dashboard to help security teams monitor and respond to alerts in a collaborative way.
SOAR Tools
Splunk Phantom is a Security Orchestration, Automation, and Response (SOAR) system. It combines security infrastructure orchestration, playbook automation, and case management capabilities to integrate your team, processes, and tools to help you orchestrate security workflows, automate repetitive security tasks, and quickly respond to threats.
IBM IBM Security QRadar SOAR is a Security Orchestration, Automation, and Response (SOAR) system. It's sdesigned to help your security team respond to cyberthreats with confidence, automate with intelligence and collaborate with consistency. It guides your team in resolving incidents by codifying established incident response processes into dynamic playbooks.
Sumo Logic Cloud SOAR is a Security Operations and Automation Incident Response Platform to facilitate and expedite timely management of Incident Response with a rich library of customizable playbooks for different threats and use cases of incident response scenarios expediting and automating response time to incident response events.
Rapid7 Insightconnect is a SOAR solution that integrates with existing solutions to orchestrate vulnerability management processes from notification to remediation. It automations functionality are managed securely end-to-end. Initially, they are encrypted in-transit via TLS and are encrypted at rest in our systems using a public key that's generated by the Orchestrator and sent to the cloud.
LogRhythm RespondX is a seamlessly integrated security orchestration, automation, and response (SOAR) that enables your team to. effectively collaborate, qualify, and manage incidents with. improved quality and speed.
Exabeam incident responder is a SOAR solution that comes with pre-defined playbooks for common incident types such as malware, phishing, and data exfiltration. These playbooks include actions that can automatically run (e.g. go get reputation data for this IP address) or guide a team member (reset this user’s password).
ServiceNow Security Operations is a security orchestration, automation, and response (SOAR) engine built on the Now Platform. It helps security and IT teams respond faster and more efficiently to incidents and vulnerabilities, Security Operations uses intelligent workflows, automation, and a deep connection with Security Operations and IT to streamline response.
SIRP is a no-code risk-based SOAR platform that was built in response to the real-world needs of our customers. Specifically, the need to base security decisions on something more relevant than generic industry systems. Some of the leading enterprises and MSSPs trust SIRP for their security automation.
Chronicle SOAR is a Security Orchestration Automation and Response (SOAR) solution that enables enterprises and MSSPs to gather data and security alerts from different sources by combining the following:
- Orchestration and automation
- Threat intelligence
- Incident response
Palo Alto Networks Cortex XSOAR is a Security Orchestration Automation and Response (SOAR) solution that comes with prebuilt bundles of integrations, playbooks, dashboards, fields, subscription services and all the dependencies needed to support specific security orchestration use cases.
Fortinet FortiSOAR is a security orchestration, automation and response (SOAR) that provides innovative case management, automation, and orchestration. It pulls together all of an organization's tools, helps unify operations, and reduce alert fatigue, context switching, and the mean time to respond to incidents.
Swimlane SOAR is is a security orchestration, automation and response (SOAR) that helps organizations manage the growing volume of alerts more efficiently by automating time-consuming incident response processes. The solution collects security alert data from previously disparate security platforms and automatically responds to alerts using automated workflows and playbooks. It is also designed to centralize data coming back from investigation tools to allow security teams the ability to view all applicable details right from within Swimlane.
Security information and event management (SIEM) software collects log data from an organization and then uses the log data to identify, categorize and analyze incidents and events.
SIEM software goals:
- Report on security incidents and events. The software can provide reports with event data, such as failed logins and malware activity.
- Send alerts about potential security issues. The software can use set parameters to determine whether an event is a potential security issue.
- An overview of notable events in your environment that represent potential security incidents.
- Details of all notable events identified in your environment, so you can undertake triage.
- A workbook of all open investigations, allowing you to track your progress and activity while investigating multiple security incidents.
- Risk analysis that lets you score systems and users across your network to identify risks.
- Threat intelligence designed to add context to your security incidents and identify known malicious actors in your environment.
- Protocol intelligence using captured packet data to provide network insights that are relevant to your security investigations, allowing you to identify suspicious traffic, DNS activity and email activity.
- User intelligence lets you investigate and monitor the activity of users and assets in your environment.
- Web intelligence to analyze web traffic in your network.
SIEM Data sources include:
- Network devices: Routers, switches, bridges, wireless access points, modems, line drivers, hubs
- Servers: Web, proxy, mail, FTP
- Security devices: IDP/IPS, firewalls, antivirus software, content filter devices, intrusion detection appliances
- Applications: Any software used on any of the above devices
- Cloud and SaaS solutions: Software and services not hosted on-premises
- Remote workforce: All devices and activity related to remote work
Datadog Security Monitoring is a SIE Security information and event management (SIEM) that comes with real-time security monitoring tool , Datadog analyzes and evaluates security and observability data in order to identify threats and reduce risks. Use configurable out-of-the-box rules—mapped to the MITRE ATT&CK™ framework—to track common attacker techniques, such as a VM enumerating all storage buckets in your account.
Logpoint is the only unified SIEM-SOAR solution that collects, analyzes and prioritizes security incidents to help analysts identify and resolve incidents fast and keep businesses safe. With built-in detection, investigation, and response playbooks,
Graylog is a log management to the cloud and aims at SIEM in the midmarket Log management vendor Graylog has released a SaaS version of its enterprise product as well as a new security offering. It provides answers to your team's security, application, and IT infrastructure questions by enabling you to combine, enrich, correlate, query, and visualize all your log data in one place.
Exabeam Fusion is a powerful and advanced cloud-native SIEM and introduces New-Scale SIEM. It unites the combined capabilities of all Exabeam products such as cloud-native data storage, rapid data ingestion, hyper-quick query performance, powerful behavioral analytics, and automation that changes the way analysts do their jobs.
Elastic Security is a platform that unifies SIEM, endpoint security, and cloud security on an open platform, equipping teams to prevent, detect, and respond to threats. It includes a security news feed, host and network data, detections, timelines, cases, and an abstracted view into the administration of the Elastic endpoint configuration.
Splunk Enterprise Security is a SIEM Platofrm solution that brings together visibility, correlation, automated response, and remediation in a single, scalable solution. It reduces the complexity of managing network and security operations to effectively free resources, improve breach detection, and even prevent breaches.
OSSEC The Open-source HIDS Security is a multiplatform, open source and free Host Intrusion Detection System (HIDS). It performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.
LogRhythm NextGen SIEM Platform is a SIEM platform delivers comprehensive security analytics, UEBA, NTA, and SOAR within a single, integrated platform for rapid detection, response, and neutralization of threats.
User and Entity Behavior Analytics (UEBA) is a category of security solutions that use innovative analytics technology, including machine learning and deep learning, to discover abnormal and risky behavior by users, machines and other entities on the corporate network with a Security Incident and Event Management (SIEM) solution.
Gartner defines UEBA solutions across three dimensions:
- Use cases — UEBA solutions provide information on the behavior of users and other entities in the corporate network. They should perform monitoring, detection and alerting of anomalies. And they should be applicable for multiple use cases–unlike specialized tools for employee monitoring, trusted hosts monitoring, fraud, and so on.
- Data sources — UEBA solutions are able to ingest data from a general data repository such as a data lake or data warehouse, or through a SIEM. They shouldn't deploy agents directly in the IT environment to collect the data.
- Analytics — UEBA solutions detect anomalies using a variety of analytics approaches–statistical models, machine learning, rules, threat signatures and more.
There is a close relation between UEBA and SIEM technologies, because UEBA relies on cross-organizational security data to perform its analyses, and this data is typically collected and stored by a SIEM.
A UEBA solution should analyze as many data sources as possible, some example data sources include:
- Authentication systems like Active Directory
- Access systems like VPN and proxies
- Configuration Management Databases
- Human resources data–new employees, departed employees, and any data that provides additional context on users
- Firewall, Intrusion Detection and Prevention Systems (IDPS)
- Anti-malware and antivirus systems
- Endpoint Detection and Response systems
- Network Traffic Analytics
- Threat Intelligence feeds
-
Endpoint Detection and Response (EDR) is a cybersecurity service that's referred to as endpoint detection and threat response (EDTR), is an endpoint security solution that continuously monitors end-user devices to detect and respond to cyber threats like ransomware and malware.
-
Extended detection and response (XDR) is a cybersecurity service that collects threat data from previously siloed security tools across an organization's technology stack for easier and faster investigation, threat hunting, and response. An XDR platform can collect security telemetry from endpoints, cloud workloads, network email, and more.
-
Managed detection and response (MDR) is a cybersecurity service that combines technology and human expertise to perform threat hunting, monitoring, and response.
-
Network Detection and Response (NDR) is a category of cybersecurity technologies that use non-signature-based methods—such as artificial intelligence, machine learning and behavioral analytics—to detect suspicious or malicious activity on the network and respond to cyber threats.
AWS Certified Security - Specialty Certification
Microsoft Certified: Azure Security Engineer Associate
Google Cloud Certified Professional Cloud Security Engineer
The Red Hat Certified Specialist in Security: Linux
Linux Professional Institute LPIC-3 Enterprise Security Certification
Cybersecurity Training and Courses from IBM Skills
Cybersecurity Courses and Certifications by Offensive Security
Citrix Certified Associate – Networking(CCA-N)
Citrix Certified Professional – Virtualization(CCP-V)
Certified Information Security Manager(CISM)
Wireshark Certified Network Analyst (WCNA)
Juniper Networks Certification Program Enterprise (JNCP)
Networking courses and specializations from Coursera
Network & Security Courses from Udemy
Network & Security Courses from edX
Qt Network Authorization is a tool that provides a set of APIs that enable Qt applications to obtain limited access to online accounts and HTTP services without exposing users' passwords.
cURL is a computer software project providing a library and command-line tool for transferring data using various network protocols(HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP or SMTPS). cURL is also used in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, media players and is the Internet transfer engine for thousands of software applications in over ten billion installations.
cURL Fuzzer is a quality assurance testing for the curl project.
DoH is a stand-alone application for DoH (DNS-over-HTTPS) name resolves and lookups.
Authelia is an open-source highly-available authentication server providing single sign-on capability and two-factor authentication to applications running behind NGINX.
nginx(engine x) is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server, originally written by Igor Sysoev.
Proxmox Virtual Environment(VE) is a complete open-source platform for enterprise virtualization. It inlcudes a built-in web interface that you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution.
Wireshark is a very popular network protocol analyzer that is commonly used for network troubleshooting, analysis, and communications protocol development. Learn more about the other useful Wireshark Tools available.
HTTPie is a command-line HTTP client. Its goal is to make CLI interaction with web services as human-friendly as possible. HTTPie is designed for testing, debugging, and generally interacting with APIs & HTTP servers.
HTTPStat is a tool that visualizes curl statistics in a simple layout.
Wuzz is an interactive cli tool for HTTP inspection. It can be used to inspect/modify requests copied from the browser's network inspector with the "copy as cURL" feature.
Websocat is a ommand-line client for WebSockets, like netcat (or curl) for ws:// with advanced socat-like functions.
-
Connection: In networking, a connection refers to pieces of related information that are transferred through a network. This generally infers that a connection is built before the data transfer (by following the procedures laid out in a protocol) and then is deconstructed at the at the end of the data transfer.
-
Packet: A packet is, generally speaking, the most basic unit that is transferred over a network. When communicating over a network, packets are the envelopes that carry your data (in pieces) from one end point to the other.
Packets have a header portion that contains information about the packet including the source and destination, timestamps, network hops. The main portion of a packet contains the actual data being transferred. It is sometimes called the body or the payload.
- Network Interface: A network interface can refer to any kind of software interface to networking hardware. For instance, if you have two network cards in your computer, you can control and configure each network interface associated with them individually.
A network interface may be associated with a physical device, or it may be a representation of a virtual interface. The "loop-back" device, which is a virtual interface to the local machine, is an example of this.
-
LAN: LAN stands for "local area network". It refers to a network or a portion of a network that is not publicly accessible to the greater internet. A home or office network is an example of a LAN.
-
WAN: WAN stands for "wide area network". It means a network that is much more extensive than a LAN. While WAN is the relevant term to use to describe large, dispersed networks in general, it is usually meant to mean the internet, as a whole. If an interface is connected to the WAN, it is generally assumed that it is reachable through the internet.
-
Protocol: A protocol is a set of rules and standards that basically define a language that devices can use to communicate. There are a great number of protocols in use extensively in networking, and they are often implemented in different layers.
Some low level protocols are TCP, UDP, IP, and ICMP. Some familiar examples of application layer protocols, built on these lower protocols, are HTTP (for accessing web content), SSH, TLS/SSL, and FTP.
-
Port: A port is an address on a single machine that can be tied to a specific piece of software. It is not a physical interface or location, but it allows your server to be able to communicate using more than one application.
-
Firewall: A firewall is a program that decides whether traffic coming into a server or going out should be allowed. A firewall usually works by creating rules for which type of traffic is acceptable on which ports. Generally, firewalls block ports that are not used by a specific application on a server.
-
NAT: Network address translation is a way to translate requests that are incoming into a routing server to the relevant devices or servers that it knows about in the LAN. This is usually implemented in physical LANs as a way to route requests through one IP address to the necessary backend servers.
-
VPN: Virtual private network is a means of connecting separate LANs through the internet, while maintaining privacy. This is used as a means of connecting remote systems as if they were on a local network, often for security reasons.
While networking is often discussed in terms of topology in a horizontal way, between hosts, its implementation is layered in a vertical fashion throughout a computer or network. This means is that there are multiple technologies and protocols that are built on top of each other in order for communication to function more easily. Each successive, higher layer abstracts the raw data a little bit more, and makes it simpler to use for applications and users. It also allows you to leverage lower layers in new ways without having to invest the time and energy to develop the protocols and applications that handle those types of traffic.
As data is sent out of one machine, it begins at the top of the stack and filters downwards. At the lowest level, actual transmission to another machine takes place. At this point, the data travels back up through the layers of the other computer. Each layer has the ability to add its own "wrapper" around the data that it receives from the adjacent layer, which will help the layers that come after decide what to do with the data when it is passed off.
One method of talking about the different layers of network communication is the OSI model. OSI stands for Open Systems Interconnect.This model defines seven separate layers. The layers in this model are:
-
Application: The application layer is the layer that the users and user-applications most often interact with. Network communication is discussed in terms of availability of resources, partners to communicate with, and data synchronization.
-
Presentation: The presentation layer is responsible for mapping resources and creating context. It is used to translate lower level networking data into data that applications expect to see.
-
Session: The session layer is a connection handler. It creates, maintains, and destroys connections between nodes in a persistent way.
-
Transport: The transport layer is responsible for handing the layers above it a reliable connection. In this context, reliable refers to the ability to verify that a piece of data was received intact at the other end of the connection. This layer can resend information that has been dropped or corrupted and can acknowledge the receipt of data to remote computers.
-
Network: The network layer is used to route data between different nodes on the network. It uses addresses to be able to tell which computer to send information to. This layer can also break apart larger messages into smaller chunks to be reassembled on the opposite end.
-
Data Link: This layer is implemented as a method of establishing and maintaining reliable links between different nodes or devices on a network using existing physical connections.
-
Physical: The physical layer is responsible for handling the actual physical devices that are used to make a connection. This layer involves the bare software that manages physical connections as well as the hardware itself (like Ethernet).
The TCP/IP model, more commonly known as the Internet protocol suite, is another layering model that is simpler and has been widely adopted.It defines the four separate layers, some of which overlap with the OSI model:
-
Application: In this model, the application layer is responsible for creating and transmitting user data between applications. The applications can be on remote systems, and should appear to operate as if locally to the end user. The communication takes place between peers network.
-
Transport: The transport layer is responsible for communication between processes. This level of networking utilizes ports to address different services. It can build up unreliable or reliable connections depending on the type of protocol used.
-
Internet: The internet layer is used to transport data from node to node in a network. This layer is aware of the endpoints of the connections, but does not worry about the actual connection needed to get from one place to another. IP addresses are defined in this layer as a way of reaching remote systems in an addressable manner.
-
Link: The link layer implements the actual topology of the local network that allows the internet layer to present an addressable interface. It establishes connections between neighboring nodes to send data.
Interfaces are networking communication points for your computer. Each interface is associated with a physical or virtual networking device. Typically, your server will have one configurable network interface for each Ethernet or wireless internet card you have. In addition, it will define a virtual network interface called the "loopback" or localhost interface. This is used as an interface to connect applications and processes on a single computer to other applications and processes. You can see this referenced as the "lo" interface in many tools.
Networking works by piggybacks on a number of different protocols on top of each other. In this way, one piece of data can be transmitted using multiple protocols encapsulated within one another.
Media Access Control(MAC) is a communications protocol that is used to distinguish specific devices. Each device is supposed to get a unique MAC address during the manufacturing process that differentiates it from every other device on the internet. Addressing hardware by the MAC address allows you to reference a device by a unique value even when the software on top may change the name for that specific device during operation. Media access control is one of the only protocols from the link layer that you are likely to interact with on a regular basis.
The IP protocol is one of the fundamental protocols that allow the internet to work. IP addresses are unique on each network and they allow machines to address each other across a network. It is implemented on the internet layer in the IP/TCP model. Networks can be linked together, but traffic must be routed when crossing network boundaries. This protocol assumes an unreliable network and multiple paths to the same destination that it can dynamically change between. There are a number of different implementations of the protocol. The most common implementation today is IPv4, although IPv6 is growing in popularity as an alternative due to the scarcity of IPv4 addresses available and improvements in the protocols capabilities.
ICMP: internet control message protocol is used to send messages between devices to indicate the availability or error conditions. These packets are used in a variety of network diagnostic tools, such as ping and traceroute. Usually ICMP packets are transmitted when a packet of a different kind meets some kind of a problem. Basically, they are used as a feedback mechanism for network communications.
TCP: Transmission control protocol is implemented in the transport layer of the IP/TCP model and is used to establish reliable connections. TCP is one of the protocols that encapsulates data into packets. It then transfers these to the remote end of the connection using the methods available on the lower layers. On the other end, it can check for errors, request certain pieces to be resent, and reassemble the information into one logical piece to send to the application layer. The protocol builds up a connection prior to data transfer using a system called a three-way handshake. This is a way for the two ends of the communication to acknowledge the request and agree upon a method of ensuring data reliability. After the data has been sent, the connection is torn down using a similar four-way handshake. TCP is the protocol of choice for many of the most popular uses for the internet, including WWW, FTP, SSH, and email. It is safe to say that the internet we know today would not be here without TCP.
UDP: User datagram protocol is a popular companion protocol to TCP and is also implemented in the transport layer. The fundamental difference between UDP and TCP is that UDP offers unreliable data transfer. It does not verify that data has been received on the other end of the connection. This might sound like a bad thing, and for many purposes, it is. However, it is also extremely important for some functions. It’s not required to wait for confirmation that the data was received and forced to resend data, UDP is much faster than TCP. It does not establish a connection with the remote host, it simply fires off the data to that host and doesn't care if it is accepted or not. Since UDP is a simple transaction, it is useful for simple communications like querying for network resources. It also doesn't maintain a state, which makes it great for transmitting data from one machine to many real-time clients. This makes it ideal for VOIP, games, and other applications that cannot afford delays.
HTTP: Hypertext transfer protocol is a protocol defined in the application layer that forms the basis for communication on the web. HTTP defines a number of functions that tell the remote system what you are requesting. For instance, GET, POST, and DELETE all interact with the requested data in a different way.
FTP: File transfer protocol is in the application layer and provides a way of transferring complete files from one host to another. It is inherently insecure, so it is not recommended for any externally facing network unless it is implemented as a public, download-only resource.
DNS: Domain name system is an application layer protocol used to provide a human-friendly naming mechanism for internet resources. It is what ties a domain name to an IP address and allows you to access sites by name in your browser.
SSH: Secure shell is an encrypted protocol implemented in the application layer that can be used to communicate with a remote server in a secure way. Many additional technologies are built around this protocol because of its end-to-end encryption and ubiquity. There are many other protocols that we haven't covered that are equally important. However, this should give you a good overview of some of the fundamental technologies that make the internet and networking possible.
JSON Web Token (JWT) is a compact URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS).
OAuth 2.0 is an open source authorization framework that enables applications to obtain limited access to user accounts on an HTTP service, such as Amazon, Google, Facebook, Microsoft, Twitter GitHub, and DigitalOcean. It works by delegating user authentication to the service that hosts the user account, and authorizing third-party applications to access the user account.
- If would you like to contribute to this guide simply make a Pull Request.
Distributed under the Creative Commons Attribution 4.0 International (CC BY 4.0) Public License.