Grinder Framework

🔎 Internet-connected Devices Census Python Framework

Screenshot

The basic CLI interface of the Grinder Framework

Description
Slides
Grinder Workflow
Grinder Map
- Screenshots
- Description
Requirements
- Legend
- 📌 Basic
- 📌 Accounts
- Additional scanning
- TLS configuration
Current Features
📌 Grinder Installing
- Setup and Configure Environment
- Running Grinder Map Server Locally
Building and Running in Docker
- Description
- Services and Images
- Provided Scripts and Preparations
- Environment
- 📌 Building
- 📌 Running
Tests
CLI Interface
- Help on Command Line Arguments
Wiki
📌 Usage Examples
📌 Add Your Own Queries

Description

The Grinder framework was created to automatically enumerate and fingerprint different hosts on the Internet using various back-end systems: search engines (such as Shodan or Censys) for discovering hosts and NMAP engine for fingerprinting and specific checks. Also, Grinder supports Vulners API to get information about available public exploits and vulnerabilities, documents related to found vulnerabilities and other features.

The Grinder framework can be used in many different areas of research, as a connected Python module in your project or as an independent ready-to-use from the box tool.

Slides

One framework to rule them all: a framework for Internet-connected device census. PHDays 2019. (Talk Page, Slides)
One Framework to rule them all: A framework for Internet-connected Device Census. OFFZONE 2019. (Talk Page, Slides)

Grinder Workflow

Grinder Map

Screenshots

The Grinder Framework can easily build an interactive map with found hosts in your browser

Also, the Grinder Framework can show you some basic information

...And some additional information

Description

To visualize gained data, the Grinder Framework provides an interactive world map with all results. Grinder map back-end that was written in Flask supports additional REST API methods to get more information about all scanned hosts or some particular host from the map, also it is possible to show some additional information about host interactively from the map.

For example, the hosts will be automatically checked for availability with ping from back-end, also for every host many additional features are available: current host can be directly opened in Shodan, Censys, and ZoomEye web interfaces, the host can be shown on Google Maps with all available information about geolocation. Also, it is possible to make an IP lookup or open raw information in JSON directly in a browser or from your application with provided API methods.

Requirements

Legend

❗ required
➕ not required to run (or required only for additional modules)

Basic

❗ Python 3.6+
❗ python3-tk library
❗ FreeType library (Python 3.8+)

Accounts

❗ Shodan and Censys accounts
Required to collect hosts, both free and full accounts are suitable. Also, it's possible to use only one account (Censys or Shodan, Shodan is preferable).
➕ Vulners account
Required to make additional reports on vulnerabilities and exploits. If this feature is not required for you, you can use Grinder without Vulners account.

Additional scanning

➕ Nmap Security Scanner 7.60+
Version 7.60 and newer has been tested with currently used in Grinder scripts (ssl-cert.nse, vulners.nse, etc.).

TLS configuration

➕ Java 8
Required to build TLS-Attacker and TLS-Scanner.
➕ TLS-Attacker 3.0
Required only for TLS scanning.
➕ TLS-Scanner 2.9
Required only for TLS scanning.

Current Features

Already Implemented

🔍 Collecting hosts and additional information using Shodan and Censys search engines
🚀 Scanning ports and services with boosted multi-processed Nmap Scanner wrapper
💉 Scanning vulnerabilities and additional information about them with Vulners database and Shodan CVEs database
📝 Retrieving information about SSL certificates
🔑 Scanning for SSL/TLS configuration and supported cipher suites
🔑 Scanning for SSL/TLS bugs, vulnerabilities and attacks
🌏 Building an interactive map with information about the hosts found
📊 Creating plots and tables based on the collected results
🔧 Custom scanning scripts support (in LUA or Python3)
📉 Confidence filtering system support
📈 Special vendors scanning and filtering support
💡 Searching for documents, security bulletins, public exploits and many more things based on detected by Grinder vulnerabilities and software

Additional Modules

🚀 Note #1: You can run multiple Python scripts simultaneously per multiple hosts, so you can build your own chain of scripts and checks to get the most information from your hosts. Feel free to add your modules with PR or give us an idea with feature issue.

🚧 Note #2: Multiple NSE scripts running task is still in WIP status. So, NSE scripts will be ran consistently, one after one. New NSE script engine for Grinder is comming up, stay tuned.

DICOM Patient Info Getter

Location: py_scripts/dicom_getter/dicom_getter.py
Description: This module allows you to grab different patient information (including files) from medical servers

HTTP Raw Response Grabber

Location: py_scripts/http_response_grabber/http_response_grabber.py
Description: This module allows you to grab the HTTP response (headers + body) in decoded raw-bytes format

HTTP Status

Location: py_scripts/http_status/http_status.py
Description: This module allows you to check the HTTP status of the resource

SCP (SSH) Grabber

Location: py_scripts/scp_grabber/scp_grabber.py
Description: This module allows you to grab and download files from different SSH servers (you need to define login/password in the module)

Sleep

Location: py_scripts/sleep/sleep.py
Description: This is just a sleep module for test

SNMP Walker (easysnmp implementation)

Location: py_scripts/snmp_walker_easysnmp/snmp_walker_easysnmp.py
Description: This module allows you to collect information from SNMP servers

SNMP Walker (system call implementation)

Location: py_scripts/snmp_walker_os/snmp_walker_os.py
Description: The same as previous, but in another way

Tensor Serving Server Checker

Location: py_scripts/tensor_serving_checker/tensor_serving_checker.py
Description: This module allows you to check, if the resource is TensorFlow Serving Server or not

Test

Location: py_scripts/test/test.py
Description: Just an empty test script to check the functionality

Development and Future Updates

Grinder Development Project

🚧 Note: The Grinder framework is still in progress and got features to improve, so all the tasks and other features will always be described in this project. If you got some awesome ideas or any other interesting things for Grinder, you can always open a pull request or some issues in this repository.

Grinder Installing

💡 Note #1: If you are familiar with pipenv package manager, all steps related to virtualenv can be replaced with pipenv sync command.

💡 Note #2: If you are familiar with Docker and docker-compose, you can build framework with Docker, see Building and Running in Docker

💡 Note #3: Only for macOS version 10.13 and higher, Python version 3.7.* and lower: to use additional scripts, which include other network libraries, you must additionally set the environment variable before starting - in your shell configuration file (~/.bashrc, ~/.zshrc) or before starting the framework:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
./grinder.py -h

Setup and Configure Environment

Install Nmap Security Scanner if not installed.
Install python3-tk package if not installed (Linux only)

sudo apt-get install python3-tk

Install FreeType library (Python 3.8+)

sudo apt-get install build-essential libfreetype6-dev python-dev

Install virtualenv if not installed

sudo pip3 install virtualenv

pip3 install --upgrade virtualenv

Clone the repository

git clone https://github.com/sdnewhop/grinder

Clone and install TLS-Attacker 3.0 (if you want to use TLS scanning features with Grinder).
Clone TLS-Scanner 2.9 in directory with Grinder and install it (if you want to use TLS scanning features with Grinder.
Create virtual environment

cd grinder
python3 -m venv grindervenv
source grindervenv/bin/activate

Check if virtual environment successfully loaded

which python3
which pip3

Install project requirements in virtual environment

pip3 install -r requirements.txt

(MacOS version 10.13 and higher, Python version 3.7.* and lower)
Export additional environment variable to run different Python3 scripts

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

Run the framework

./grinder.py -h

Set your Shodan, Censys and Vulners keys via a command line arguments on every run

./grinder.py -sk YOUR_SHODAN_KEY -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -vk YOUR_VULNERS_KEY

or via an environment variable permanently

export SHODAN_API_KEY=YOUR_SHODAN_API_KEY_HERE
export CENSYS_API_ID=YOUR_CENSYS_API_ID_HERE
export CENSYS_API_SECRET=YOUR_CENSYS_API_SECRET_HERE
export VULNERS_API_KEY=YOUR_VULNERS_API_KEY_HERE

Deactivate virtual environment after use and restore default python interpreter

deactivate

Running Grinder Map Server Locally

First, complete all steps from the Setup and Configure Environment/Grinder Installing section above.
After the scan is completed, go to the "map" folder

cd map/

Run Flask (use --help key for more information)

flask run

Open in your browser

http://localhost:5000/

Also, Grinder map server provides simple API methods such as /api/viewall/, /api/viewraw/<host_id>, you can learn more from list of application routes

flask routes

Building and Running in Docker

Description

The Grinder Framework also supports building as a set of Docker images: one for the framework itself, one for the map and the last one temporary image for the TLS-Scanner. You can use them separately from each other, or all at once - all of these services are linked via docker-compose.yml file.

Services and Images

tls-scanner as grinder/tls-scanner:2.9 image
grinder as grinder/grinder-framework:1.0 image
map as grinder/grinder-map:1.0 image

Provided Scripts and Preparations

To make the building process simple and convenient, a set of scripts provided in the repository to simplify routine actions.

docker_build.sh script simply runs docker-compose build, so you can do it by yourself.
docker_run.sh prepare all the required files and directories to link your host files with the ones in containers.

Environment

First of all, you need to provide all the required environment variables via the .env or docker-compose file itself.

Required variables are (docker-compose):

SHODAN_API_KEY: ${SHODAN_API_KEY:-YOUR_SHODAN_API_KEY_HERE}
CENSYS_API_ID: ${CENSYS_API_ID:-YOUR_CENSYS_API_ID_HERE}
CENSYS_API_SECRET: ${CENSYS_API_SECRET:-YOUR_CENSYS_API_SECRET_HERE}
VULNERS_API_KEY: ${VULNERS_API_KEY:-YOUR_VULNERS_API_KEY_HERE}

Or in the same way for the .env file:

SHODAN_API_KEY=...
CENSYS_API_ID=...
CENSYS_API_SECRET=...
VULNERS_API_KEY=...

Building

To build the Grinder Framework as an Alpine-based set of Docker images you can use the script docker_build.sh:

chmod +x docker_build.sh
./docker_build.sh

Running

To run the Grinder Framework with all included services (Map, Framework and TLS-Scanner) you can use the script docker_run.sh:

chmod +x docker_run.sh
./docker_run.sh

After that, you can open map at http://localhost:5000/, shell of the Grinder Framework will be automatically open inside the container. All the data will be saved on your hosts directly via Docker volumes and mappings.

Tests

To run basic tests for different scanning and analytics modules, you need to change directory to tests/:

cd tests/

And run basic tests with the next command - please, pay attention that you need to provide API keys for some modules (like Shodan, Censys) because tests are implemented to check all real functional features of these search engines in Grinder modules and wrappers:

pytest --shodan_key SHODAN_API_KEY --censys_id CENSYS_ID_KEY --censys_secret CENSYS_SECRET_KEY

🚧 Note: tests are still WIP, so please, feel free to create issues If you encounter any problems with it. Currently tests provided for some basic modules and features (Censys, Shodan, Filemanager, Database).

CLI Interface

Help on Command Line Arguments

  -h, --help            show this help message and exit
  -r, --run             Run scanning
  -u, --update-markers  Update map markers
  -q QUERIES_FILE, --queries-file QUERIES_FILE
                        JSON File with Shodan queries
  -sk SHODAN_KEY, --shodan-key SHODAN_KEY
                        Shodan API key
  -vk VULNERS_KEY, --vulners-key VULNERS_KEY
                        Vulners API key
  -cu, --count-unique   Count unique entities
  -cp, --create-plots   Create graphic plots
  -ci CENSYS_ID, --censys-id CENSYS_ID
                        Censys API ID key
  -cs CENSYS_SECRET, --censys-secret CENSYS_SECRET
                        Censys API SECRET key
  -cm CENSYS_MAX, --censys-max CENSYS_MAX
                        Censys default maximum results quantity
  -sm SHODAN_MAX, --shodan-max SHODAN_MAX
                        Shodan default maximum results quantity.
  -nm, --nmap-scan      Initiate Nmap scanning
  -nw NMAP_WORKERS, --nmap-workers NMAP_WORKERS
                        Number of Nmap workers to scan
  -vs, --vulners-scan   Initiate Vulners API scanning
  -vw VULNERS_WORKERS, --vulners-workers VULNERS_WORKERS
                        Number of Vulners workers to scan
  -ht HOST_TIMEOUT, --host-timeout HOST_TIMEOUT
                        Default host timeout in seconds for scanning with
                        Vulners and Nmap core
  -tp TOP_PORTS, --top-ports TOP_PORTS
                        Quantity of popular top-ports in addition to Shodan
                        ports
  -sc, --script-check   Initiate custom scripts additional checks
  -scw SCRIPT_WORKERS, --script-workers SCRIPT_WORKERS
                        Number of script checking workers
  -scm, --script-mute   Suppress scripts output (stdout, stderr)
  -vc VENDOR_CONFIDENCE, --vendor-confidence VENDOR_CONFIDENCE
                        Set confidence level for vendors
  -qc QUERY_CONFIDENCE, --query-confidence QUERY_CONFIDENCE
                        Set confidence level for queries
  -v [VENDORS [VENDORS ...]], --vendors [VENDORS [VENDORS ...]]
                        Set list of vendors to search from queries file
  -ml MAX_LIMIT, --max-limit MAX_LIMIT
                        Maximum number of unique entities in plots and results
  -d, --debug           Show more information
  -ts, --tls-scan       Check for possible TLS attacks and bugs (require TLS-
                        Scanner)
  -tsp TLS_SCAN_PATH, --tls-scan-path TLS_SCAN_PATH
                        Path to TLS-Scanner.jar (if TLS-Scanner directory not
                        in Grinder root, else not required)
  -vr, --vulners-report
                        Make additional vulners reports
  -ni, --not-incremental
                        Turn off incrememental scan - make clean scan (without
                        previous results)

Wiki

Additional extended documentation for the framework is available on the repository wiki, including additional information about flags, the internal structure of the framework, queries, and more.

Usage Examples

Show Help

Description

Show all available CLI keys for the Grinder Framework

Example

./grinder.py -h

Basic Enumeration

Description

Run the most basic enumeration with Shodan and Censys engines without map markers and plots updating (results will be saved in database and output JSON)

Template

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -r

Example

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q queries/servers.json -r

Enumeration with Limited Results

Description

Run an enumeration where maximum Censys results is 555 hosts per query and maximum Shodan results is 1337 hosts per query

Template

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -cm CENSYS_RESULTS_LIMIT -sm SHODAN_RESULTS_LIMIT -r

Example

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q queries/servers.json -cm 555 -sm 1337 -r

Enumeration with Nmap Scanning

Description

Run an enumeration with 10 Nmap Network Scanner scanning workers

Template

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -nm -nw NUMBER_OF_NMAP_WORKERS -r

Example

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q queries/servers.json -nm -nw 10 -r

Enumeration with Additional Analytics, Map and Plots

Description

Run an enumeration, count unique entities, create plots and update Grinder Map markers data

Template

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -cu -cp -u -r

Example

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q queries/servers.json -cu -cp -u -r

Enumeration with Analytics from Vulners

Description

Run an enumeration with Vulners scanning and Vulners additional reports, quantity of Vulners workers is equal to 10

Template

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -vs -vw NUMBER_OF_VULNERS_WORKERS -vr -vk YOUR_VULNERS_API_KEY_HERE -r

Example

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q queries/servers.json -vs -vw 10 -vr -vk YOUR_VULNERS_API_KEY_HERE -r

Enumeration with TLS Configuration and Attacks Scanning

Description

Run an enumeration with TLS scanning features

Template

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -ts -r

Example

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q queries/servers.json -ts -r

Enumeration with Additional Filtering

Description

Run an enumeration with filtering by vendors (only Nginx and Apache, for example) and confidence levels (only "Certain" level, for example) for queries and vendor

Template

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -u -q FILE_WITH_QUERIES.json -v VENDOR_TO_INCLUDE_IN_1 VENDOR_TO_INCLUDE_IN_2 -qc QUERY_CONFIDENCE -vc VENDOR_CONFIDENCE -r

Example

./grinder.py -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -u -q queries/servers.json -v nginx apache -qc certain -vc certain -r

Enumeration with Additional Custom Scripts

Description

Run an enumeration with custom scripts which are described in JSON file with queries in 10 workers

Template

./grinder.py -sc -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -sc -scw NUMBER_OF_SCRIPT_WORKERS -r

Example

./grinder.py -sc -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q queries/servers.json -sc -scw 10 -r

Enumeration with Additional Debug Information

Description

Run Grinder with debug information about scanning

Template

./grinder.py -d -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q FILE_WITH_QUERIES.json -r

Example

./grinder.py -d -sk YOUR_SHODAN_API_KEY_HERE -ci YOUR_CENSYS_ID -cs YOUR_CENSYS_SECRET -q queries/servers.json -r

Add Your Own Queries

Description

To add your own vendors and products with queries you can simply create a new JSON file in the directory with queries and choose it while running Grinder in the "run" scan mode.

Queries Template

[
    {
        "vendor":"Your vendor name here ('Apache', for example; any string is allowed and required)",
        "product":"Your product name here ('HTTP Server', for example; any string is allowed and required)",
        "shodan_queries":[
            {
                "query":"Shodan query here ('Apache', 'Server: Apache', for example; any string is allowed and required)",
                "query_confidence":"Query confidence level ('tentative', 'firm' or 'certain' - you can sort and modify your search with these keywords, use query confidence flag for this purpose)"
            }
        ],
        "censys_queries":[
            {
                "query":"Censys query here ('Apache', 'Server: Apache', for example; any string is allowed and required)",
                "query_confidence":"Query confidence level ('tentative', 'firm' or 'certain' - you can sort and modify your search with these keywords, use query confidence flag for this purpose)"
            }
        ],
        "scripts":{
            "py_script":"3 types of definitions are allowed: you can use a simple string here, for example, 'test/test.py'; you can use a list here, for example, ['test1/test1.py', 'test2/test2.py']; you can use a dictionary here, for example, {'test': 'test/test.py'}",
            "nse_script":"Currently, you can run only 1 NSE script per time, this is WIP. Define the script with string, for example, 'test/test.nse'"
        },
        "vendor_confidence":"Vendor confidence level ('tentative', 'firm' or 'certain' - you can sort and modify your search with these keywords, use vendor and product confidence flags for this purpose)"
    }
]

Queries Example

🚧 Note: Queries in the queries/ directory may be different due to the different stages of development, but mostly all of them are still supported and tested. The most actual query template and an example are provided below, so if you need or want to try your queries, you can use this example to get the freshest features.

[
    {
        "vendor":"Apache Software Foundation",
        "product":"Apache HTTP Server",
        "shodan_queries":[
            {
                "query":"Apache",
                "query_confidence":"certain"
            }
        ],
        "censys_queries":[
            {
                "query":"Apache",
                "query_confidence":"certain"
            }
        ],
        "scripts":{
            "py_script":"http_status/http_status.py",
            "nse_script":""
        },
        "vendor_confidence":"certain"
    },
    {
        "vendor":"Nginx",
        "product":"Nginx",
        "shodan_queries":[
            {
                "query":"Nginx",
                "query_confidence":"certain"
            }
        ],
        "censys_queries":[
            {
                "query":"Nginx",
                "query_confidence":"certain"
            }
        ],
        "scripts":{
            "py_script":[
                "http_response_grabber/http_response_grabber.py",
                "http_status/http_status.py",
                "test/test.py"
            ],
            "nse_script":""
        },
        "vendor_confidence":"certain"
    },
    {
        "vendor":"Flask",
        "product":"Flask",
        "shodan_queries":[
            {
                "query":"Flask",
                "query_confidence":"certain"
            }
        ],
        "censys_queries":[
            {
                "query":"Flask",
                "query_confidence":"certain"
            }
        ],
        "scripts":{
            "py_script":{
                "grabber":"http_response_grabber/http_response_grabber.py",
                "status":"http_status/http_status.py",
                "test":"test/test.py"
            },
            "nse_script":""
        },
        "vendor_confidence":"certain"
    }
]

Grinder Framework

Screenshot

Table of Contents

Description

Slides

Grinder Workflow

Grinder Map

Screenshots

Description

Requirements

Legend

Basic

Accounts

Additional scanning

TLS configuration

Current Features

Already Implemented

Additional Modules

DICOM Patient Info Getter

HTTP Raw Response Grabber

HTTP Status

SCP (SSH) Grabber

Sleep

SNMP Walker (easysnmp implementation)

SNMP Walker (system call implementation)

Tensor Serving Server Checker

Test

Development and Future Updates

Grinder Installing

Setup and Configure Environment

Running Grinder Map Server Locally

Building and Running in Docker

Description

Services and Images

Provided Scripts and Preparations

Environment

Building

Running

Tests

CLI Interface

Help on Command Line Arguments

Wiki

Usage Examples

Show Help

Description

Example

Basic Enumeration

Description

Template

Example

Enumeration with Limited Results

Description

Template

Example

Enumeration with Nmap Scanning

Description

Template

Example

Enumeration with Additional Analytics, Map and Plots

Description

Template

Example

Enumeration with Analytics from Vulners

Description

Template

Example

Enumeration with TLS Configuration and Attacks Scanning

Description

Template

Example

Enumeration with Additional Filtering

Description

Template

Example

Enumeration with Additional Custom Scripts

Description

Template

Example

Enumeration with Additional Debug Information

Description