BulkScanner
A tool for SOC Analysts to analyze observables in bulk.
Introduction • Key Features • Download and Installation • Default Credentials • How To Use • Currently Integrated Tools • About secret_key.yaml file • Technology used in this project • License
Introduction
BulkScanner is a web based tool designed and developed for a security/SOC analysts to analyze observables in bulk such as IP addresses, hashes, urls, and domains. BulkScanner is based on open-source intelligence analyzers such as VirusTotal, AbuseIPDB, X-Force Exchange(IBM), metadefender etc.
Key Features
- Analyze observables in bulk
- Accept large log file (CSV only)
- Custom Column names
- Adding API keys for each tool
- Store each analyzed observable in database
- First check for observable in the database before making request to the selected tool.
- If observable present in database, will not send request
- If observable not present in database, will send request for analyzing. We can save our time and API calls by doing this.
- Create a unique group of selected obserables before analyzing. Remove the duplicated objects and save API calls by doing this.
- Ignore Private IP addresses.
- Store failed observable in a seperate table so we can save the API calls by sending the requests of failed observables again and again. (Observables that have no records are considered failed objects.)
- The location information for IP addresses is obtained using a tool specifically designed for identifying geographical locations, called geolocation-db.com.
- The Observable Details page to generate concise and precise reports.
- Admin panel for admin actions.
Download and Installation
You can download the latest version of BulkScanner by clicking download, or click on Code > Download Zip. Or you may clone the repository "https://github.com/Salman7870/bulkscanner.git"
Pre-requisites for BulkScanner
- Python 3.10.5 or above must be installed
Easy Install
This installation process is designed for non-techincal users.
Step 1
Download the zip file by clicking download and extract it.
Step 2
Execute or run the bat file "run.bat" inside the project directory and that's it. Wait sometime, after completion, visit to http://127.0.0.1:8000.
What happens when click on run.bat?
- It will first check if python is installed or not on your machine.
- if python is installed, then the script will automatically create a virtual enivroment for you and install all the required packages to run the project. After installation, you will see the django server is running on your CMD console. Just go to browser and run http://127.0.0.1:8000
- If Python is not installed, a message will display on the console "Python is not installed on this machine."
- Finally, it will deactivate the virtual environment if user closes the CMD console or Terminal.
Note This installation is for windows operating systems only. To run and install BulkScanner on Linux based machines, refer to Technical Install guide below.
Technical Install
Step 1
Download or clone the repository.
Step 2
Create a virtual environment by running python -m venv venv in the project directory and activate it by venv\Scripts\activate
Step 3
Install all the required packages by running pip install -r requirements.txt
Step 4
Run the django server by python manage.py runserver and access the app by http://127.0.0.1:8000
Database Configuration
By default, SQLite database is used in this project. You use various databases. Below are the supported databases in Django.
- PostgreSQL
- MariaDB
- MySQL
- Oracle
- SQLite
Note : SQLite database may be slow in performance when number of records increases and not recommended for large data.
Default Credentials
Username: admin
Email: admin@admin.com
Password: admin
Make sure to change default password for the user. You can also create your own super user by python manage.py createsuperuser
Or to create a normal user, just visit http://127.0.0.1:8000/accounts/register to register a new user.
How To Use
Add API Keys
First of all, add API keys of the tools which you want to use such as VirusTotal and AbuseIPDB.
You can add multiple API keys and can be recoganize by Owner Email or Name.
Add Column names
Columns are the keys in log files such as source_ip, destination_ip, md5_hash, etc..
Suppose the below CSV log file. Here, we will add source_ip and destination_ip column names so we can select it when analyzing or scanning this file.
To do this, click on Add Columns and enter the column name.
Scan a file
Now you are ready to go. Select your desire tool under Tools, then select API key, column name and upload your CSV log file. Click on Scan Now button.
Currently Integrated Tools
Currently, Following tools/analyzers have been integrated:
GeoLocation-DB tool is used to get location information
About secret_key yaml file
Django SECRET_KEY is a randomly generated string of characters that is used to provide cryptographic signing for various security-related features in Django framework, such as authentication, sessions, and CSRF (Cross-Site Request Forgery) protection.
The SECRET_KEY should be kept secret and not shared with anyone, as it is used to generate secure hashes and tokens that are used to authenticate users and protect against attacks.
It is placed in settings.py file as SECRET_KEY = "random_string"
To address the issue of avoiding the sharing of SECRET_KEY, I have added a peace of code the settings.py file to assign a unique SECRET_KEY to each user during their initial installation.
Once the application is run for the first time, the code in settings.py checks for the presence of a SECRET_KEY in the secret_key.yaml file. If a key exists, it is retrieved from the file. However, if it does not exist, a new, robust secret key is generated and automatically stored in the secret_key.yaml file. This ensures that every user who downloads this project has their own individual and exclusive SECRET_KEY for their usage of the application.
Now everytime, when a user run the application, it will use the secret key store in the secret_key.yaml file.
Support
Technology used in this project
- Python
- Django
- HTML
- CSS
- JavaScript
Credits
- Tabler template is used.
- VirusTotal
- AbuseIPDB
License
MIT
GitHub @Salman7870 · LinkedIn @muhammad-sulaiman7870