haitham911 / icapeg

Open Source ICAP server

ICAPeg

Open Source multi-vendor ICAP server

Scan files requested via a proxy server using ICAPeg ICAP server, ICAPeg is an ICAP server connecting web proxies with API based scanning services and more soon!. ICAPeg currently supports VirusTotal,VMRAY , MetaDefender & Clamav for scanning the files following the ICAP protocol. If you don't know about the ICAP protocol, here is a bit about it:

What is ICAP?

ICAP stands for Internet Content Adaptation Protocol. If a content (for example: file) you've requested over the internet to download or whatever, needs adaptation(some kind of modification or analysis), the proxy server sends the content to the ICAP server for adaptation and after performing the required tasks on the content, the ICAP server sends it back to the proxy server so that it may return the adapted content back to the destination. This can occur both during request and response.

To know more about the ICAP protocol, check this out.

Things to have

Before starting to play with ICAPeg, make sure you have the following things in your machine:

1. Golang(latest enough to be able to use go mod)

   A sample installation of go version 1.14:

   Prepare the apt packages

  sudo apt update

  sudo apt upgrade

Link of download of version 1.14 https://dl.google.com/go/go1.14.linux-amd64.tar.gz

Use the command

  wget https://dl.google.com/go/go1.14.linux-amd64.tar.gz

Untar in /usr/local

  tar -C /usr/local -xzf go1.14.linux-amd64.tar.gz

Add /usr/local/go/bin to the PATH environment variable:

  export PATH=$PATH:/usr/local/go/bin

Note: this command adds the PATH temporarily to the environment variables, the path is removed if SSH is broken, or system reboots.

2. A proxy server

squid is an example in this readme

3. A scanner vendor. ICAPeg now supports VirusTotal, MetaDefender , VMRay & Clamav as scanner vendors. Make sure that you setup your scanner vendor properly. You can setup your scanners for both RESPMOD & REQMOD. Although, not every scanner supports every mods. Check this out to know which scanner supports which mods.

Setup VirusTotal:

Insert VirusTotal as your scanner vendor in the config.toml file

  resp_scanner_vendor = "virustotal"

Or,

  req_scanner_vendor = "virustotal"

In that same file, add a VirusTotal API key in the api_key field of the [virustotal] section. Here is how you can get it.

Setup MetaDefender:

Insert MetaDefender as your scanner vendor in the config.toml file

  resp_scanner_vendor = "metadefender"

Or,

  req_scanner_vendor = "metadefender"

In that same file, add a MetaDefender API key in the api_key field of the [metadefender] section. Here is how you can get it.

Setup VMRay:

Insert vmray as your scanner vendor in the config.toml file

  resp_scanner_vendor = "vmray"

Or,

  req_scanner_vendor = "vmray"

In that same file, add a VMRay API key in the api_key field of the [vmray] section. Get your api key by requesting a free trial.

Setup Clamav:

Insert clamav as your scanner vendor in the config.toml file

  resp_scanner_vendor = "clamav"

Next, provide the clamd socket file path(getting back to this in a bit) in the config.toml file inside the clamav section

  socket_path = "<path to clamd socket file>"

Here is how you setup clamav and generate the socket file

NOTE: All the settings of ICAPeg is present in the config.toml file in the repo. Also before selecting your vendors as the scanners, keep in mind to check whether that certain vendor supports the modification mode or not. For example, when adding virustotal as the resp_scanner_vendor, check under the configuration of virustotal if the resp_supported flag is true or not. Likewise for req_scanner_vendor and for any other vendors. Also you can provide none in the resp/req_scanner_vendor/vendor_shadow fields to indicate no vendor is provided & ICAPeg is just gonna avoid processing the requests.

How do I turn this thing on!!

To turn on the ICAPeg server, proceed with the following steps (assuming you have golang installed in you system):

1. Clone the ICAPeg repository

  git clone https://github.com/egirna/icapeg.git

2. Enable go mod

  export GO114MODULE=on

In case not using go version 1.14, you could discover your version

  go version

      You should use the corresponding export command

      1.14 ===> export GO114MODULE=on

      1.13 ===> export GO113MODULE=on

      etc.

3. Change the directory to the repository

  cd icapeg/

4. Add the dependencies in the vendor file

  go mod vendor

5. Build the ICAPeg binary by

  go build .

6. Finally execute the file like you would for any other executable according to your OS, for Unix-based users though

  ./icapeg

You should see something like, ICAP server is running on localhost:1344 .... This tells you the ICAP server is up and running OR, you can do none of the above and simply execute the run.sh shell file provided, by

 ./run.sh

That should do the trick.

2. Now that the server is up and running, the next thing to do is setup a proxy server which can send the request body to the ICAPeg server for adaptation. Squid looks like just the thing for the job, go to the site provided and set it up like you want. After setting up your proxy server for example squid, change its configuration file:

Open squid.conf file

  sudo nano /etc/squid/squid.conf

Add the following lines at the bottom of your ACLs configurations

  icap_enable on
  icap_service service_resp respmod_precache icap://127.0.0.1:1344/respmod-icapeg
  adaptation_access service_resp allow all

Add the following line at the end of the file

  cache deny all

A sample conf file for squid exists in the repository in a file squid.conf

Save and close the file Press CTRL + x, then press Y, then Enter

Restart squid:

  systemctl restart squid

How do I know its working!

3. Now that you have squid running as well, you can test it out by trying to download/access a file from the Internet(through the proxy) and see the magic happens! You'll be able to download/access the file if its alright,

   If you try and download something not malicious you should see something like this in the logs:

   To test properly using malicious files , visit the Eicar Test File Site, and try to download a malicious file.

   For example, open the following link

   www.eicar.org/download/eicar_com.zip,

   There will be terminal logs such as:

   And you are gonna see something like this in the browser:

   And the details of the malicious file is shown by clicking on "details" button:

Oh, and do not forget to setup your Browser or Machine 's proxy settings according to the squid.

If you are still not getting any logs and the file is downloaded whatsoever, please check your proxy(squid) setup.

Things to keep in mind

1. You will have to restart the ICAP server each time you change anything in the config file.

2. You will have to restart squid whenever you restart the ICAP.

3. You need to configure your network(or your browser)'s proxy settings to go through squid.

More on ICAPeg

1. Remote ICAP Servers & Shadowing

2. Logging

Contributing

This project is still a WIP. So you can contribute as well. See the contributions guide here.

License

ICAPeg is licensed under the Apache License 2.0.