If you seek to consume any REST API, fetch some items/events/messages, format them into valid syslog messages and feed them into any Syslog destination Rest2Syslog (R2S) is a perfect choice!

Why syslog? Many products/platforms out there support ingestion of data (inbound messages) using the syslog standard for messages. Some product examples include: Kafka, Elasticsearch, Splunk, Redis, QRadar, the list goes on and on. If you want to send messages from your Platform's REST API (eg Message Source) into other products (eg Message Destination), it is very likely that syslog is a viable protocol. R2S will help you get the job done faster - all you need to do is extend R2S to support your specific RESTful APIs. Read on for the full details on how to extend R2S.

Why R2S on top of syslog-ng? syslog-ng is a free and open-source implementation of the syslog protocol for Unix and Unix-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport. More over, syslog-ng is extensible by design and allows writing custom sources and destinations. This allows us to implement REST2Syslog as a custom source.

REST2Syslog collects data from any REST API extract relevant fields, formats syslog messages, and sends them over to any configurable syslog destination. It is heavily based on syslog-ng and implented as a python source plugin

REST2Syslog (R2S) features:

Execute your API calls periodically using a configurable interval parameter
Provides authentication flow for tokens/refresh tokens. Authentication scheme is configurable.
Provides out of the box support for pagination : many APIs requires iterating over pages of data.
State persistency - supports fetching delta since last invocation. This is crucial for efficient quota consumption. Some API providers will limit your rate and daily quota so this is important!
Parses response into individual items
Transform your Items into standard message formats such as IBM QRadar LEEF or HP ArcSight CEF
handles errors gracefully
handles sending messages as syslog messages
extensions API - write your own custom extensions (read below for details)
unit tested with >92% code coverage

The initial implementation includes a Proofpoint CASB Alerts extension plus an Events extension. Those extensions will pull alerts/events from the Proofpoint CASB platform and stream those items into any valid syslog-ng destination.

Writing custom extensions

In order to extend R2S with your custom extensions, you will need to implement three python classes: a API Adaptor, a Paginator and an Item formatter. Please review the existing extensions for more details. You can find two extensions inside the folder r2s/extensions/proofpoint/pcasb.

I'll be glad to accpet pull requests for additional extensions!

Feel free to send any feedback/question. just open an issue.

Deployment

Dependencies

python3 make sure you have python
requests
syslog-ng (>= 3.18)

Installation

Install syslog-ng (=> 3.18)
Copy / Clone the r2s scripts
- git clone this project into a local folder
- in case syslog-ng is started/stopped using systemctl you should append the root folder (eg - r2s) path to the PYTHONPATH environment variable like this:
```
PYTHONPATH="<full-path-to-your-python-file>"
```
  for example:
```
PYTHONPATH="/opt/sysconfig/syslog-ng/Rest2Syslog/"
```
  For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting syslog-ng OSE. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file:
```
PYTHONPATH="<full-path-to-your-python-file>"
```
- Alternatively to a dedicated folder you can copy the python r2s scripts into sbin

Edit the syslog-ng configuration file (usually located at /etc/syslog-ng/syslog-ng.conf)

Add this syslog-ng source s_r2s. you should change the paramters (see parameters description below the following snippet)

source s_r2s{
    python(
      class("r2s.source.REST2SyslogSource")
      options("company_name","Company_Name")
      options("product_name","Product_Name")
      options("product_version","Product_Version")
      options("interval",60)
      options("auth_url","https://url.to.auth.endpoint/example")
      options("alerts_url","https://url.to.alerts.endpoint/example")
      options("events_url","https://url.to.events.endpoint/example")
      options("api_key","your_api_key")
      options("client_id","your_client_id")
      options("client_secret","your_client_secret")
      options("max_pages",0)
      options("extensions","pcasb_alerts,pcasb_events")
      options('pcasb_alerts.formatter_module','r2s.extensions.proofpoint.pcasb.alerts_formatter')
      options('pcasb_alerts.formatter_class','PCASBAlertsFormatter')
      options('pcasb_alerts.api_adaptor_module','r2s.extensions.proofpoint.pcasb.alerts_api_adaptor')
      options('pcasb_alerts.api_adaptor_class','PCASBAlertsAPIAdaptor')
      options('pcasb_alerts.api_paginator_module','r2s.extensions.proofpoint.pcasb.alerts_paginator')
      options('pcasb_alerts.api_paginator_class','PCASBAlertsPaginator')
      options('pcasb_events.formatter_module','r2s.extensions.proofpoint.pcasb.events_formatter')
      options('pcasb_events.formatter_class','PCASBEventsFormatter')
      options('pcasb_events.api_adaptor_module','r2s.extensions.proofpoint.pcasb.events_api_adaptor')
      options('pcasb_events.api_adaptor_class','PCASBEventsAPIAdaptor')
      options('pcasb_events.api_paginator_module','r2s.extensions.proofpoint.pcasb.events_paginator')
      options('pcasb_events.api_paginator_class','PCASBEventsPaginator')              
    );
};

in addition to the above syslog-ng source, you should also add some destination. Here is an example destination to a remote syslog server. Any valid syslog-ng destination will do!
```
destination d_tcp { syslog("10.10.0.1" transport("tcp") port(514) ); };
```
finally, wire the source and the destination in order to drive traffic (aka items) from the source into the destination.
```
log {source(s_r2s); destination(d_tcp); };
```

Architecture

The following table summarizes key components in the R2S system with a description of what they do and how they interconnect with other components in the system:

Component Name	Description
Syslog-ng client	the main process running r2s is a syslog-ng client. Usually this process is setup as system service which is launched automatically on system startup. The syslog-ng client loads r2s if it is properly defined in the syslog-ng.conf file as described above. Once r2s is loaded, syslog-ng will call the `run` method which is the entry point in r2s.
REST2SyslogSource	This is the main class in r2s. It is implementing a syslog-ng Source (see syslog-ng documentation if you're not familiar with what is a Source). In turn, this component loads any pre-defined extensions as described in the syslog-ng.conf file under the key "extensions". Per each of the loaded Extensions, a call to `doWork` will occur every X seconds as defined in the Sleep "interval" parameter.
Extension	r2s can load multiple Extensions. each extension loads its own `Paginator` and calls its `fetchPageItems` until there are no more pages left (eg - max_pages reached) or no new items exist in the API response. Also - the extension will stop processing if the Service is halted for some reason.
Paginator	On bootstrap, this compoent will load an API Adaptor and a Formatter as configured in the r2s configuration parameters. The main method is `fetchPageItems` where the Paginator calls the API Adapator to fetch the next page, followed by a call to the Formatter in order to wrap each response item for message parsing and formatting (eg - LEEF / CEF format)
API Adaptor	The API Adaptor, implements the API calls. It handles authentication, and items fetching. the main method here is `fetchItems` which calls the REST API and extact the response payload. When extending R2S the formatter is one of the required extension components.
Formatter	wraps API specific entities and tranlate them into a well defined format. This component abstracts away the details of how to translate a specific response json scheme. When extending R2S the formatter is one of the required extension components.

Testing

Code coverage (from the projects's root directory):

coverage run --source . --omit *_test*,*infra* -m pytest tests/

Code coverage reporting:

coverage report

Code coverage html report:

coverage html

chenbekor / Rest2Syslog