cowj

[C]onfiguration [O]nly [Web] on [J]VM

[TOC]

Goal

COWJ is pronounced as Cows, and stands for:

Configuration-Only-Web-over-JVM

It should be very clear from the naming that:

objective is to optimize back-end development and replacing it with configurations.

Back End Development

Development Today

It is pretty apparent that a lot of back-end "development" goes under:

Trivial CRUD over data sources
Reading from data sources and object massaging - via object mapper, thus mapping input to output
Aggregation of multiple back-end services
Adding random business logic to various section of the API workflows

COWJ aims to solve all these 4 problems, such that what was accomplished by 100 of developers can be done by less than 10, a ratio of 90% above in being effective.

How To Do it?

Deprecation : Service Framework

Lately, 10+ people gets allocated to maintain "how to create a service" end point. This must stop. The following must be made true:

Creating a service end point should be just adding configuration
Writing the service should be just typing in scriptable code
Input / Output parameters are to be assumed typeless JSON at worst
We can, at best do a lazy type checking to validate Schema - alternatives
1. RAML
2. Open API
3. JSON Schema

"Business" Logic

There should not be any "business logic" in the code. They are susceptible to change, hence they should be hosted outside the API end points - DSL should be created to maintain.

Data Store Access

Any "Service" point requiring any "data store" access need to declare it, specifically as part of the service configuration process. Objective of the engine would be to handle the data transfer. JSON is the choice for data transfer for now.

Unit Testing - Code Coverage

If one is willing to use this, one must wonder - what Makes Anything 'Prod Ready' ? Only suitable answer is the core components must be excessively well tested, and should have really good instruction and branch coverage.

Cowj All Coverage

Thus the engine core is immensely tested, and is at per with industry standard quality.

Cowj Core Coverage

Cowj Plugin Coverage

Even plugins are reasonably tested, and are ready for production usage.

Current Implementation

Service Configuration

Here is the config:

# This shows how COWJ service is routed

# port of the server
port : 8000

# routes information
routes:
  get:
    /hello : _/scripts/js/hello.js
    /param/:id  : _/scripts/js/param.js

  post:
    /hello : _/scripts/js/hello.js

# route forwarding local /users points to json_place
proxies:
  get:
    /users: json_place/users
  post:
    /users: json_place/users


# filters - before and after an URI
filters:
  before:
    "*" : _/before.zm
  after:
    "*": _/after.zm

# how to load various data sources - plugin based architecture 
plugins:
  # the package 
  cowj.plugins:
    # items from each package class::field_name
    curl: CurlWrapper::CURL
    redis: RedisWrapper::REDIS

# data store connections
data-sources:
  redis :
    type : redis
    urls: [ "localhost:6379"]

  json_place:
    type: curl
    url: https://jsonplaceholder.typicode.com
    proxy: _/proxy_transform.zm # responsible for message transform

cron:
  cache:
    exec: _/cache.md
    boot: true
    at: "*/5 * * * *"

It simply defines the routes - as well the handler script for such a route. In this way it is very similar to PHP doctrine, as well as DJango or Play.

Scripting

Is Polyglot. We support JSR-223 languages - in built support is provided right now for:

JavaScript
Python via Jython
Groovy
ZoomBA

Underlying we are using the specially cloned and jetty 11 migrated spark-java fork:

https://github.com/nmondal/spark-11

Also it uses ZoomBA extensively : https://gitlab.com/non.est.sacra/zoomba

Here is how it works:

Client calls server
Server creates a Request, Response pair and sends it to a handler function
A script is the handler function which receives the context as req,resp and can use it to extract whatever it wants.
DataSources are loaded and injected with _ds variable.

Here is one such example of routes being implemented:

// javascript
let x = { "id" : req.params("id") };
x;// return

The context is defined as:

https://sparkjava.com/documentation#request

https://sparkjava.com/documentation#response

Shared Memory

Cowj supports a global, non session oriented global memory - which can be used to double time as a poor substitute for in memory cache - and it is a ConcurrentHashMap - accessible by the binding variable _shared.

The syntax for using this would be:

v = _shared.<key_name> // groovy, zoomba 
v = _shared[<key_name>] // groovy, zoomba, js, python

See the document "A Guide to COWJ Scripting" found here - Scripting

Filters

These are how one can have before and after callback before and after any route pattern gets hit.

A before filter gets hit before hitting the actual route, while an after filter gets hit after returning from the route, so one can modify the response if need be.

Classic example of before filter is for auth while after filter can be used to modify response type by adding response headers.

Before

A typical use case from auth:

routes:
  get:
    /users: _/users.zm
filters:
  before:
    "*" : _/auth.zm

Now inside auth.zm :

// auth.zm
def validate_token(){ /* verification here */ }
token = req.headers("auth")
assert("Invalid Request", 403) as { validate_token(token) }

Finally

A typical use case to json format error messages:

routes:
  get:
    /users: _/users.zm
filters:
  finally:
     "*" : _/finally.zm

Now inside finally.zm :

// finally.zm
if ( resp.status @ [200:400] ){ return }
error_msg = resp.body
resp.body( jstr( { "error" : error_msg } ) )

This ensures all errors are formatted as json.

Cron

In the cron section one can specify the "recurring" tasks for a project. The key becomes name of the task, while:

exec : script that needs to be executed
boot : to run while system startup
at : cron expression to define the recurrence

Data Sources

If the idea of COWJ is to do CRUD, where it does CRUD to/from? The underlying data is provided via the data sources.

How data sources work? There is a global map for registered data sources, which gets injected in the scripts as _ds :

_ds.redis

Would access the data source which is registered in the name of redis .

Right now there are the following data sources supported:

JDBC

JDBC data source - anything that can connect to JDDBC drivers.

some_jdbc:
  type: jdbc # shoule have been registered before
  driver : "full-class-for-driver"
  connection: "connection-string"
  properties: # properties for connection
    x : y # all of them will be added

CURL

External Web Service calling. This is the underlying mechanism to call Proxies to forwarding data.

some_curl:
  type: curl # shoule have been registered before
  url : "https://jsonplaceholder.typicode.com" # the url 
  headers: # headers we want to pass through
    x : y # all of them will be added

Google Storage

Exposes Google Storage as a data source.

googl_storage:
  type: g_storage # shoule have been registered before

Redis

Exposes Redis cluster as a data source:

 redis :
   type : redis # should register before 
   urls: [ "localhost:6379"] # bunch of urls to the cluster

Firebase Notification Service

Firebase as a data source.

Plugins

All data sources are implemented as plug and play architecture such that no code is required to change for adding plugins to the original one.

This is how one can register a plugin to the system - the following showcases all default ones:

# how to load various data sources - plugin based architecture 
plugins:
  # the package 
  cowj.plugins:
    # items from each package class::field_name
    curl: CurlWrapper::CURL
    fcm: FCMWrapper::FCM
    g_storage: GoogleStorageWrapper::STORAGE
    jdbc: JDBCWrapper::JDBC
    redis: RedisWrapper::REDIS

As one can see, we have multiple keys inside the plugins which corresponds to multiple packages - and under each package there are type of datasource we want to register it as, and the right side is the implementor_class::static_field_name.

In plain language it reads:

A static filed CURL of the class cowj.plugins.CurlWrapper implements a DataStore.Creator class and is being registered as curl type of data store creator.

Library Folder

From where plugins should be loaded? If one chose not to compile their code with COWJ - as majority of the case would be - there is a special folder in the base director defaults to lib. All jars in all directories, recursively will be loaded in system boot and would be put into class path,

One can naturally change the location of the lib folder by:

lib: _/some_other_random_dir

How to author COWJ plugins can be found in here Writing Custom Plugins

Proxies

Path/Packet forwarding. One simply creates a base host - in the data source section of type curl and then use that key as base for all forwarding.

In the Yaml example the following routing is done:

localhost:5003/users 
--> https://jsonplaceholder.typicode.com/users

System responds back with the same status as of the external web service as well as the response from the web service gets transferred back to the original caller.

This transform is coded in the curl type as follows:

data-sources:
  json_place:
    type: curl
    url: https://jsonplaceholder.typicode.com
    proxy: _/proxy_transform.zm # responsible for message transform

The proxy section has the script to transform the following to be forwarded to the destination server :

request object
headers has a mutable map of request headers
queries has mutable map of all query parameters
body has the string which is request.body()

Evidently at a forward proxy level, these are the parameters one can change before forwarding it to destination.

The transformation function / script is expected to return a map of the form:

{
  headers : {
    key : value
  },
  query : {
    key : value
  },  
  body : "request body"
}

In case the script does not return a map - pushed values will be used to be extracted from the script context and used as a response.

Type System

We support json schema based input validation. To read more see Writing Input Validations

Running

Build the app.
Run the app by going to the app/build/libs folder by:

java -jar cowj*  <config-file>

The jar has been created such as to have classpath property set to run as long as all the dependencies are in the deps folder.

saptarshiweb / cowj