ramesaliyev / mom

Proof of concept for Message-Oriented-Middleware based architecture.

MOM Based Architecture POC

This is a demonstration of how to create simple yet capable architecture with MOM (Message-Oriented-Middleware) approach to achieve well scalable and distributed systems. On the tip of the iceberg we're only seeing an admin panel, where you can register, login, order and list jobs etc. But under the hood (or below the sea level?) whole app running by queuing messages to do things.

Here is a diagram to visualise how our architecture works.

Table of Contents

• Core Concept
• Additional Pieces
  • Sandbox
  • Scripts
• Installation and Running
  • Cloning Repo
  • Start Everything
  • Start Only Data Services
  • Stopping a Service
  • Starting a Service with Rebuild
  • Starting a Service outside of Docker
• Technologies Used
• Management Screens
• Connection Points
• Using Sandbox
• Debugging
• Resources for Better Understanding

Core Concept

The core concept is; when there is a work which need to be done; services wont communicate to related service directly. Instead of that they will leave a message to message queue, which will be consumed by service (worker) that responsible of that work.

For example, in traditional systems when an user want to register, he or she calls the api and api will create a new db record and return some response to user. But here, API doesn't create db record itself, instead of that; it leaves a message to message queue which have details about what should be done (and data like email, password etc.) and waits for some db-worker to take care of it.

This waiting technique actually named RPC and it means that message leaver will wait until job is done, and will do something with returned data (like informing user directly by returning a response). We're using RPC only in jobs like user register or job creation for make an example. RPC messages will have top priority (10) which cause them to be processed by worker before ordinary (lower priority) messages.

On the other hand, there are jobs which are not in a hurry and needs to be done sometime. Like math tasks. There is a screen in panel where you can schedule math tasks. Like sum or multiply these numbers etc. And there is a math-worker inside worker service which will call another external math service to get results of this calculations. And to inform user after these kind of deferred works, we're using socket connections. (Our math service will respond between 0~10 second delay for demonstration purposes.)

But what if user not in panel and there is no socket connection? Well, after a deferred task, beside trying to inform user over socket, of course we should also persist result of that job into database. (By leaving a message to message queue for db-worker.) So even user not in panel, he/she will see information on next login. (I didnt implement this, but this is the idea.)

Additional Pieces

Sandbox

Seperated from our main application; the purpose of sandbox is making experiments for grasp better understanding over concepts, libraries etc. For more information and to learn how to use sandbox see using sandbox section.

Scripts

Some bash scripts to use when needed.

Installation and Running

To install and run application you only need to have installed and running docker.

Cloning Repo

git clone [e-mail]:ramesaliyev/mom.git

Start Everything

docker-compose up --build

Starting everything will take some time, so be patient. When everything has started you can;

• Navigate to admin panel at localhost:9090
• See other management screens

Start Only Data Services

docker-compose up postgresql_server postgresql_adminer rabbitmq_server redis_server redis_commander

Stopping a Service

docker-compose stop <servicename>

# example
docker-compose stop api

Starting a Service with Rebuild

docker-compose up --build <servicename>

# example
docker-compose up --build api

Starting a Service outside of Docker

You can start every service outside of docker.

1- Navigate to service folder.

cd services/be-api

2- Install dependencies.

npm i

3- Start in development mode.

npm start

4- Or Production mode.

npm run start:prod

Notices:

1. In docker every service starts in production mode by default.
2. There is no watch mode in development. So you need to restart a service if you change something.
3. You need to rebuild docker image if you want to start a service in docker after you change something.

Technologies Used

• Backend
  • Docker
  • JWT
  • NodeJS
    • AMQP.Node
    • Nest Framework
    • TypeORM
  • PostgreSQL
  • RabbitMQ
    • Delayed Message Exchange
    • Management
  • Redis
  • Socket.io
  • TypeScript
• Frontend
  • Axios
  • Material-UI
  • React
  • React-Router
  • Redux
  • Redux-Saga

Management Screens

• Our Panel: localhost:9090
• PostgreSQL: localhost:9091
  • system: PostgreSQL
  • server: postgresql_server
  • username: poc_db_user
  • password: poc_db_pass
  • database: poc_db_name
• RabbitMQ: localhost:9092
  • username: poc_rmq_user
  • password: poc_rmq_pass
• Redis: localhost:9093

Connection Points

• API:
  • hostname: localhost:7070
• Socket:
  • hostname: localhost:7080
• External Math Service API:
  • hostname: localhost:7090
• PostgreSQL
  • hostname: localhost:7071
  • others same as above
• RabbitMQ
  • hostname: localhost:7072
  • others same as above
• Redis
  • hostname: localhost:7073

Using Sandbox

There is a sandbox to ease making experiments with things. It consist of mostly little scripts written in plain javascript by following tutorials etc. You can add something under the sandbox and run it within container. Since folder already added as volume you dont need to restart anything. To enter in sandbox;

docker exec -it poc_sandbox /bin/sh

And when you in, simply run scripts as you want;

node rabbitmq/tutorial/hello-world/send.js

Create a new terminal window and run the receiver also;

node rabbitmq/tutorial/hello-world/receive.js

You can run multiple receivers to observe things.

Debugging

These debug instructions are for VSCode only.

Frontend

1. Install latest version of VS Code and VS Code Chrome Debugger Extension.

2. Start your app (outside of the docker) by running npm start.

3. Now start debugging in VS Code by choosing correct configuration (ex: Fe/Panel).

4. You can now write code, set breakpoints, make changes to the code, and debug your newly modified code, all from your editor.

Services

1. Navigate to .ts file you would want to debug.

2. Now start debugging in VS Code by choosing correct configuration (ex: Be/Api).

3. You can now write code, set breakpoints, make changes to the code, and debug your newly modified code, all from your editor.

For example: you want to debug api, navigate to src/modules/auth/auth.controller.ts and add breakpoint to login method. Then in postman or from frontend, trigger the api.

Resources for Better Understanding

• Docker
  • Building Efficient Dockerfiles - Node.js
• MOM
  • Wikipedia
• RabbitMQ
  • Dictionary
  • Official
    • JS Tutorial
    • Github Repo of Tutorial
    • AMQP Concepts
    • Documentation
    • Queues
  • Other
    • AMQP.Node API Reference
    • What is RabbitMQ?
    • Getting Started with RabbitMQ and NodeJS
    • Management Interface
    • RabbitMQ Exchanges, Routing Keys and Bindings
    • RabbitMQ and NodeJS, A better way.
    • How To Do User Notifications From RabbitMQ Messages
• TypeScript
  • In 5 Minutes
  • Handbook