This repository is purely for reference and is illustrative in it is purpose. Please do not use in production as is, use this as a guide or starting point for a production level implementation.
This project illustrates the use of Nats.io built in Scatter-Gather streaming pattern within an ExpressJS API. The goal behind this implementation is to showcase a loosely coupled pub-sub Rest API Scatter-Gather pattern, wherein the implementation of an API request is handled by multiple subscribers.
This pattern is useful in many ways, but there are a few freebies from this implementation :
- Any programming language can use Nats.io API to initiate Request/Response directly with Nats.io bypassing HTTP entirely
- The API can be written in NodeJS|Go|Python|Java|.NET Core C#, and the Subscriber(s) can be written in NodeJS|Go|Python|Java|.NET Core C#
- Multiple subscribers can be added and or removed at runtime, increasing or decreasing the number of results per API call
- The Nats Subject can be subscribed to by (n) additional subscribers, for activities like logging, direct push to a DataLake, and so on.
What will be seen in this example is a stock ticker being published to a Nats Subject with multiple APIs from Polygon.io and Alpha Vantage replying to the request token asynchronously (but appearing synchronous).
Before you continue, ensure you have met the following requirements:
- Nats Server or Nats Docker Server installed and running
- If installing the Go Server, Go must be installed
- NodeJS v18 or higher installed
- Npm installed
- Get a free AlphaVantage key!
- Get a free Polygon.io key!
This repository uses dotenv, feel free to create a .env file to set the ALPHA_VANTAGE_KEY, or override other aspects of the program.
- ALPHA_VANTAGE_KEY : The API Key provided by Alpha Vantage (defaults to no-key)
- POLYGON_IO_KEY : The API Key provided by Alpha Vantage (defaults to no-key)
- PORT : The port the express server should run on (defaults to 3000)
- SUBJECT : The Nats subject the Request/Reply will be issued on (defaults to stock.request)
- NATS_SERVER : The Nats server that will be facilitating Pub-Sub (defaults to localhost)
- 'cd' to the root of this repository (where it was cloned)
- Create a file in the root named .env
- Add the AlphaVantage key in this file as follows : ALPHA_VANTAGE_KEY=< insert key here >
- Add the Polygon.io key in this file as follows : POLYGON_IO_KEY=< insert key here >
- run npm install from the command line
- open a terminal to the root of this repository and run :
- npm run daily-adjusted-subscription
- allow the subscription a few additional seconds to bind, 503 errors may be experienced during this binding time
- open a second terminal to the root of this repository and run :
- npm run api
- open Postman or a browser and execute a GET request to :
- http://localhost:3000/stock-info?symbol=AAPL
- The symbol can be changed to get different results
- Try MSFT, IBM, CRM, GOEV, AVAV, TJX, CRM, V
- Repeat steps 4 through 6 opening a new terminal for each of the below commands (issued at the root of this repository)
- npm run ema-subscription
- npm run sma-subscription
- npm run overview-subscription
- As these processes are started (remember to run each in its own terminal/console), the REST Response will have an additional item in the response totalling 4 responses for 1 API call.
A REST API that is only aware of a Nats Subject, and that it is the publisher. (n) amount of Request-Reply subscribers, and or receive only subscribers can be added to that Nats Subject, any replies are published back to the issuing Requestor.
- Notice that the API has no prior knowledge as to what the subscribers are going to provide, nor how many there will be
- Validation can be put in place or a canonical can be used to ensure correctness of the reply
- This repository is heavily commented to provide context as to what and why, if in VS Code feel free to collapse all comments if they are obtrusive
- On Mac -> Press ⌘ + K then ⌘ + /
- On Windows & Linux -> Press Ctrl + K then Ctrl + /
- adding additional subscribers and observe the results, when does it break, why does it break?
- adding additional non-reply subscribers, that perform other duties based on the stock ticker in the Nats Msg
- writing a subscriber in Go|Java|.NET Core|Python, and observe the results
- writing the REST api in Go|Java|.NET Core|Python, and observe the results