Live app on Heroku 🕺
https://nodejs-realtime-rest-api.herokuapp.com
Table of contents:
- A realtime Node.js based REST API
A realtime REST API to manage the sales of "Fictional Motor Co", a global automobiles manufacturer and seller. They require a scalable REST API with the possibility of consuming part of the data in realtime. Of course, all of this with the corresponding security and, CRUD functionality.
This is a demo project to provide an example of my skills for building a REST compliant API, with the addition of a layer for real-time (thanks to WebSockets). This time, I’ve decided to use Feathers, a Node.js framework oriented for building real-time applications and REST APIs. I've made an extensive use of the latest version of ECMAScript on the backend and, used a few cool features of Vue.js on the frontend. Other modern development tools used are: NPM (the Node.js package manager), Sequelize (a promise-based Node.js ORM for relational databases), MySQL server (for data storage), Docker (helps to create the necessary environment for developing or running the application) and, the Swagger toolset (for exploring and interacting with the API). And, of course, I’ve used Git for code control version and, a basic knowledge of the Unix Shell for interacting with the respective CLI (command-line interface) for Git, Docker and, Feathers. The full project has been developed on macOS Catalina (v10.15.6) on top of a MacBook Air mid 2012.
You can use this project for your needs under your total responsibility. You can, for example, fork it and, use it as a foundation for your own project if you found it useful.
Fictional Motor Co manufactures and sells automobiles globally, with new vehicles sold through regional sales offices.
Global headquarters requires each of the regional offices to submit information on vehicle sales, and this is expected to be near real-time (within 30 seconds of a complete sale). This enables headquarters to effectively manage the manufacturing portion of the business.
Sales data is stored in a central location within the headquarters infrastructure, some transformation is applied to the input data. Manufacturing facilities use the data to increase and decrease production, including the procurement of necessary parts. In addition, the data is used for aggregate reporting purposes.
Different vehicle models are produced in different areas of the world, with some of the vehicle parts built in locally owned facilities, while others come from external, local and overseas producers.
Each production facility has different supply chains and lead times, and as such everyone needs to access information about new vehicles as often as is best for them.
All manufacturing plants poll the data regarding new sales, some poll every minute and some once a day. The amount of data extracted can be enormous.
- Produce an API specification that will allow regional sales offices to send sales information to headquarters.
- Produce an API specification that will be used by manufacturing facilities to extract information on new car sales from headquarters.
- Produce an architecture diagram of the solution, include the APIs and the persistence layer. Suppose we will use a cloud-based provider.
- Produce some simple prototypes.
- Safety is essential.
- If the details are not detailed, you are expected to make valid assumptions about what they would be. i.e data payloads, acceptable security, etc.
- Produce some tests or a test installation. (think PostMan or an automated testing suite/tool).
As I said before, Feathers is the core framework for the project. I’ve decided to use it after a small research about the state of the art of current Node.js frameworks. Feathers is a lightweight web-framework for creating real-time applications and REST APIs using JavaScript or TypeScript.
Feathers can interact with any backend technology, supports over a dozen databases and works with any frontend technology like React, VueJS, Angular, React Native, Android or iOS. In this project, the interface with the database is done thanks to Sequelize (a promise-based ORM for Node.js, that works with Postgres, MySQL, SQLite and, Microsoft SQL Server). Sequelize also provides an effortless data validation and, much more. Feathers also can integrate with Express (a solid foundation currently used by almost any existing Node.js framework), something I decided to do, because it provides better JSON error responses.
Sequelize has choosen as the ORM for the application, because it allows to define the models schemas and, validate the fields. It provides model level validations and, will return the validation errors to the client in a nice consistent format.
Feathers provides instant CRUD functionality via Services, exposing both a RESTful API and real-time backend through websockets automatically.
And, to finish, it also provides easy integration with more than 180 OAuth providers. In this case, the project uses GitHub as a third party OAuth provider.
So, my work was: first to known all this tools and technologies, understand how they work reading technical docs and finally, build it all together to provide the required functionality.
Also, it’s worth to mention that I’ve followed the security considerations detailed on the official Feathers Guides. In particular, there is a full section about security. The following points of the security section are the relevant ones for this project:
- Using hooks to check security roles to make sure users can only access data they should be permitted to.
- Escape any SQL (typically done by the SQL library) to avoid SQL injection. A major benefit for using an ORM, like Sequelize in this project, is that they make use of prepared statements, which is a technique to escape input in order to prevent SQL injection vulnerabilities. In June 2019, Snyk (a company focused on security tools for developers) discovered attack vectors that could lead to SQL injection. The Sequelize maintainers promptly released fixes for the affected versions.
- JSON Web Tokens (JWT’s) are only signed. They are not encrypted. Therefore, the payload can be examined on the client. This is by design. DO NOT put anything that should be private in the JWT payload unless you encrypt it first.
- Don't use a weak secret for you token service. The generator creates a strong one for you automatically. No need to change it.
- Password storage inside `@feathersjs/authentication-local uses bcrypt. We don't store the salts separately since they are included in the bcrypt hashes.
- By default, JWT's are stored in Local Storage (instead of cookies to avoid CSRF attacks. For JWT, we use the HS256 algorithm by default (HMAC using SHA-256 hash algorithm). If you choose to store JWT's in cookies, your app may have CSRF vulnerabilities.
You need Git >= v2.24.3 and, Docker Engine >= v18.06.0.
$ git clone https://github.com/josepcrespo/nodejs-realtime-rest-api.git &&
cd nodejs-realtime-rest-api &&
docker-compose build --no-cache --force-rm &&
docker-compose upThe project runs on http://localhost:3030/.
You can also check a live version running on Heroku. You can not run the tests, because is a production deployment. Anyway, you can check the current tests coverage at https://nodejs-realtime-rest-api.herokuapp.com/tests-coverage/ and, do everything else including the access to view the realtime status page for the manufacturing facilities.
The recommended way of installing the project is using the Docker approach. Anyway, you have the option to install it all locally.
Requirements
-
Git >=
v2.24.3. If your are a developer, you probably have Git already installed. If not, visit the official downloads page as it provides appropriate instructions for different operating systems. -
Node.js >=
v10.0.0. Feathers docs recommends to use the latest available version. The docs also recommend the use of Node Version Manager (on macOS or other Unix based operating systems). Other methods for installing Node.js are:
⇢⇢⇢⇢ Using the installer available on the official downloads page.
⇢⇢⇢⇢ If you develop with macOS, you can use the Homebrew package manager:
$ brew install node⇢⇢⇢⇢ If you develop with a Debian based operating system, the easiest way to install Node is using the Advanced Packaging Tool (a.k.a. APT). You need to install the node core and the package manager separately:
$ sudo apt install nodejs$ sudo apt install npm✌️ After a successful installation, the
node(nodejsif using a Debian flavored Linux distro) andnpmcommands should be available on the terminal and show something similar when running the following commands:
$ node --version
v14.0.0$ npm --version
v6.14.8- A MySQL compatible server (the project has been developed with MySQL, probably it works with MariaDB, but this scenario has not been tested). This project has been developed using the version
5.7. Versions greater than5.7have major changes on the authentication method used for connecting with the server that causes unnecessary headaches. For installing a MySQL server you have multiple options:
⇢⇢⇢⇢ Manually downloading and installing the appropriate installer (you should choose the product version and, the operating system) from the official downloads page.
⇢⇢⇢⇢ Using an all-in-one package that provides you a MySQL server like: XAMPP, WAMP or, MAMP.
⇢⇢⇢⇢ If you develop with macOS, you can use the Homebrew package manager:
$ brew install mysql@5.7⇢⇢⇢⇢ If you develop with a Debian based operating system, the easiest way to install a MySQL server is using APT:
$ sudo apt install mysql-server=5.7.29-0ubuntu0.18.04.1When the installation is done, take note of your MySQL server connection parameters, you need to know:
- A database user with permissions.
- The database user password.
- The IP address or domain name of the server.
- The port number where the service is exposed.
- A database named
fictional_motor_company.
When you have this parameters at hand, you need to edit the /config/default.json file. Find the following line on the file:
"mysql": "mysql://root:secret@mysql_server:3306/fictional_motor_company"and, change it accordingly to your local MySQL server connection parameters. The template is:
"mysql": "mysql://<user>:<password>@<ip_address>:<port_number>/<database_name>"Installation
Open a shell and navigate where you want to install the project. Then run:
$ git clone https://github.com/josepcrespo/nodejs-realtime-rest-api.gitEnter into the project root directory and install the dependencies:
$ npm install
⚠️ Make sure you are not currently running any other service on your host that can interfere with the Docker services, for instance, a MySQL server or, a web server like NGINX or Apache. Since, the domain names and/or ports used could coincide and ruin the proper functioning of the application.
⚠️ The data does not persist betweendocker-compose downand,docker-compose upcommand executions.
Requirements
-
Git >=
v2.24.3. If your are a developer, you probably have Git already installed. If not, visit the official downloads page as it provides appropriate instructions for different operating systems. -
Docker Engine >=
v18.06.0. Just visit the official Docker Desktop page and, download the appropriate version for your operating system.
Installation
Open a shell and navigate where you want to install the project. Then run:
$ git clone https://github.com/josepcrespo/nodejs-realtime-rest-api.gitMake sure Docker is running on your machine. Enter into the project root directory and run the following command for downloading the necessary Docker images and, building the Docker containers:
$ docker-compose build --no-cache --force-rm
⚠️ The process of downloading and, building can take a while depending on your internet connection download capacity and, the power of your development machine.
You don't need to worry about any dependencies because the project setup for Docker, installs everything you need to run the project.
Useful commands
If you changed something in the /docker-compose.yml or something from inside the /dockerfiles directory, you need to re-build the containers first. Run the following command on the root directory of the project:
$ docker-copose up --buildIf you want to stop the containers running the project, run this command on the root directory of the project:
$ docker-compose downThe API will be exposed at http://localhost:3030/ after executing anyone of the following commands.
- Development server
Navigate to the project root directory and run:
$ npm run devYou need to keep this shell open and, you can see if any log appears during the execution of the application.
- Production mode
Navigate to the project root directory and run:
$ npm run start- Development server
By default, the project runs in development mode when using Docker. Just navigate to the project root directory and run:
$ docker-compose upYou need to keep this shell open and, you can see if any log appears during the execution of the application.
- Production mode
You need to change the Dockerfile located at /dockerfiles/node_runtime/Dockerfile. Open it an chage this line:
CMD ["npm", "run", "dev"]
with this one:
CMD ["npm", "run", "start"]
and, finally:
$ docker-compose up -d --buildUsing the -d option, the server runs in background.
The project comes with only two users registered into the database (one with “admin” permissions and, the other one with basic “user” permissions). If you want to seed the database with dummy data, it can be easily done running the tests against the database used by the application.
Use the mysql connection string from /config/default.json on /config/test.json and then, run the tests.
Feathers provides "Login with GitHub" functionality using OAuth 2.0, out of the box without much effort.
OAuth is an open authentication standard supported by almost every major platform. It is what is being used by the login with Facebook, Google, GitHub and, all this kind of buttons in a web application. From the Feathers perspective the authentication flow is pretty similar. Instead of authenticating with the local strategy by sending a username and password, we direct the user to authorize the application with the login provider. If it is successful we find or create the user on the users service with the information we got back from the provider and issue a token for them.
After a successful login the third party provider (GitHub in our case), will redirect back the user to our application with a valid JWT or, an error message in other case.
In order to log in with GitHub, visit http://localhost:3030/oauth/github. You will be redirected to GitHub and asked to authorize the authentication into our application, using your GitHub account. If everything went well, you will see a JWT, valid for 24 hours, that you can use for making requests to the API endpoints that require authentication. Keep in mind that all users are created with "salesman" role permissions as default (this role only can create new sales with the /sales service).
Login with GitHub example:
Performing a curl request using the returned JWT after login with GitHub:
❗ You need to enter github.com and, logout from your session if you want to test the full "Login with GitHub" flow again. Or, you can just visit http://localhost:3030/oauth/github all the times you want to obtain a new valid JWT, GitHub will not ask for authorization since you already granted before.
❗ The authentication client will not use the token from the OAuth login if there is already another token logged in.
The project comes with three users already registered so you can easily start to test the API. First user comes with “admin” privileges (this user role can perform any operation with the API). Second user, only have “salesman” privileges (this user role only allows to create new sales with the /sales service). And, the third, only have “manufacturer” privileges (this user role only allows to get and find sales with the /sales service).
Admin user:
{
"email": "admin@fictionalMotor.com",
"password": "asdf1234",
"permissions": "admin"
}Salesman user:
{
"email": "salesman1@fictionalMotor.com",
"password": "asdf1234",
"permissions": "salesman"
}Manufacturer user:
{
"email": "manufacturer1@fictionalMotor.com",
"password": "asdf1234",
"permissions": "manufacturer"
}If you want to login with a user, you need to set the strategy property to local and, of course, provide valid credentials. Below is an example of a body that should be sent using the POST method to the /authorization API endpoint:
{
"strategy": "local",
"email": "admin@fictionalMotor.com",
"password": "asdf1234"
}and, here you have an example using curl command:
curl -X POST "http://localhost:3030/authentication" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"strategy\":\"local\",\"email\":\"salesman1@fictionalMotor.com\",\"password\":\"asdf1234\"}"
Feathers service methods that provide CRUD functionality are:
find: Find all data (potentially matching a query).get: Get a single data entry by its unique identifier.create: Create new data.update: Update an existing data entry by completely replacing it.patch: Update one or more data entries by merging with the new data.remove: Remove one or more existing data entries.
When used as a REST API, incoming requests get mapped automatically to their corresponding service method like this:
| Service method | HTTP method | Path |
|---|---|---|
| service.find({ query: {} }) | GET | /users |
| service.find({ query: { permissions: 'admin' } }) | GET | /users?permissions=admin |
| service.get(1) | GET | /users/1 |
| service.create(body) | POST | /users |
| service.update(1, body) | PUT | /users/1 |
| service.patch(1, body) | PATCH | /users/1 |
| service.remove(1) | DELETE | /users/1 |
The project comes with a full Swagger UI setup so, you can play with the API directly on the docs page. All Feathers services exposed by the API have their own documentation for each method, examples, live execution of queries and, their respective responses.
Swagger UI allows anyone — be it your development team or your end consumers — to visualize and interact with the API’s resources without having any of the implementation logic in place. It’s automatically generated from our OpenAPI (formerly known as Swagger) Specification, with the visual documentation making it easy for back end implementation and client side consumption.
You can visit http://localhost:3030/docs/swagger-ui.html to see it in action and, perform almost any operation available on the API or, just explore it. Just keep in mind that all endpoinds require authentication, so you need to provide your credentials using the “Authorize” button placed on the top right of the page.
Here you have an example of authenticating with local strategy and, retrieving a users list using Swagger UI:
⚠️ You need a local client for consuming APIs, such as Postman API Client or Insmonia Core for using the JWT provided by GitHub Oauth to authenticate API requests.
Click on the button below for importing a Postman Collection into your Postman API Client. The Collection is built with all the API endpoints so you can test everything the API exposes with this great tool.
The collection is called Fictional Motor Co REST API.
👀 Remember to set a valid JWT on the Authorization tab of a Postman request (if the request requires authentication) using the Bearer token option.
The project comes full of tests (not 100%, but close). Tests are done using the Mocha framework and the Node.js native Assert module. After running tests, Istanbul creates a comprehensive report of the test coverage, directly in the console and as an HTML page.
There is no tests for CRUD operations because all this functionallity is already tested by the Feathers framework internally. Only a few edge cases are tested, for example, when a request interacts somehow with a custom Feathers Hook.
Open the /config/test.json file. Find the following line on the file:
"mysql": "mysql://root:secret@mysql_server:3306/fictional_motor_company_tests"and, change it accordingly to your local MySQL server connection parameters. The template is:
"mysql": "mysql://<user>:<password>@<ip_address>:<port_number>/<database_name>"Move into your project’s root directory and run:
$ npm run test
⚠️ Remember to stop any services you may have running locally on your host machine to avoid unexpected behaviors and interferences with the Docker containers configurations.
Move into your project’s root directory.
Start the project's Docker containers if not running yet:
$ docker-compose upand, then:
$ docker exec -it nodejs-realtime-rest-api_node_server_1 npm run testAfter running the tests, you can see a coverage report, in plain text format, thanks to Istanbul. Here you can see the full output after running the tests, all passing fine:
Feathers application tests
✓ starts and shows the index page (95ms)
GitHub OAuth login
✓ The GitHub OAuth login page loads (871ms)
404 HTML status code responses
info: Page not found {"type":"FeathersError","name":"NotFound","code":404,"className":"not-found","data":{"url":"/path/to/nowhere"},"errors":{}}
✓ shows a 404 HTML page
info: Page not found {"type":"FeathersError","name":"NotFound","code":404,"className":"not-found","data":{"url":"/path/to/nowhere"},"errors":{}}
✓ shows a 404 JSON error without stack trace
authentication
✓ registered the `authentication` service
local strategy
✓ authenticates user and creates accessToken (166ms)
sales hook: create.process
✓ creates a `sale` and attaches the ID of the `user` who created him (160ms)
sales hook: create.validate
✓ Throws a BadRequest when tries to create a `sale` without `model`, `engine`, `doors`, `color`, `extras`. (159ms)
✓ Throws a BadRequest when tries to create a `sale` without `model`. (149ms)
✓ Throws a BadRequest when tries to create a `sale` without `engine`. (152ms)
✓ Throws a BadRequest when tries to create a `sale` without `doors`. (141ms)
✓ Throws a BadRequest when tries to create a `sale` without `color`. (144ms)
✓ Throws a BadRequest when tries to create a `sale` without `extras`. (157ms)
sales hook: patch.process
✓ updates a `sale` and attaches the ID of the `user` who updated him (180ms)
sales hook: patch.validate
✓ Throws a BadRequest when tries to update a `sale` with an empty `model` (154ms)
✓ Throws a BadRequest when tries to update a `sale` with an empty `engine` (170ms)
✓ Throws a BadRequest when tries to update a `sale` with an empty `doors` (161ms)
✓ Throws a BadRequest when tries to update a `sale` with an empty `color` (156ms)
✓ Throws a BadRequest when tries to update a `sale` with an empty `extras` (146ms)
users hook: create.validate
✓ Throws a BadRequest when tries to create a `user` without `githubId` and, `email`
✓ Throws a BadRequest when tries to create a `user` without `githubId` and, `password`
✓ Throws a BadRequest when tries to create a `user` without `githubId, `email` and, `password`.
users hook: get.validate
✓ A `user` without `admin` permissions can not get details from another user (403ms)
users hook: patch.validate
✓ Patches a `user` (375ms)
✓ A `user` with `admin` permisions can PATCH other `user` (320ms)
✓ A `user` with `user` permisions can not PATCH other `user` (282ms)
✓ Throws a BadRequest when tries to update a `user` with an empty `email` (144ms)
✓ Throws a BadRequest when tries to update a `user` with an empty `password` (146ms)
✓ Throws a BadRequest when tries to update a `user` with an empty `githubId` (137ms)
✓ Throws a BadRequest when tries to update a `user` with an empty `permissions` (141ms)
users hook: remove.validate
✓ A `user` with `admin` permissions can delete other `user` (287ms)
✓ A `user` can not delete himself (149ms)
sales
✓ registered the `sales` service
✓ creates a `sale` (151ms)
users
✓ registered the `users` service
✓ creates a `user` and, encrypts his `password` (138ms)
✓ removes `password` for external requests (257ms)
✓ creates a `user` with default permissions (154ms)
38 passing (7s)
-------------------------------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-------------------------------|---------|----------|---------|---------|-------------------
All files | 93.68 | 89.01 | 82.5 | 93.68 |
src | 87.5 | 33.33 | 60 | 87.5 |
app.hooks.js | 100 | 100 | 100 | 100 |
app.js | 100 | 100 | 100 | 100 |
authentication.js | 84.62 | 100 | 50 | 84.62 | 7-9
channels.js | 25 | 25 | 25 | 25 | 7-47
logger.js | 100 | 100 | 100 | 100 |
sequelize-to-json-schemas.js | 100 | 100 | 100 | 100 |
sequelize.js | 100 | 50 | 100 | 100 | 22
src/hooks/sales | 100 | 100 | 100 | 100 |
sales.create.process.js | 100 | 100 | 100 | 100 |
sales.create.validate.js | 100 | 100 | 100 | 100 |
sales.patch.process.js | 100 | 100 | 100 | 100 |
sales.patch.validate.js | 100 | 100 | 100 | 100 |
src/hooks/users | 100 | 92.5 | 100 | 100 |
users.create.validate.js | 100 | 90.91 | 100 | 100 | 11
users.get.validate.js | 100 | 85.71 | 100 | 100 | 15
users.patch.validate.js | 100 | 94.74 | 100 | 100 | 12
users.remove.validate.js | 100 | 100 | 100 | 100 |
src/middleware | 100 | 100 | 100 | 100 |
index.js | 100 | 100 | 100 | 100 |
src/models | 95.45 | 100 | 83.33 | 95.45 |
sales.model.js | 90.91 | 100 | 66.67 | 90.91 | 87
users.model.js | 100 | 100 | 100 | 100 |
src/services | 100 | 100 | 100 | 100 |
index.js | 100 | 100 | 100 | 100 |
src/services/sales | 100 | 100 | 100 | 100 |
sales.class.js | 100 | 100 | 100 | 100 |
sales.hooks.js | 100 | 100 | 100 | 100 |
sales.service.js | 100 | 100 | 100 | 100 |
src/services/users | 87.18 | 25 | 50 | 87.18 |
users.class.js | 100 | 100 | 100 | 100 |
users.hooks.js | 100 | 100 | 100 | 100 |
users.service.js | 80.77 | 25 | 33.33 | 80.77 | 42-47
-------------------------------|---------|----------|---------|---------|-------------------The project already comes with coverage reports in HTML format. Only run this command if you changed something in your tests and wants to publish the new reports. Coverage reports in HTML format can be re-published to the /public/tests-coverage/ directory, following this steps:
First move to your project’s root directory.
For running locally:
$ npm run publish-coverageor, alternatively using Docker, start the project's Docker containers if not running yet:
$ docker-compose upand, then:
$ docker exec -it nodejs-realtime-rest-api_node_server_1 npm run publish-coverageYou can view the output here http://localhost:3030/tests-coverage/.
- Getting started with Feathers
- Feathers API documentation
- Feathers cookbook
- Why Feathers uses JWT for sessions
- Better JSON errors with Feathers
- Role and permissions with Feathers
- Validation with Feathers
- HTTP vs Websockets: A performance comparison
- Dockerize a Feathers appliation
- Dockerize a Node.js application with MySQL (and dump)
- Dockerizing a Node.js Web Application
- How to crate a MySQL Instance with Docker Compose
- Docker Compose healthcheck
- Docker Compose wait for container X before starting container Y
- How to use Docker Volumes to code faster