christophermoura / superbowleto

:football: A microservice to issue, register and manage boletos

superbowleto

🏈 A microservice to issue, register and manage boletos

Table of Contents

• Technology
• Developing
  • First Install
  • Running tests
  • Installing new dependencies
• Testing
• Lambdas and Data Flow
  • 1. Boleto: create
    • a) Provider could process the boleto
    • b) Provider could not process the boleto
  • 2. Boleto: index
  • 3. Boleto: show
  • 4. Boleto: process boletos-to-register queue
  • 5. Boleto: register

Technology

Here's a brief overview of our technology stack:

• Docker and Docker Compose to create our development and test environments.
• Serverless Framework to manage and deploy our AWS Lambda functions.
• AWS SQS as a queue manager to process things like boletos to register.
• Postgres as to store our data Sequelize as a Node.js ORM.
• Babel to transpile our code written in modern Javascript and we use multiple Webpack configurations to bundle our code for production, test and development.
• Ava as a test runner and Chai to do some more advanced test assertions.
• Yarn to install npm dependencies.

Developing

In order to develop for this project you must have Docker and Docker Compose installed.

First Install

If you never developed in this repo before:

1. Clone the repository:

$ git clone [e-mail]:pagarme/superbowleto

2. Build the base image:

$ docker-compose build test

Running tests

Tests are separate in functional, integration and unit. You can either run them separately or run them all.

• Run all tests:

  $ docker-compose run test

• Run only functional tests:

  $ docker-compose run test yarn run test-functional

• Run only integration tests:

  $ docker-compose run test yarn run test-integration

• Run only unit tests:

  $ docker-compose run test yarn run test-unit

Installing new dependencies

We install our dependencies (aka npm dependencies) inside the Docker image (see our Dockerfile to understand it better).

This gives us the advantage of caching the dependencies installation process, so when we build the image again, it's already cached by Docker and the image can be easily distributed with all its dependencies installed.

However, if you need to install any new dependency, you must rebuild the image, otherwise, your dependency will not be available inside the container.

You can install dependencies and rebuild the image by running:

$ docker-compose run test yarn add jquery
$ docker-compose build test

Testing

Tests are found inside the test/ directory and are separate by type: functional, integration and unit. It's also common to have some helpers folders alongside the tests.

• Unit tests are used to test the smallest units of functionality, typically a method or a function ^ref.

  The folder structure of the unit tests tend to mirror the folder structure of the src folder. For instance, we generally see the following folder structure:

  ├── src
  │   ├── index.js
  │   └── lib
  │       └── http.js
  └── test
      └── unit
          ├── index.js
          └── lib
              └── http.js
  

• Integration tests build on unit tests by combining the units of code and testing that the resulting combination functions correctly^ref.

  The folder structure of the integration tests tend to mirror the folder structure of the src folder. For instance, we generally see the following folder structure:

  ├── src
  │   ├── index.js
  │   └── lib
  │       └── http.js
  └── test
      └── integration
          ├── index.js
          └── lib
              └── http.js
  

• Functional tests check a particular feature for correctness by comparing the results for a given input against the specification. Functional tests don't concern themselves with intermediate results or side-effects, just the result^ref.

  The folder structure of functional tests does not need to mirror the source folder, and the files can be organized as they seem fit. One way to organize this files is by feature or user-story. For instance, take a look at the example below, where boleto/create.js and boleto/register.js are complete user stories:

  ├── test
      └── functional
          └── boleto
              └── create.js
              └── register.js
  

• Helpers do not test anything, but instead provide tools for the tests. Inside the helpers folders one can have fixtures (also know as "mocks"), or some util functions.

  For instance, if you need credit card information to perform various tests in many different places, or if you need an util function that is called before your tests are ran, you could place them inside a helpers folder in order to not repeat yourself:

  export const creditCardMock = {
    number: 4242424242424242,
    holder_name: 'David Bowie',
    expiration_date: 1220,
    cvv: 123,
  }
  
  export const cleanUpBeforeTests = () => {
    db.reset();
  }

  Helpers folders can be created at any level within the test folder structure. If some helper is used only for unit tests, it should reside within test/unit/helpers. If the helpers is used across all tests, it should reside within test/helpers. If there's a helper that is used only for testing the http module on integration tests, then it should reside within test/integration/http/helpers.

Lambdas and Data Flow

This project is designed in AWS Lambda functions and has some logical branches and business domain rules that can be hard to understand. This section documents what every Lambda function does how it fits in the entire project.

1. Boleto: create

Create a new boleto.

After creating the boleto (on our database), we will try to register the boleto withing the provider. Here, there's two possible outcomes: a) the provider could be reached, could process the boleto and gave us a status (either registered or refused); or b) the provider could not be reached or could not process the boleto (giving us an unknown/undefined/try_later status).

a) Provider could process the boleto

The following steps illustrate the case where the provider could be reached and it could process the boleto.

1. The Client makes an HTTP request to create a boleto.
2. We create the boleto in the Database with status issued.
3. We try to register the boleto within the Provider.
4. The provider returns an answer (either registered or refused).
5. We update the boleto status in the Database.
6. We return the response to the Client (HTTP response).

Diagram built with mermaid.js. Check out the source code at docs/diagrams

b) Provider could not process the boleto

The following steps illustrate the case where the provider could be reached and it could process the boleto.

1. The Client makes an HTTP request to create a boleto.
2. We create the boleto in the Database with status issued.
3. We try to register the boleto within the Provider.
4. The provider could not be reached or could not process the boleto.
5. We update the boleto status in the Database to pending_registration.
6. We send the boleto (boleto_id and issuer) to an SQS queue called boletos-to-register. This queue will be processed by another Lambda later.
7. We return the response to the Client (HTTP response) with the status = pending_registration.

Diagram built with mermaid.js. Check out the source code at docs/diagrams

Example:

POST /boletos

Content-Type: application/json

{
  "queue_url": "http://yopa/queue/test",
  "expiration_date": "Tue Apr 18 2017 18:46:59 GMT-0300 (-03)",
  "amount": 2000,
  "instructions": "Please do not accept after expiration_date",
  "issuer": "bradesco",
  "payer_name": "David Bowie",
  "payer_document_type": "cpf",
  "payer_document_number": "98154524872"
}

201 Created

Content-Type: application/json

{
  "queue_url": "http://yopa/queue/test",
  "status": "issued | registered | refused",
  "expiration_date": "Tue Apr 18 2017 18:46:59 GMT-0300 (-03)",
  "amount": 2000,
  "instructions": "Please do not accept after expiration_date",
  "issuer": "bradesco",
  "issuer_id": null,
  "title_id": "null",
  "payer_name": "David Bowie",
  "payer_document_type": "cpf",
  "payer_document_number": "98154524872"
}

2. Boleto: index

Retrieve all boletos.

Diagram built with mermaid.js. Check out the source code at docs/diagrams

Example:

GET /boletos

Content-Type: application/json

{
  "count": "10",
  "page": "1"
}

200 Ok

Content-Type: application/json

[{
  "id": "bol_cj1o33xuu000001qkfmlc6m5c",
  "status": "issued",
  "queue_url": "http://yopa/queue/test",
  "expiration_date": "Tue Apr 18 2017 18:46:59 GMT-0300 (-03)",
  "amount": 2000,
  "instructions": "Please do not accept after expiration_date",
  "issuer": "bradesco",
  "issuer_id": null,
  "title_id": "null",
  "payer_name": "David Bowie",
  "payer_document_type": "cpf",
  "payer_document_number": "98154524872"
}]

3. Boleto: show

Find one boleto by id.

Diagram built with mermaid.js. Check out the source code at docs/diagrams

Example:

GET /boletos/:id

Content-Type: application/json

{
  "id": "bol_cj1o33xuu000001qkfmlc6m5c"
}

200 Ok

Content-Type: application/json

{
  "id": "bol_cj1o33xuu000001qkfmlc6m5c",
  "status": "issued",
  "queue_url": "http://yopa/queue/test",
  "expiration_date": "Tue Apr 18 2017 18:46:59 GMT-0300 (-03)",
  "amount": 2000,
  "instructions": "Please do not accept after expiration_date",
  "issuer": "bradesco",
  "issuer_id": null,
  "title_id": "null",
  "payer_name": "David Bowie",
  "payer_document_type": "cpf",
  "payer_document_number": "98154524872"
}

4. Boleto: process boletos-to-register queue

Process the boletos-to-regiter-queue. This Lambda function is triggered by the Schedule Event (runs every n minutes).

When a boleto can't be registered within the provider at the moment of its creation, it will be posted to a SQS Queue called boletos-to-register. This function is responsible for processing this queue. Here are the steps:

1. This function is triggered by a Schedule Event that runs every n minutes.
2. Using sqs-quooler we then start to poll items from SQS (sqs.receiveMessage)
3. For each message received we invoke a Lambda function called register with the boleto information and the SQS Message as parameters.
4. sqs-quooler package will repeat this pollItems->receive->invokeRegisterLambda cycle until this Lambda dies (max 5 minutes).

Diagram built with mermaid.js. Check out the source code at docs/diagrams

5. Boleto: register

Register a boleto. This Lambda function is invoked by another Lambda function: process boletos-to-register queue.

This function registers boletos that went to the boletos-to-register queue. Here are the steps of execution:

1. The function is invoked by the queueProcessor and receives boleto ({ id, issuer}) and message (the raw SQS message from boletos-to-register queue).
2. We use the boleto id to find the boleto on Database.
3. We check if the boleto can be registered, i.e. if the status of the boleto is either issued or pending_registration.
4. If the boleto can be registered, we try to register it within the provider.
5. If the provider could not process the boleto, we stop executing here. The SQS Message will then go back to boletos-to-register queue and will be later processed.
6. We update the boleto status with either registered or refused.
7. IMPORTANT: After the boleto is updated, we notify the boleto owner by sendin an SQS message to the queue the owner specified on the boleto creation (aka: boleto.queue_url). The owner will then handle the processing of these SQS Messages. That's the only way we can notify the boleto owner that a boleto that went to boletos-to-register queue was updated. That's why it's mandatory to pass a queue at the moment of the boleto creation.

Diagram built with mermaid.js. Check out the source code at docs/diagrams