gilmaicor / history

History and Persister services

The History and Persister services can be considered as "siblings", History has the role of retrieving the historical data that was recorded in MongoDB, but in order for these data to reach MongoDB it was necessary for Persister to take them there.

To better understand the scenario where these two services are employed, let's exemplify the data flow:

1. The devices send their measurement data to the dojot platform through some protocol, for example, MQTT;
2. The data reaches a Broker* designed to work with the same protocol as the device;
3. After receiving the data, the Broker forwards it to Kafka so that the interested services can consume it;
4. The Persister service comes on the scene, which is notified by Kafka when there is device data to be persisted in MongoDB to create a historical basis of measurements;
5. Once Persister has saved the data to MongoDB, the History service is able to see it;
6. History service provides an endpoint (REST API) so that platform users can consult the historical data of measurements made by the devices, so through the History it is possible to retrieve any historical data.

(*) A Broker is a component that allows a group of devices to send their data and be managed by it, using their own native protocols.

Table of Contents

• History and Persister services
• Table of Contents
• History service
  • History service configuration
  • How to install History service
    • History service in standalone mode
    • History service on Docker
  • How to run the History service
• Persister service
  • Persister service configuration - Kafka related configuration - MongoDB related configuration
  • How to install Persister service
    • Persister service in standalone mode
    • Persister service on Docker
  • Running the Persister service
• Tests
• Dependencies
  • Dojot Services
  • Others Services
• Documentation
  • Generate API documentation
• Issues and help

History service

The History service is used when it is necessary to obtain historical data from devices. Through its REST interface it is possible to apply filters to obtain only the desired data from a device, like the attributes to be returned, the time period in which the data was measured, the quantity and order in which the data was returned.

See the skeleton of the request that can be made to History service:

http://{host}:{port}/device/{device_id}/history?attr={attr}&dateFrom={dateFrom}&dateTo={dateTo}&lastN|firstN={lastN|firstN}

History service configuration

The settings are made through environment variables, so we have the following:

Environment variable	Purpose	Default Value
AUTH_URL	Auth url address	"http://auth:5000"
DEVICE_MANAGER_URL	Device Manager url address	"http://device-manager:5000"
HISTORY_DB_ADDRESS	History database's address	"mongodb"
HISTORY_DB_PORT	History database's port	27017
HISTORY_DB_REPLICA_SET	History database's replica set address	None
LOG_LEVEL	Sets the log level	"INFO"

---

How to install History service

History service in standalone mode

The service is written in Python 3.6 and to run it in standalone mode you need to install the libraries that the service depends on. A very common problem is when we need to use different versions of the same library in different Python projects. This can lead to conflicts between versions and a lot of headaches for the developer. To solve this problem, the most correct thing is to create a virtual environment for each project. Basically, a virtual environment packages and stores in a specific directory all the dependencies that a project needs, making sure that no packages are installed directly on the operating system. Therefore, each project can have its own environment and, consequently, its libraries in specific versions.

There are several tools that create Python virtual environments, one of the most well-known is virtualenv. This is what we are going to use in this example.

We will use pip (Package Installer for Python) to install virtualenv, in this case, we will use pip in version 19.0.3.

Make sure to install the pip according to your operating system, then install virtualenv. That done, we will be able to create and manage our virtual environment to run the service.

First let's create a new virtual environment using the following command:

# The most common is to create virtualenv at the
# root of the project that it will belong to.
$> virtualenv ./venv

After creating a virtual environment, we need to activate it so that we can install the necessary project packages. For this, we use the following command:

$> source ./venv/bin/activate

NOTE THAT to exit the virtual environment we use the deactivate command, as we can see below:

# When you no longer want to run the History service,
# you need to leave the virtual environment created for it:
$> deactivate ./venv

To install all dependencies, execute the following commands.

# Some libraries depend on the specific version of the pip:
$> python -m pip install pip==19.0.3

$> pip install versiontools

$> pip install -r ./requirements/requirements.txt

The next and last command will generate a .egg file and install it into your virtual environment:

$> python setup.py install

History service on Docker

Another alternative is to use Docker to run the service. To build the container, from the repository's root:

# you may need sudo on your machine:
# https://docs.docker.com/engine/installation/linux/linux-postinstall/
$> docker build -t <tag-for-history-docker> -f docker/history.docker .

NOTE THAT you can use the official images provided by dojot in its DockerHub page.

---

How to run the History service

Beforehand, you need an already running dojot instance in your machine. Check out the dojot documentation for more information on installation methods.

To run the History service on standalone mode, just set all needed environment variables and execute:

# from the repository's root:
$> ./docker/falcon-docker-entrypoint.sh start

To run the service on docker, you can use:

# may need sudo to run
$> docker run -i -t <tag-for-history-docker>

Persister service

The History service (as we saw earlier) is only capable of retrieving data from the MongoDB instance, it is not able to store any data. It is the Persister's duty to make this happen: it is notified whenever a new message has arrived in Kafka and stores it in MongoDB.

Persister service configuration

The settings are made through environment variables, so we have the following:

Environment variable	Purpose	Default Value
AUTH_URL	Auth url address	"http://auth:5000"
DATA_BROKER_URL	Data Broker address	"http://data-broker"
DEVICE_MANAGER_URL	Device Manager address	"http://device-manager:5000"
LOG_LEVEL	Define minimum logging level	"INFO"
PERSISTER_PORT	Port to be used by persister sevice's endpoints	8057

Kafka related configuration

Environment variable	Purpose	Default Value
DOJOT_SERVICE_MANAGEMENT	Global service to use when publishing dojot management events	"dojot-management"
DOJOT_SUBJECT_TENANCY	Global subject to use when publishing tenancy lifecycle events	"dojot.tenancy"
DOJOT_SUBJECT_DEVICES	Global subject to use when receiving device lifecycle events	"dojot.device-manager.device"
DOJOT_SUBJECT_DEVICE_DATA	Global subject to use when receiving data from devices	"device-data"
KAFKA_ADDRESS	Kafta address	"kafka"
KAFKA_PORT	Kafka port	9092
KAFKA_GROUP_ID	Group ID used when creating consumers	"history"

MongoDB related configuration

Environment variable	Purpose	Default Value
HISTORY_DB_ADDRESS	History database's address	"mongodb"
HISTORY_DB_PORT	History database's port	"27017"
HISTORY_DB_REPLICA_SET	History database's replica set address	None
HISTORY_DB_DATA_EXPIRATION	Time (in seconds) that the data must be kept in the database	"604800" (7 days)
MONGO_SHARD	Activate the use of sharding or not	False

---

How to install Persister service

Persister service in standalone mode

The Persister service is installed in exactly the same way as the History service, just note that what changes is the virtual environment to be created, in this case, the directory name of the virtual environment must refer to the Persister service. If you want, you can also use the same virtual environment as History service, no problem, but be aware of that.

Persister service on Docker

Like the standalone mode, the Persister on Docker is identical to the History service's one, paying attention only to the name of the docker image to be generated. In this case, it must refer to the Persister service.

NOTE THAT you can use the official images provided by dojot in its DockerHub page.

---

Running the Persister service

Beforehand, you need an already running dojot instance in your machine. Check out the dojot documentation for more information on installation methods.

To run the Persister, after setting the environment variables, execute:

$> python -m history.subscriber.persister

Tests

History has some automated test scripts. We use [Dredd] (http://dredd.org/en/latest/) to execute them:

# you may need sudo for this:
$> npm install -g dredd@5.1.4 --no-optional

Also install the python test dependencies:

$> pip install -r ./tests/requirements.txt

$> pip install codecov

Run Dredd automated test scripts:

$> ./tests/start-test.sh

Run the pytest module to run the coverage tests:

$> python -m pytest --cov-report=html --cov=history tests/

Dependencies

The service dependencies are listed in the next topics.

• Dojot Services: They are dojot services;
• Others Services: They are external services;

Dojot Services

• Auth;
• Device Manager;
• Data Broker;

Others Services

• Kafka (tested using Kafka version 2.12);
• MongoDB (tested using MongoDB version 3.2);

Documentation

Check the documentation for more information:

• Latest History API documentation
• Development History API documentation
• Latest dojot platform documentation

Generate API documentation

The API documentation for this service is written as API blueprint. To generate a simple web page from it, you must execute the commands below:

# you may need sudo for this:
$> npm install -g aglio

# static webpage
$> aglio -i ./docs/history.apib -o ./docs/history.html

# serve apis locally
$> aglio -i ./docs/history.apib -s

Issues and help

If you found a problem or need help, leave an issue in the main dojot repository and we will help you!