Environment built on HotR.O.D. (Jaeger demo application) to produce trace data emulating a running distributed application, instrumented through the OpenTracing API and configurable to introduce issues in its behaviour at runtime. The detailed description can be found in the thesis elaborate Towards observability with (RDF) trace stream processing.
This basic application offers the user the possibility to make only one type of request interacting with the Frontend
service. Each request made crosses all four services:
Frontend
service provides a simple web-app to make HTTP GET requests to the/dispatch
endpoint. Clicking one of the four buttons available, each related to a specificcustomerId
, it is possible to ask for a driver to be dispatched at the customer location. The request is sent to the back-end service that calls all others micro-services and responds with the driver's license plate number and the expected time of arrival.Customer
service is called through anHTTP GET
call and it emulates a query to an SQL database returning data about the customer identified by the givencustomerId
:location
andcustomerName
.Driver
service exposes thefindNearest
service through Thrift over TChannel. Emulating a Redis database, it randomly generates ten drivers available in the area of the customer and for each of them returns thedriverId
(license plate number) and `locationDriverv.Route
service is called through anHTTP GET
call and computes the ETA (Estimated Time of Arrival) from the driver to the customer location.
The four microservices are instrumented with the OpenTracing API for GoLang and the Jaeger
tracer is configured to handle traces (sampling is not active). In order to exemplify the goal of distributed tracing some operations are intentionally misconfigured or not optimised: latencies of some operations are manually increased, some errors are randomly generated, and artificial bottlenecks are created (in red in the SD).
We changed the networking configurations in HotR.O.D. to enable each service to run as a different process on a different machine and thus to enable containerization of services. We build a unique Docker image for HotR.O.D. that can be used to run separately each service and a docker-compose file to instantiate HotR.O.D. with each service running on a different container.
To allow spawning more than one instance we made the docker-compose file configurable through environment variables. In this way it is possible to easily launch multiple instances of the application also on the same machine, reachable at different addresses and not clashing exploiting project namespace (-p
option of docker-compose
).
To allow spawning more than one instance we made the docker-compose file configurable through environment variables. In this way it is possible to easily launch multiple instances of the application also on the same machine reachable at different addresses and not clashing exploiting project namespace (-p
option of docker-compose
).
In order to enable scalability, we guarantee a "for-instance" scaling, i.e., to scale the application we can run another instance of all four services. Each instance is spawned and configured through the docker-compose
file and has its internal sub-network exposing only the frontend
endpoint. We opted for this solution because it ensures requests sent to a specific `frontend service are served only from micro-services of that instance, giving guarantees on the configurations provided for the given instance. To clarify this concept we show an example, let's suppose we want to launch two instances configured differently, we can execute the following commands:
HOTROD_INSTANCE="hotrod1" FIX_DB_QUERY_DELAY="--fix-db-query-delay=2ms" HOST_PORT_FRONTEND=8080 docker-compose -f hotrod-docker-compose.yml -p hotrod1 up
HOTROD_INSTANCE="hotrod2" HOST_PORT_FRONTEND=8090 docker-compose -f hotrod-docker-compose.yml -p hotrod2 up
In this example, the instance hotrod1
will have a simulated query delay in service customer
slower than the default value (300 ms) of instance hotrod2
. Since we do not allow to scale each service independently, we can ensure that each request sent to the endpoint :8080/dispatch
will have the query delay configured for hotrod1
, and each request sent to the endpoint :8090/dispatch
will have the default query delay configured for hotrod2
.
Rim also provides a way to generate requests on instances reproducibly and systematically. To this scope it uses:
- a Javascript library makerequests.js to generate in-browser load tests given a set of parameters,
- a UI integrated into the HotR.O.D. application to use the library, and
- a GoLang script, configurable with downloadable files produced by the library, to execute load tests on target instances.
We decided to build a Javascript library so that it can be integrated with the UI of the frontend
service. In this way, it is possible to avoid resorting to external tools for load testing.
The UI provided allow to choose the clickable object to make requests on, or a random selection (for each request made, between all clickable elements specified). Also, it allows to set a textual seed
to initiate the random pseudo-generator to guarantee reproducibility. The default method to generate requests is to specify a number of requests to be generated all at once. Otherwise, it is possible to select a distribution and provide parameters to determine how requests will be generated.
Modern browsers allow only for a limited set of concurrent requests, therefore, to generate a higher number of requests, we also implement an option to generate a GoLang file makeRequestsTimes.go
containing endpoint URLs and timing of requests to be executed. This file can be run together with the makeRequests.go
file provided to generate a set of go-routines executing requests as specified through the graphical interface. Moreover, the file can be easily tuned to implement more complex logic in the execution of the load test and to target multiple instances of the application.
The docker-compose
file provided in this repository uses a Docker image bounding the HotR.O.D. app and a customized jaeger-agent
to enable the deployment of Kaiju-TSP and Jaeger to process traces sampled.