bookinfo-extddbb

Service Mesh configuration for bookinfo sample application with external ratings database using an egress Gateway for routing TCP traffic - Egress TCP blog post.

In this example, the components used are as follows:

Prerequisites

OCP up and running.
DNS zone (external hosted zone in this example). Thus, I can use an alias instead of the external service name. The idea is to abstract the applications from the external service's name using the Service Entry object.
Openshift Service Mesh installed Openshift Service Mesh.
Egress configured in SMCP Egress config.

MySQL instances

Three MySQL instances are deployed outside the Mesh in the ddbb project: mysql-1, mysql-2 and mysql-3. Each mysql instance has a different rating number that will be consumed by the ratings application:

mysql-1: Ratings point equals 1.
mysql-2: Ratings point equals 5.
mysql-3: Ratings point equals 3.

MySQL instances have been deployed using BuildConfigs. A MySQL image is customized for this use case using the files located in the examples folder and pushed into the OCP registry. You can see the BuildConfig template that I have used here.

For this example you can use any MySQL instance.

Once the image has been created, it is time to deploy the three MySQL instances. As I said, you can use any MySQL instance but keep in mind that the scripts previously mentioned should be executed. I have used DeploymentConfig for deploying each MySQL, and a Template with custom values for each instance.

The MysqlTemplate uses a params.env file with custom values. Once the template has been processed, some files are created: Secret, Service, PVC and DeploymentConfig.

The command for processing the template and create the Openshift's objects is:

Create ddbb project

oc new-project ddbb

Create ImageStream

oc apply -f mysql-deploy/imagestream-mysql-1.yaml
oc apply -f mysql-deploy/imagestream-mysql-2.yaml
oc apply -f mysql-deploy/imagestream-mysql-3.yaml

Create secret with MySQL credentials used by buildconfig

oc create secret generic mysql-credentials-1 --from-env-file=./mysql-deploy/params.env
oc create secret generic mysql-credentials-2 --from-env-file=./mysql-deploy/params-2.env
oc create secret generic mysql-credentials-3 --from-env-file=./mysql-deploy/params-3.env

Update the mandatory fields in the buildconfig file and create it

oc apply -f mysql-deploy/buildconfig-mysql-1.yaml
oc apply -f mysql-deploy/buildconfig-mysql-2.yaml
oc apply -f mysql-deploy/buildconfig-mysql-3.yaml

Run buildconfig

oc start-build mysql-1
oc start-build mysql-2
oc start-build mysql-3

Now, the mysql-1, mysql-2 and mysql-3 images are available in the Openshift Internal Registry.

Deploy mysql-1

oc process -f mysql-deploy/mysql-template.yaml --param-file=mysql-deploy/params.env | oc create -f -

Deploy mysql-2

oc process -f mysql-deploy/mysql-template.yaml --param-file=mysql-deploy/params-2.env | oc create -f -

Deploy mysql-3

oc process -f mysql-deploy/mysql-template.yaml --param-file=mysql-deploy/params-3.env | oc create -f -

All the MySQL instances should be running in ddbb project.

Egress TCP using Service Entry. TCP routing from sidecar to egress and from egress to external service.

Explanation

Ratings application consumes external MySQL databases (Ratings config here). This application will connect to mysql.external host, which will be resolved by the Service Entry object. Then, the Service Entry object will route the traffic to two different external databases with different weight 80/20 (the application does not know that it is connecting to two different databases). Also, the ratings-custom application consumes a MySQL database too (Ratings-custom config here).

App diagram

The traffic flow is:

The sidecar intercept the request from the app container (ratings) to mysql.external.
The Virtual Service and Destination Rule objects route the request from the sidecar (bookinfo) to the egress Gateway (istio-system).
At this point, the Virtual Service and Service Entry objects resolve the endpoints and route the traffic through the egress Gateway.

Deploy Bookinfo application

It is time to deploy the bookinfo sample application. In this use case, only one bookinfo application is deployed in bookinfo project.

Create bookinfo project

oc new-project bookinfo

Add bookinfo project to Service Mesh

oc create -f ossm-config/smmr.yaml

Deploy bookinfo application

oc apply -f examples/bookinfo/bookinfo.yaml

Exposing the bookinfo application using the default ingresscontroller

Get the default ingress domain and replace the $EXTERNAL_DOMAIN variable in examples/bookinfo/bookinfo-gateway.yaml and examples/bookinfo/ocp-route.yaml files

oc -n openshift-ingress-operator get ingresscontrollers default -o json | jq -r '.status.domain'

Create Gateway, Virtual Services and Destination Rules.

oc apply -f examples/bookinfo/bookinfo-gateway.yaml
oc apply -f examples/bookinfo/destination-rule-all-mtls.yaml
oc apply -f examples/bookinfo/ocp-route.yaml

At this point, the bookinfo application is up and running, but ratings application is consuming the internal database instead of the MySQL deployed previously. The application is accessible from outside the cluster using the ingress gateway.

export GATEWAY_URL=$(oc get route bookinfo-bookinfo-gateway -n istio-system -o jsonpath='{.spec.host}')
curl $GATEWAY_URL/productpage -I

Exposing the bookinfo application using Ingress Controller sharding by using route labels

This step is optional. In case you want to use Ingress Controller sharding, you need to following the steps below:

Create the sharded ingress crontroller. For this, you must decide the dns domain that this router will be configured with and how the ingress controller will apply the sharding (route labels, namespace labels). i.e. apps-sharded.ocp.example.com. Set the domain in the (sharded Ingress Controller) and create it (in this case, the sharding is applied using route labels):

oc apply -f ocp/ingresscontroller.yaml

Check that routers have been deployed:

oc get pod -n openshift-ingress

Configure Gateway and Virtual Service:

oc apply -f examples/bookinfo/sharded-ingress/bookinfo-gateway.yaml
oc apply -f examples/bookinfo/sharded-ingress/ocp-route.yaml

Set external database as ratings database for bookinfo sample application

Deploy ratings application with MySQL configuration

oc process -f examples/bookinfo/bookinfo-ratings-v2-mysql.yaml --param-file=./examples/bookinfo/params.env | oc apply -f -

Route all the traffic destined to the reviews service to its v3 version and route all the traffic destined to the ratings service to ratings v2-mysql that uses the MySQL databases previously deployed.

oc apply -f examples/bookinfo/virtual-service-ratings-mysql.yaml

At this moment, the bookinfo application is trying to retrieve the ratings info from the external DDBB. As you can see, the ratings service is currently unavailable. Now, it's time to create the Istio objects to route the traffic through an egress Gateway in order to reach the external DDBB.

Create Istio objects

for file in ossm-tcp-egress/case-1/**/*.yaml; do oc apply -f $file; done

Once created the Istio objects, ratings service should works and retrieve data from the external database. Since the traffic is splitted 80/20 between two different databases, the data retrieved should be different if you run some requests to the application.

Add an additional bookinfo application

Istio's TCP traffic capabilites are more limited than HTTP. There is a topic created in Istio for using TCP Service Entries with the same port Istio topic and the result is not good. Istio uses only the port to identify the routing for TCP traffic if no addresses are set. So, what happens in bookinfo project if two applications are trying to connect to port 3306 and whose destination is different external databases?

I tried to solve this issue in bookinfo project using sourceLabels for identify where the traffic is coming from, and it works in bookinfo project, but a new problem is generated in istio-system project.

Using sourceLabels I am able to match the traffic from the different app containers, and this traffic is routed to the Egress Gateway's K8S Service located in istio-system, port 443 in this use case. At this point, I need to know where the traffic is coming from and where I want to route it, and it is impossible using TCP. For HTTP traffic it can be done using HTTPMatchRequest with any field, and for HTTPS traffic it can be done using TLSMatchAttributes with sniHosts, for instance.

The problem is solved setting a different port in the Egress Gateway, one port for each external database.

In summary:

Two bookinfo applications deployed in bookinfo project. App bookinfo will use the mysql-1 and mysql-2 external databases, and the App bookinfo-custom will use the mysql-3 external database.
Use sourceLabels in bookinfo project Virtual Service mysql-egress.
Add an additional port to the Egress Gateway (https-9443 port).
One Service Entry object per external database.

In this use case, the traffic flow is:

The sidecar intercept the request from the app container (ratings or ratings-custom) to mysql.external or mysql-3.external.
The Virtual Service and Destination Rule objects route the request from the sidecar (bookinfo) to the egress Gateway (istio-system).
At this point, the Virtual Service and Service Entry objects resolve the endpoints and route the traffic through the egress Gateway.

Deploy Custom Bookinfo application

It is time to deploy the custom bookinfo application. Now, two bookinfo applications will be running in bookinfo project.

Deploy custom bookinfo application

oc apply -f examples/bookinfo/custom/bookinfo-custom.yaml
oc apply -f examples/bookinfo/custom/bookinfo-gateway_custom.yaml
oc apply -f examples/bookinfo/custom/ocp-route-custom.yaml
oc apply -f examples/bookinfo/custom/destination-rule-all-mtls_custom.yaml

Deploy ratings application with MySQL configuration

oc process -f examples/bookinfo/custom/bookinfo-ratings-v2-mysql_custom.yaml --param-file=./examples/bookinfo/custom/params.env | oc apply -f -

oc apply -f examples/bookinfo/custom/virtual-service-ratings-mysql_custom.yaml

At this point, the bookinfo application is up and running and set with external database, but ratings application is not able to retrieve any data from mysql-3 instance.

export GATEWAY_URL=$(oc get route bookinfo-bookinfo-gateway-custom -n istio-system -o jsonpath='{.spec.host}')
curl $GATEWAY_URL/productpage -I

Now, it's time to create the Istio objects to route the traffic through an egress Gateway in order to reach the mysql-3 external DDBB.

Create Istio objects

for file in ossm-tcp-egress/case-2/**/*.yaml; do oc apply -f $file; done

Once created the Istio objects, ratings service should works and retrieve data from the mysql-3 external database. Bookinfo and bookinfo-custom applications are working properly now.

Cleanup

MySQL Instances

Delete MySQL DeploymentConfigs

oc process -f mysql-deploy/mysql-template.yaml --param-file=mysql-deploy/params.env | oc delete -f -
oc process -f mysql-deploy/mysql-template.yaml --param-file=mysql-deploy/params-2.env | oc delete -f -
oc process -f mysql-deploy/mysql-template.yaml --param-file=mysql-deploy/params-3.env | oc delete -f -

Delete BuildConfigs

oc delete -f mysql-deploy/buildconfig-mysql-1.yaml
oc delete -f mysql-deploy/buildconfig-mysql-2.yaml
oc delete -f mysql-deploy/buildconfig-mysql-3.yaml

Delete secrets

oc delete secret mysql-credentials-1
oc delete secret mysql-credentials-2
oc delete secret mysql-credentials-3

Delete ImageStreams

oc delete -f mysql-deploy/imagestream-mysql-1.yaml
oc delete -f mysql-deploy/imagestream-mysql-2.yaml
oc delete -f mysql-deploy/imagestream-mysql-3.yaml

Delete OCP project

oc delete project ddbb

Bookinfo

Delete Istio objects

for file in ossm-tcp-egress/case-1/**/*.yaml; do oc delete -f $file; done

Delete ratings-v2 app

oc process -f examples/bookinfo/bookinfo-ratings-v2-mysql.yaml --param-file=./examples/bookinfo/params.env | oc delete -f -

Delete ratings and reviews routing

oc delete -f examples/bookinfo/virtual-service-ratings-mysql.yaml

Delete Routing objects

oc delete -f examples/bookinfo/bookinfo-gateway.yaml
oc delete -f examples/bookinfo/destination-rule-all-mtls.yaml
oc delete -f examples/bookinfo/ocp-route.yaml

Delete Bookinfo app

oc delete -f examples/bookinfo/bookinfo.yaml

Bookinfo-custom

Delete Istio objects

for file in ossm-tcp-egress/case-2/**/*.yaml; do oc delete -f $file; done

Delete custom bookinfo application

oc delete -f examples/bookinfo/custom/bookinfo-custom.yaml
oc delete -f examples/bookinfo/custom/bookinfo-gateway.yaml
oc delete -f examples/bookinfo/custom/ocp-route-custom.yaml
oc delete -f examples/bookinfo/custom/destination-rule-all-mtls_custom.yaml

Delete ratings application with MySQL configuration

oc process -f examples/bookinfo/custom/bookinfo-ratings-v2-mysql_custom.yaml --param-file=./examples/bookinfo/custom/params.env | oc delete -f -

Delete ratings and reviews routing

oc delete -f examples/bookinfo/custom/virtual-service-ratings-mysql_custom.yaml

Remove bookinfo from project from Service Mesh Members

oc delete -f ossm-config/smmr.yaml

Delete OCP project

oc delete project bookinfo

fperearodriguez / bookinfo-extddbb

bookinfo-extddbb

Prerequisites

MySQL instances

Egress TCP using Service Entry. TCP routing from sidecar to egress and from egress to external service.

Explanation

App diagram

Deploy Bookinfo application

Exposing the bookinfo application using the default ingresscontroller

Exposing the bookinfo application using Ingress Controller sharding by using route labels

Set external database as ratings database for bookinfo sample application

Add an additional bookinfo application

Deploy Custom Bookinfo application

Cleanup

MySQL Instances

Bookinfo

Bookinfo

Bookinfo-custom

About

Languages