sustacek / client-extensions-deep-dive-devcon-2023

Client Extension Deep Dive Workshop Script

Before Workshop

Note that this setup might take as long as 15 minutes so please run these steps before the workshop!

If you'd like help with these commands you can reach out on this Liferay community Slack channel: #devcon-2023-client-extensions-101-workshop

1. Clone repo:

   git clone https://github.com/LiferayCloud/client-extensions-deep-dive-devcon-2023.git

2. Change into workspace

   cd client-extensions-deep-dive-devcon-2023

3. Initialize the bundle (this downloads dependencies so it might take a while)

   ./gradlew initBundle

4. Start DXP

   1. Make sure you have java8 or java11 installed and in the system path (java -version) to confirm

   2. Linux/Mac:

      ./bundles/tomcat-9.0.73/bin/catalina.sh run

   3. Windows:

      .\bundles\tomcat-9.0.73\bin\catalina.bat run

5. DXP should pop up automatically on http://localhost:8080

6. Log in with user [e-mail] and password test

   Note - if the page is unresponsive you might have to refresh the page once before logging in (known issue).

7. Change the password to something you can remember

8. Build all projects

   ./gradlew build

9. That's it! You can leave DXP running if you are about to complete the workshop exercise, or shut it down

Workshop Exercise

Introduction

What are Client Extensions again?

Client extensions are a generic mechanism for customizating or extending DXP which run outside of DXP.

The extensions are defined in a client-extension.yaml file where we specify their properties.

This config file is deployed to DXP in order to give it the configuration information necessary to communicate with our Client Extension. For secure communcation between DXP and your client extension, OAuth2 can be used easily by specifing oAuth2Application* type of client extension. To learn more about Client Extensions visit the reference documentation.

In this workspace we will use Client Extensions to build the following use case:

• A ticket management system

• Requirements:

  • Defines a Customized Data Schema
  • Applies the Corporate brand and style
  • Provides a Customized User Application
  • Implements Programmatic Documentation Referral

In the end it should look like the following image:

Defining a Customized Data Schema

Our domain model is Ticket but DXP doesn't have the concept of a Ticket. Traditionally, we would have used Service Builder to model this but today we will use the Objects feature of DXP to define it.

Picklist

We will start the definition of our domain model by creating some Picklists. A picklist is a predetermined list of values a user can select, like a vocabulary. We can use Picklists when modelling Objects where an attribute needs to be constrained to specific values. For instance; status, priority, region and so on.

The Picklists we need are already defined in the project client-extensions/list-type-batch.

This project's client-extension.yaml declares a client extension of type: batch (see Batch Client Extensions) which is used to import DXP resources without requiring us to write any code. Resources are exported from DXP's Import/Export Center (in the JSONT format required for client extensions) and placed in the project's batch directory. Note that batch engine data files are not generated by hand. However, they are intended to be editiable by humans.

Execute the following commmand from the root of the workspace to deploy the picklists:

./gradlew :client-extensions:list-type-batch:deploy

Watch the tomcat logs to see that the client extension deployed.

Object Definition

Now we can deploy our Ticket object defeinition which we have already definded for you and put into this project client-extensions/ticket-batch.

Again, this project's client-extension.yaml declares a client extension of type: batch. It's batch directory contains the batch engine data file where the Ticket object definition is defined.

Execute the following commmand from the root of the workspace to deploy the Ticket object:

./gradlew :client-extensions:ticket-batch:deploy

Watch the tomcat logs to see that the client extension deployed.

When defining a domain model using Objects a set of headless APIs are automatically generated for you without any additional effort.

You can view these APIs in DXP's built in headless API browser by following this link: Tickets Headless API

Action item: Please view the endpoints of the headless API now.

We created the first ticket by hand, but in the scenario where you have pre-existing data, you can import it using batch (several of these operations do need to be performed in order)

• Lets deploy some pre-existing tickets

./gradlew :client-extensions:ticket-entry-batch:deploy

• Now you can see these ticket entires in the DXP UI

We've acheived our first business requirement: Define a Customized Data Schema. Let's move onto the next.

Apply the Corporate brand and style

Most organizations, after some level of maturity, have established a brand and style which, ideally, is carried through each new project. There are a number of existing client extensions available to support this use case as opposed to traditional Theme module. These are a subset of the client extensions referred to collectively as Front-End Client Extensions.

The client-extensions/tickets-theme-css project's client-extension.yaml declares a client extension of type: themeCSS (see Theme CSS Client Extension) which is used to replace the two core CSS resources from the portal's OOTB themes without modifying DXP.

Execute the following commmand from the root of the workspace to deploy the tickets-theme-css project:

./gradlew :client-extensions:tickets-theme-css:deploy

Watch the tomcat logs to see that the client extension deployed.

At this point let's return to the main page of our site. Let's apply the tickets-theme-css to the home page as demonstrated in the following video:

We've acheived our second business requirement: Apply the Corporate brand and style. Let's move onto the next.

Provide a Customized User Application

Our next business requirement is to build a customized user application. Today in DXP, there are low code mechanisms for doing this which directly support objects but which are not yet enabled as client extensions. So today we are going to solve this using the Custom Element Client Extension which enables use to build portal applications based on HTML 5 Web Components. In this case, using React.

The project is the client-extensions/current-tickets-custom-element. This project is a Javascript project with a package.json file that has a .scripts.build property which allows it to be seamlessly integrated into the workspace build (the workspace handles this integration for you and even handles the precise Javascript build tool installation and initialization & build tasks like $pkgman install and $pkgman run build. Here in this workspace $pkgman is yarn out of preference only.)

While we inspect this project, let's take a short sidebar and consider the client-extension.yaml's assemble block (see Assembling Client Extensions).

The assemble block

Note that in each of the previous projects we did already have the assemble block but let's take a minute to review it here. As was eluded to in the previous paragraph the workspace build knows how to seamlessly integrate certain non-Gradle builds. This is true of most Front End client extensions. However it doesn't know what to include in the LUFFA.

assemble:
    - from: build/assets
      into: static

The assemble block allows you to declare what resources need to be included in the LUFFA.

Execute the following commmand from the root of the workspace to deploy the current-tickets-custom-element project:

./gradlew :client-extensions:current-tickets-custom-element:deploy

Watch the tomcat logs to see that the client extension deployed.

At this point let's return to the main page of our site. Let's remove the main grid section and add the current-tickets-custom-element in place of it as demonstrated in the following video

Note that this app uses the auto-generated Ticket headless APIs.

We've acheived our third business requirement: Provide a Customized User Application. Let's move onto the last.

Implement Programmatic Documentation Referral

Our last business requirement is to implement a business logic that will improve the speed of resolving tickets so that we can serve customers more efficiently using an programmatic strategy to assess ticket details and adding information directly for the customer and maybe reducing the amount of research support agents need to perform in order to resolve the issue.

The client-extensions/ticket-spring-boot project's client-extension.yaml declares a client extension of type: objectAction (see Object Action Client Extension) which enables Object event handler which is implemented as a REST endpoint to be registered in Liferay.

Before we proceed we will make one small change to one of the previously deployed client extensions and redeploy it. Edit the file client-extensions/ticket-batch/batch/ticket-object-definition.batch-engine-data.json.

On line 46 change the value of "active" from false to true. Save the file and then (re)execute the command:

./gradlew :client-extensions:ticket-batch:deploy

One small sidebar about this notion of redeployment. It is intended that all deployment operations from the workspace should be idempotent (or that redeployments should both be effective and not result in error). This is not only important as a mechanism to speed up iterative development, but as a means to move changes between environments; such as moving future changes from a DEV to UAT or UAT to PROD.

Back to the business logic.

Please take a moment to look at the file client-extensions/ticket-spring-boot/src/main/java/com/liferay/ticket/TicketRestController.java

The key takeaways should be that:

• the body of the request is the payload which contains all the information relevant to the object entry for which the event was triggered
• the endpoint receives and validates JWT tokens which are signed by DXP and issued specifically for the clientId provisioned for the OAuth2Application also specified in the client-extension.yaml using the client extension of type: oAuthApplicationUserAgent

In a separate terminal, execute the following commmand from the root of the workspace to deploy the ticket-spring-boot project and at the same time start the microservice:

(cd client-extensions/ticket-spring-boot/ && ../../gradlew deploy bootRun)

Watch the tomcat logs to see that the client extension deployed.

To witness that the microservice will not allow unauthorized requests run the following curl command in a separate terminal while the microservice is running:

curl -v -X POST http://localhost:58081/ticket/object/action/documentation/referral

Note the response returns an error.

Finally, return to the main page of our site and click the Generate a New Ticket button. Review the outcome and verify that:

1. a ticket was created
2. the documentation referrals are added

We've acheived our third business requirement: Implement Programmatic Documentation Referral.

Try making other changes to the projects and redeploying the changes. In the case of the microservice make sure not only to execute the deploy task but also to restart it after any changes.

Ticket cleanup with cron job (Extra credit)

Now that we can create tickets, at some point, we need to clean them up. Let's create a cron job that will delete all tickets that are marked 'done' or 'duplicate'. To do this we will use a spring boot application that when executed, it will connect to DXP using the generated headless/REST API for ticket objects using another type of client extension type: oAuthApplicationHeadlessServer. This type of OAuth2 application is using the client credentials flow and is associated with a special account defined for this purpose (the current default uses the instance admin). One thing that we need to know is that client credential flow in OAuth2 require both a client_id and client_secret, so there will be some additional steps to perform in order to get this working locally.

The client-extensions/ticket-cleanup-cron project's client-extension.yaml declares a client extension of type: oAuth2ApplicationHeadlessServer (see OAuth2ApplicationHeadlessServer Client Extension) which defines an OAuth2 Application using client credntials flow.

Execute the following command

./gradlew :client-extensions:ticket-cleanup-cron:deploy

Note: See tomcat log for when the client extension is deployed. Now the oAuthApplication has been created in DXP.

If this were an LXC deployment, the cron schedule is specified in the LCP.json and would be scheduled accordingly. Since we are using a local deployment, we will simulate the cron execution by executing the application ourself. However, since this is a client_credentials type of OAuth2 Application we must provide both the client_id and client_secret. In our sample the code already gets the client_id by looking it up via the external reference code. However, we must copy the secret from the DXP UI.

1. Go to the DXP UI and navigate to Control Panel > Security > OAuth 2 Administration
2. Select the Ticket Cleanup Oauth Application Headless Server application and click on the Edit button for the Client Secret field. Copy the value.
3. In the terminal run the following command:

./gradlew :client-extensions:ticket-cleanup-cron:bootRun --args='--ticket-cleanup-oauth-application-headless-server.oauth2.headless.server.client.secret=<PASTE_IN_CLIENT_SECRET>'

Note: when you run the application you should see a message about the number of tickets that were deleted.

2023-06-14 18:18:23.027  INFO 29047 --- [           main] c.l.t.TicketCleanupCommandLineRunner     : Amount of tickets: 11
2023-06-14 18:18:23.028  INFO 29047 --- [           main] c.l.t.TicketCleanupCommandLineRunner     : Deleting ticket: 44767
2023-06-14 18:18:23.134  INFO 29047 --- [           main] c.l.t.TicketCleanupCommandLineRunner     : Deleting ticket: 44795

LXC Deployment (Extra Credit)

In order to deploy to LXC, we need the following as requirements:

1. LXC extension environment with LCP credentials
2. Access to DXP Virtual Instance connected to the LXC extension enviroment

Assuming you have everything above, we can now deploy our extensions to LXC. The following steps will deploy the client extensions to LXC:

1. From root workspace run this command: ./gradlew clean build
2. Execute lcp login and enter your credentials
3. Execute lcp deploy --extension <path_to_cx_zip> and select the LXC environment for each client extension zip

First lets deplay the list-type-batch extension which is the first one we need to deploy since ticket-batch depends on it.

lcp deploy --extension client-extensions/list-type-batch/dist/list-type-batch.zip

In the LCP console logs for this extension wait until you see

Jun 16 16:53:26.429 build-58 [listtypebatch-vhp9k] Execute Status: STARTED
Jun 16 16:53:27.228 build-58 [listtypebatch-vhp9k] Execute Status: COMPLETED

Next lets deploy the ticket-batch extension

lcp deploy --extension client-extensions/ticket-batch/dist/ticket-batch.zip

In the LCP console logs for this extension wait until you see

Jun 16 16:59:24.734 build-59 [ticketbatch-cnhtt] Execute Status: STARTED
Jun 16 16:59:25.532 build-59 [ticketbatch-cnhtt] Execute Status: COMPLETED

Now that we have deployed both of the batch type extensions, lets verify in the DXP UI that our object has been imported.

1. Go to the DXP UI and navigate to Control Panel > Object > Objects
2. Verify that the Ticket object is listed

Next we can deploy both of the frontend client extensions at the same time.

lcp deploy --extension client-extensions/current-tickets-custom-element/dist/current-tickets-custom-element.zip
lcp deploy --extension client-extensions/tickets-theme-css/dist/tickets-theme-css.zip

Since these are frontend client extensions, the resources will be loaded by the browser, so we need to make sure the client extension workloads (a Caddy fileserver) are visible on the network (which means the dns entries and global loadblancer will resolve the requests). You can view this using the network tag of the LCP Console:

https://console.liferay.cloud/projects/<your_ext_project>/network/endpoints

Wait until you see both the ingress endpoints are green.

Now we can deploy the microservice client extension.

lcp deploy --extension client-extensions/tickets-spring-boot/dist/tickets-spring-boot.zip

If it isn't working, see the troubleshooting section down below. If it is working you should see the servie available and in the logs you should see a message like this:

Jun 16 17:46:26.730 build-65 [ticketspringboot-74fcf56d76-tll5v] 2023-06-16 22:46:26.729  INFO 8 --- [           main] rayOAuth2ResourceServerEnableWebSecurity : Using client ID id-99677fc4-b15d-5968-4a1b-88e63897f9

This means your microservice is correctly talking with DXP and will be able to verify JWT tokens.

Self-Hosted (local tomcat) Troubleshooting

Batch deployment throws error

If you deploy the batch client extension to the local tomcat/osgi/client-extensions or dockerDeploy before you start the server, you may see an error when it tries to process the batch client extension. This is a known issue where the batch client extension is processed too soon by the headless batch import process. To fix this, simply reploy the batch client extension using gradlew deploy again.

Batch Order is not correct

If you try to deploy ticket-batch or ticket-entry-batch client extensions before you deploy the list-type-batch this will result in an error because ticket-batch depends on list-type-batch resources that must be deployed first. This is a known issue that will be addressed in the future.

OAuth2 Scopes are not applied

If you receive a HTTP 401 error or 403 not allowed, this may be because the OAuth2 scopes were not properly applied. To fix this you must edit the OAuthApplication in the DXP control panel UI and go to the "Scopes" tab and make sure the scopes that you are expected to be set, have indeed be set. In this example application is ths Ticket User Agent application and the Scopes that should be set are the C_Ticket.everything

LXC Troubleshooting

Here are some possible problems you may run into when deploying to LXC and how to try to troubleshoot them.

Spring boot microservice not starting (no logs show)

Possible error in DXP

If you do not see your microservice client extension is starting (lcp deployment never finishes), it is likely because DXP did not process your client-extension configuration correctly or had some error. You can check the DXP logs to see if there is an error processing your client extension configuration.

Possible error in DXP server configuration

It is possible that the DXP environment in the cloud is not configured correctly, namely the DXP virtual instance may work in the UI but the headless apis are not working, perhaps because of some middleware. Ensure that the /o/oauth2 headless apis are working by executing the following command:

curl https://dxp-env.lfr.cloud/o/oauth2/jwks

This should return the JSON Web Key Set (JWKS) for the DXP environment. If it does not, then the headless apis are not working and you will need to troubleshoot the DXP environment.

{"keys":[{"kty":"RSA","kid":"authServer","alg":"RS256","n":...}]}

Here you could use the internal diagnostics tool to try to determine why the microservice is not starting once it is generally available.

Spring Boot microservice starts but is killed (not enough memory)

If you the LCP console logs for the spring-boot microservice you see that is starts, but it shows that the spring-boot process is being killed like this:

Jun 16 17:22:22.897 build-62 [ticketspringboot-7c9d7f4999-pqcv2] [INFO  tini (1)] Spawned child process '/usr/local/bin/liferay_jar_runner_entrypoint.sh' with pid '7'
Jun 16 17:22:22.897 build-62 [ticketspringboot-7c9d7f4999-pqcv2] [INFO  tini (1)] Main child exited with signal (with signal 'Terminated')

It could be because the pod does not have enough memory. Edit the client-extensions/ticket-spring-boot/LCP.json and set the memory to a higher amount and redploy.

./gradlew :client-extensions:ticket-spring-boot:build
lcp deploy --extension client-extensions/tickets-spring-boot/dist/tickets-spring-boot.zip

Spring boot microservice starts but is killed (/ready endpoint not available)

If the spring boot microservice is starting but is immediately killed, you may see a message like this:

Jun 16 17:22:22.897 build-62 [ticketspringboot-7c9d7f4999-pqcv2] [INFO  tini (1)] Spawned child process '/usr/local/bin/liferay_jar_runner_entrypoint.sh' with pid '7'
Jun 16 17:22:22.897 build-62 [ticketspringboot-7c9d7f4999-pqcv2] [INFO  tini (1)] Main child exited with signal (with signal 'Terminated')

This may be because LCP could not detect that the service was ready. Review the LCP.json and notice the /ready path. Ensure that this path is able to respond to the platform within the specified timeout.