E2E Test Suite

TestNG based Java test suite focused on e2e testing application services running in the cloud.

Requirements

java jdk 11
maven >= 3.3.1
kcat (only to run Kcat tests)

Build and check checkstyle

mvn install -DskipTests

Executing tests

Running full test suite

mvn test

Running single test

mvn verify -Dit.test=ClassName

Running subset tests defined by maven profile

mvn verify -Psmoke

Writing tests

Test method must be annotated with annotation @Test.
Every new test class must extend TestBase
Environment variables are stored in static Environment class
Every new and variable used in testsuite must be added to table or env variables
The test class must be included in one of the TestNG suites in suites/

Environment Variables

This environment variables are used to configure tests behaviour, the ENVs marked with the double asterisk (**) are required to run all tests. But not all tests require all ENVs, if you want to run a single test you can check ahead the required variables in the test sourcecode javadoc.

Environment variables can also be configured in the config.json file.

Name	Description	Default value
`CONFIG_FILE`	relative or absolute path to the config.json	$(pwd)/config.json
`LOG_DIR`	path where the test suite will store additional collected logs and artifacts	$(pwd)/target/logs
`PRIMARY_USERNAME`	** https://sso.redhat.com username with quota to provision the tested service (See Create test users if you don't have one)
`PRIMARY_PASSWORD`	** primary user password
`SECONDARY_USERNAME`	** https://sso.redhat.com username in the same organization as the primary user
`SECONDARY_PASSWORD`	** secondary user password
`ALIEN_USERNAME`	** https://sso.redhat.com username in a different organization respect the primary user
`ALIEN_PASSWORD`	** alien user password
`OPENSHIFT_API_URI`	the base URI for the application services mgmt APIs (See Test Environments)	`https://api.stage.openshift.com`
`REDHAT_SSO_URI`	users authentication endpoint for application services mgmt APIs	`https://sso.redhat.com`
`REDHAT_SSO_LOGIN_FORM_ID`	HTML `id` value of the login `<form>` the SSO application will present after redirect	`#rh-password-verification-form`
`OPENSHIFT_IDENTITY_URI`	users authentication endpoint for application services instances APIs (See Test Environments)	`https://identity.api.stage.openshift.com`
`OPENSHIFT_IDENTITY_LOGIN_FORM_ID`	HTML `id` value of the login `<form>` the SSO application will present after redirect	`#rh-password-verification-form`
`DEV_CLUSTER_SERVER`	** the API server URI of a OpenShift cluster with the binding operator installed
`DEV_CLUSTER_TOKEN`	** the cluster user or service account token
`DEV_CLUSTER_NAMESPACE`	the namespace where to create test resources (See Create test namespace on the dev cluster)	`mk-e2e-tests`
`CLI_VERSION`	the CLI version to download from the app-services-cli repo	`latest`
`CLI_PLATFORM`	windows/macOS/linux	`auto-detect`
`CLI_ARCH`	the CLI arch and os to download from the app-services-cli repo	`amd64`
`CLI_EXCLUDE_VERSIONS`	a regex that if match will exclude the the CLI versions matching it while looking for the latest CLI release	`alpha`
`GITHUB_TOKEN`	the github token used to download the CLI if needed
`LAUNCH_KEY`	A string key used to identify the current configuration and owner which is used to generate unique name and identify the launch	`change-me`
`SKIP_TEARDOWN`	Skip the whole test teardown in most tests, although some of them will need top re-enable it to succeed	`false`
`SKIP_KAFKA_TEARDOWN`	Skip only the Kafka instance cleanup teardown in the tests that don't require a new instance for each run to speed the local development	`false`
`DEFAULT_KAFKA_REGION`	Change the default region where kafka instances will be provisioned if the test suite doesn't decide otherwise	`us-east-1`
`KAFKA_INSECURE_TLS`	Boolean value to indicate whether the Kafka and Admin REST API TLS is insecure (for self-signed certificates)	`false`
`KAFKA_INSTANCE_API_TEMPLATE`	URL template for the Kafka Admin REST API. May be used to specify plain-text HTTP or an alternate port	`https://admin-server-%s/rest`

Config File

All Environment variables can also be configured in a JSON file, by default they are loaded from the config.json file located in the project root, but an alternative JSON file can be configured used the the CONFIG_FILE env.

Config file example:

{
  "PRIMARY_USERNAME": "my-primary-user",
  "PRIMARY_PASSWORD": "password",
  "SECONDARY_USERNAME": "my-secondary-user",
  "SECONDARY_PASSWORD": "password",
  "ALIEN_USERNAME": "my-alien-user",
  "ALIEN_PASSWORD": "password",
  "DEV_CLUSTER_SERVER": "https://api.my.cluster:6443",
  "DEV_CLUSTER_TOKEN": "token",
  "DEV_CLUSTER_NAMESPACE": "a-custom-namespace-if-local",
  "LAUNCH_KEY": "my-username-if-local"
}

You can create multiple local *.config.json file that will be ignored by git (ex: prod.config.json) and you can easily switch between them in IntelliJ using the EnvFile plugin and setting the CONFIG_FILE env in the .env file in the project root.

Test Environments

The default targeted environment is the application services stage env.

Stage

Env	Value
`OPENSHIFT_API_URI`	`https://api.stage.openshift.com`
`OPENSHIFT_IDENTITY_URI`	`https://identity.api.stage.openshift.com`

Production

Env	Value
`OPENSHIFT_API_URI`	`https://api.openshift.com`
`OPENSHIFT_IDENTITY_URI`	`https://identity.api.openshift.com`

Profiles

Name	Description
default	run kafka, registry, devexp and quickstarts test suites
sandbox	run the sandbox test suite to test the openshift sandbox cluster
quickstarts	run the cucumber quickstarts test suite

Report to ReportPortal

When executing the tests is possible to send the results and logs in real time to ReportPortal using the ./hack/testrunner.sh script and the following ENVs:

Name	Description	Default value
`REPORTPORTAL_ENDPOINT`	ReportPortal URL	`https://example.com`
`REPORTPORTAL_ACCESS_TOKEN`	The Access Token
`REPORTPORTAL_LAUNCH`	The launch name to user	`mk-e2e-test-suite`
`REPORTPORTAL_PROJECT`	The project where to report the result	`rhosak`

Report to Prometheus

Tests can report metrics to Prometheus to analyze or monitor behaviours, like the number and frequency of failed API requests that are retried automatically. A push gateway is required to send the metrics to prometheus, and it is configured with the following ENVs:

Name	Description	Default value
`PROMETHEUS_PUSH_GATEWAY`	Prometheus Push Gateway URL	`https://example.com`

Short guides

Create test users

The test users needs to be manually created before running the test suites, but not all users are required for all tests.

Organization

Go to https://cloud.redhat.com/ and click on Create an account
You can choose Personal or Corporate
Fill all the required data and create the account

This will automatically create a new Organization where the user is the organization owner.

Primary user

Go to https://www.redhat.com/wapps/ugc/protected/usermgt/userList.html and login with the organization owner.
Click on Add New User and fill all the required data for the primary user

By default, the account will be able to create evaluation instances which will be enough to run single tests

Secondary user

Go to https://www.redhat.com/wapps/ugc/protected/usermgt/userList.html and login with the organization owner.
Click on Add New User and fill all the required data for the secondary user

The secondary user is exactly like the primary user, but by default it wouldn't have access to instances created by the primary user

Alien user

Go to https://cloud.redhat.com/ and click on Create an account
You can choose Personal or Corporate
Fill all the required data for the alien user and create the account

This will create the Alien user in a new organization

Create test namespace on the dev cluster

oc login --token=KUBE_ADMIN_TOKEN --server=DEV_CLUSTER_SERVER

Execute the ./hack/bootstrap-mk-e2e-tests-namespace.sh script
```
./hack/bootstrap-mk-e2e-tests-namespace.sh
```
Update the staging Vault with the new dev cluster token

Recreate the service account for the sandbox cluster

oc login --token=KUBE_ADMIN_TOKEN --server=DEV_CLUSTER_SERVER

Execute the ./hack/bootstrap-mk-e2e-tests-namespace.sh script

SANDBOX=true NAMESPACE=THE_SANDBOX_NAMESPACE ./hack/bootstrap-mk-e2e-tests-namespace.sh

Update the sandbox Vault with the new dev cluster token

Update the rhoas-model dependency

Clone the operator the bf2/operator from here:https://github.com/bf2fc6cc711aee1a0c2a/operator

git clone https://github.com/redhat-developer/app-services-operator.git
cd app-services-operator

Checkout the right version you want to update to

Example:
```
git checkout 0.5.0
```
Build the all the JARs (See operator README.md for the required maven and java versions)
```
mvn package
```

Copy the built rhoas-model.VERSION.jar to the e2e-test-suite lib/ directory

Example:

cp source/model/target/rhoas-model-1.0.0-SNAPSHOT.jar path/to/e2e-test-suite/lib

If the rhoas-model version has changed you need to update the pom.xml in the e2e-test-suite repository
Switch then to the e2e-test-suite repo and verify the build
```
mvn verify -Psmoke
```
If the build pass, commit the changes and open a PR, otherwise fix the issues and then open a PR

Maintainers

David Kornel kornys@outlook.com
Henrich Zrncik hzrncik@redhat.com
Tony Davidson tdavidso@redhat.com

grdryn / e2e-test-suite