AWS OpenSearch Integration testing.
niedbalski opened this issue · comments
Description
This issue tracks the implementation of integration testing on top of the new out_opensearch plugin implementation.
1.9.0-rc4 implements out_opensearch fluent/fluent-bit@6fdaa9e
The testing matrix that will be initially supported is:
| Fluent-bit | EKS Version | Event | OpenSearch Release |
|---|---|---|---|
| master | 1.21 | Push to master / PR to plugin | Nightly build, latest release |
| master | 1.20 | Push to master / PR to plugin | Nightly build, latest release |
| master | 1.19 | Push to master / PR to plugin | Nightly build, latest release |
| 1.9 | 1.21 | Push to 1.9 branch / PR to branch | Nightly build, latest release |
| 1.9 | 1.20 | Push to 1.9 branch / PR to branch | Nightly build, latest release |
| 1.9 | 1.19 | Push to 1.9 branch / PR to branch | Nightly build, latest release |
The components to be used and implemented are:
-
OpenSearch helm chart https://opensearch.org/docs/latest/opensearch/install/helm/ , Implement a basic validation test based on the ElasticSearch test https://github.com/calyptia/fluent-bit-ci/tree/main/integration/tests/elasticsearch
-
For the AWS hosted version (https://aws.amazon.com/opensearch-service/features/), the following
is the cluster type matrix supported.
| Cluster type | Master node type | Data node type |
|---|---|---|
| Dedicated master node | 1 x m5.large.search or m6g.large.search | 1 x >= M5 |
Tasks
- Create an independent AWS account that has a billing account associated.
- Associate a new IAM role that allows access to OpenSearch and EC2
- Add the AWS credentials to the infra repository and configure the AWS provider
- Add the terraform resources to handle the 3 Kubernetes cluster types according to the following specs:
- Add the terraform resources https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticsearch_domain
to create an OpenSearch index/cluster. - Implement the integration test that uses the versions outlined on the spec.
Further Notes
N/A
All tasks have been completed, I am closing this issue as no further work has to be performed.