Giters

opensearch-project / opensearch-catalog

The OpenSearch Catalog is designed to make it easier for developers and community to contribute, search and install artifacts like plugins, visualization dashboards, ingestion to visualization content packs (data pipeline configurations, normalization, ingestion, dashboards).

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[FEATURE] Testing for integrations in CI

Swiddis opened this issue · comments

commented

Is your feature request related to a problem?
We currently don't have very good CI for validating whether integrations work. There are a few tests that I historically have manually tried to do every time I want to check an integration, which often are missed by devs and increase dev time, and also have been historically missed in bug bashes (e.g. many current 2.9 integrations fail check (4)):

  1. Check the config for any clear issues, make sure it loads correctly*.
  2. Move the integration to a local directory and run osints diff to find type errors.
  3. Try to create the integration manually with "try it" and ensure that panels aren't erroring.
  4. Try to connect to the integration's own sample index to ensure that there's no dormant type mismatches caused during index creation.
  5. Checks for cross-interference: Creating multiple copies of integrations, creating sequences of integrations, trying to make joins, all that fun stuff. These are more complicated to test and should really be looked at after (1)-(4) are met.

* - This step has been automated in the current plugin CI, but will need to be moved to this repository when we do the catalog move.

What solution would you like?
At a glance, all of these tests except maybe (3) and (5) seem simple to automate. We should have a CI pipeline, ideally at a unit test level, which can perform all of these on the catalog. That said, there is a wider issue of how we actually want to introduce the CI pipeline to this repository. For example, there are existing tests in the plugin written in Typescript, while the CLI code here is written in Python, and the Typescript tests depend on plugin methods to run.

At a glance, my instinct is to have one small test action on this repo that does unit testing to validate integrations for (1) and (2), and a larger "soak test" action that does a full integration test with the integrations plugin for (3), (4), and (5).

What alternatives have you considered?

  • Rewrite Typescript tests in Python, add a CI pipeline to the CLI that includes validating any catalog that we put here.
  • Copy CI over with new validation methods and separately run Python and Typescript CI, which might duplicate a lot of code.
  • We could also look at installing the Integrations plugin within the CI, but this will also mean inheriting dashboards-observability's long build times for only a few methods.
  • Convert CLI logic to Typescript and somehow merge it with the Integrations plugin, saving duplication but still having the issue of tests and content being in different locations.

Do you have any additional context?