[Idea] Make elastic-package friendly for single assets

[Nicolas Ruflin]

elastic-package was built to make working with packages as easy as possible and have tooling around. For some assets like dashboards, export functionality exists (elastic-package export dashboards). One of the reasons it exists for dashboards is that there additional transformations are needed. But if a dev has an asset like ingest pipeline, the asset has to be manually exported (and converted to yaml) as an example.

Many of our users (internal and external) version elastic stack assets in Git somewhere and have built their own tooling to import / export / diff these assets. If a user has only a single asset like an ingest pipeline, using a package for it might seem overkill. It would be nice if these user could also use elastic-package to import / export these assets. The idea is that we could standardise the tooling. As soon as users have multiple assets, packages with versions start to make sense and users are already setup to take the next step.

The benefit of the above is also that it simplifies getting assets into packages easier as export for all the assets would exist.

[Andrew Wilkins]

@ruflin what would be the purpose and benefit over using the elasticstack Terraform provider? Perhaps instead of extending elastic-package to support individual assets, the elasticstack provider could be extended to support integration packages?

The benefit I see in building on Terraform is that it will fit into existing infrastructure automation. For example, you could compose the elasticstack provider with the ec (Elastic Cloud) provider -- create an ESS deployment, and then install an integration package in it.

It doesn't have to be either/or, but I think we should be clear about the purpose if we have two options.

[Nicolas Ruflin]

I have never created a terraform provider so some of the following assumptions might be wrong.

The code to import / export / diff assets should only exist once. elastic-package is a binary that you can easily run on any platform, spin up a stack for testing and do quite a few more things related to packages and assets. A user just wanting to import / export an asset should not have to learn about terraform and how it works. So I would build all the logic around how to load assets into elastic-package. My assumption is, this makes it available to a broader audience.

The terraform provider should use all this logic, be it using elastic-package directly or modules from elastic-package, my understanding is both are in Golang. The terraform provider offers loading single assets or integration packages, but both is based on a single logic.

[Andrew Wilkins]

The code to import / export / diff assets should only exist once.
...
A user just wanting to import / export an asset should not have to learn about terraform and how it works.

Fair enough. It seems to me the distinction is this:

• elastic-package is for interactive use, and for imperative scripting. Similar to AWS CLI, Azure CLI, gcloud.
• the elasticstack Terraform provider is for declarative configuration, using the exact same formats as supported by elastic-package.

IMO elastic-package would be a bit of a misnomer, as it's then not really about integration packages, but about managing Elastic resources in general. Ideally we would have an Elastic CLI that can be used to create Elastic Cloud deployments, issue search requests, manage templates/pipelines/ILM/integration packages, etc.

[Nicolas Ruflin]

IMO elastic-package would be a bit of a misnomer, as it's then not really about integration packages, but about managing Elastic resources in general. Ideally we would have an Elastic CLI that can be used to create Elastic Cloud deployments, issue search requests, manage templates/pipelines/ILM/integration packages, etc.

Agree, it would be great to have a CLI for the Elastic Stack / Cloud. Naming of elastic-package is not ideal in this context at the same time it already has built in lots of related tooling like setting up stack in different environments.

[Mike Place]

it would be great to have a CLI for the Elastic Stack / Cloud. Naming of elastic-package is not ideal in this context at the same time it already has built in lots of related tooling like setting up stack in different environments.

I'd actually take this a step further and ask if we couldn't think about something that looks more like an SDK for building CLI tooling. There's a whole ecosystem of command-line tooling out there that we could be building. Moreover, publishing an SDK for CLI tooling really speaks to what SREs out there in the world are trying to do, which is to take an Observability solution and tailor it to their individual environments.