[discuss] Add support elastic agent policies and packages
marc-gr opened this issue · comments
For some time we have been investigating what could be the best approach to have E2E benchmark capabilities for the elastic-agent
integrations. Some PoC work has been done to both extend current tooling to be able to work with elastic-agent
and to figure out the best way to also extend elastic-package
to be able to run these benchmarks.
A lot of the work required to happen in elastic-package
(or in any new tool built for that purpose) would overlap with a lot of rally
's functionality (mostly on reporting and being able to compare benchmark runs), and would most likely be lacking some of rally
's other capabilities that could come as handy (for example setting up tracks with queries used in the dashboards built in the integrations).
For this could be a good idea to explore extending rally
in a way that can deal with elastic-agent
policies and work with corpora
in a way that is useful for integrations.
This changes do not intend to make rally
handle the required infrastructure to host elastic-agent
but to allow policies to be part of a track definition and benefit from its reporting and querying capabilities along with its already existing support in all the performance related ecosystem.
An example of a possible track definition for an elastic-agent
setup could be:
{
"policies": [
{
"id": "an-id",
"name": "A new policy",
"description": "A description",
"namespace": "default",
"monitoring_enabled": ["logs", "metrics"],
}
],
"corpora": [
{
"name": "panw-logs",
"documents": [
{
"base-url": "http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/logs",
"source-file": "panw-181998.json.bz2",
"document-count": 2708746
}
]
}
],
"operations": [
{
"policy-id": "an-id",
"name": "install-panw",
"operation-type": "install-package",
"package": "panw",
"version": "3.5.0",
"input": "logfile",
"vars": {},
"data_stream": {
"name": "panos",
"vars": {
"paths": ["/var/logs/panw.log"],
}
}
},
{
"name": "log-write",
"operation-type": "log-write",
"corpora": "panw-logs",
"rate": 10,
"hosts": ["agent-host-1", "agent-host-2"],
"target-file": "/var/logs/panw.log"
}
]
}
This is adding some new elements:
policy
: defines a set of policies that can be used- a new operation
install-package
that will configure the specific package - a new operation
log-write
that will write the logs fromcorpora
into a file at the specified hosts
This is just a first draft. Some things to be taken into account:
- The expectation is that agent hosts are created outside of the rally workflow
- A number of new operations to write logs in different ways (eg to a UDP or TCP socket, etc) to support the different inputs available to integrations will be required, or maybe this can be part of an external tool that
rally
can use.
With this additions to rally we would be able to:
- Extend
elastic-package
to launch tracks if needed - Allow for better sizing capabilities for
elastic-agent
users.
Hello, and thanks for considering Rally for your Elastic Agent benchmarking needs! I'll have to discuss more with the team, but it does look like at first sight that with our help you could add something like this to Rally without disrupting existing Elasticsearch users and reusing some of the Rally infrastructure.
Sorry for the naive questions, but I'm not familiar with Elastic Agent and Fleet at all.
- What are the prerequisites before running an Elastic Agent track like this? Is a target Elasticsearch cluster enough?
- Can you please explain how Rally would in practice run the above track? I guess elastic-package will install the package but I don't know how the log-write operation will work nor what it actually does. And I don't know how to define policies
- What metrics do you care about? I know there's the ingest pipelines telemetry device at least.
- Is there any relationship between this and the
elastic-package benchmark
command?
- What are the prerequisites before running an Elastic Agent track like this? Is a target Elasticsearch cluster enough?
An elasticsearch setup with a fleet server and a kibana instance. Also having the required elastic agent hosts installed to use if needed to send the corpora data instead to do it directly to ES.
- Can you please explain how Rally would in practice run the above track? I guess elastic-package will install the package but I don't know how the log-write operation will work nor what it actually does. And I don't know how to define policies
Both policy installation and package installation are done through the kibana fleet APIs
The operations to send the data are the ones that I am more unsure about, since integrations can use a number of different inputs (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html), so maybe this complexity could be taken out to an external tool that rally can reference as a single operation.
- What metrics do you care about? I know there's the ingest pipelines telemetry device at least.
node stats, ingest pipelines and host metrics from the agent hosts (which I assume would use a new telemetry device from what you mention), also query performance of the dashboards, but I am pretty sure this is already in place
Is there any relationship between this and the elastic-package benchmark command?
that command support only synthetic benchmarks for pipelines, would be useful to extend it to be able to generate or use tracks like the ones described, for example, if this ends up being a thing
hello, chiming in
that command support only synthetic benchmarks for pipelines, would be useful to extend it to be able to generate or use tracks like the ones described, for example, if this ends up being a thing
we are working on a elastic-package command (elastic/elastic-package#1110) in order to generate rally tracks, we'd need the same support for packages installation through fleet API discussed here