MarathonLabs / marathon

Cross-platform test runner

Home Page:https://docs.marathonlabs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add blocked list of tests for batching strategy

testing0mon21 opened this issue · comments

Is your feature request related to a problem? Please describe.

When using the batching strategy in our Marathontool configuration, I often encounter situations where some tests should not be batched together. For instance, certain tests may require exclusive execution or have specific dependencies that cannot be satisfied when executed in parallel. Currently, the only way to achieve this is by creating separate configurations for each test suite, which can lead to a high number of redundant configuration files.

Describe the solution you'd like

I would like to propose an option to specify a "batching strategy" or "test exclusion" per test or test suite in the Marathontool configuration. This could be achieved by introducing a new property or annotation that allows us to define which tests should not be batched together. For example, we could use a @NoBatch annotation on specific test methods or classes.

Describe alternatives you've considered

One possible alternative is to create separate configurations for each test suite that requires exclusive execution. However, this would lead to a proliferation of configuration files and make it more difficult to manage and maintain our testing setup.

Another option could be to implement a custom batching strategy using Marathontool's extensibility features (e.g., custom executors or reporters). While this would provide a high degree of flexibility, it would also require significant development effort and might not be feasible for non-technical users.

Additional context

Here is an example configuration file that demonstrates the issue:

`
batchingStrategy:
type: "fixed-size"
size: ${BATCH_COUNT}
blocklist:
- type: "annotation"
values:
- "com.example.SmokeTest"

`
In this example, we would like to exclude tests with annotation SmokeTest from being batched together.