artilleryio / artillery

Hello Artillery team !
Just a little issue with scenario names : it seems like they simply ignore parenthesis.

Version info:

Artillery: 2.0.10

Running this command:

yarn artillery run config.yaml --scenario-name="Get 2 animal pictures (but with parenthesis)"

I expected to see this happen:

Artillery running my script.

Instead, this happened:

worker error, id: 1 Error: Scenario Get 2 animal pictures (but with parenthesis) not found in script. Make sure your chosen scenario matches the one in your script exactly.

Files being used:

config.yaml

config:
    # This is a test server run by team Artillery
    # It's designed to be highly scalable
    target: "http://asciiart.artillery.io:8080"
    phases:
        - duration: 60
          arrivalRate: 1
          rampTo: 5
          name: Warm up phase
scenarios:
  - name: Get 1 animal pictures
    flow:
        - loop:
            - get:
                url: "/dino"
          count: 100
  - name: Get 2 animal pictures (but with parenthesis)
    flow:
        - loop:
            - get:
                url: "/dino"
            - get:
                url: "/pony"
          count: 100

More informations :

If the two scenarios have the same name except the parenthesis, for example:

scenarios:
  - name: Get 1 animal pictures
    flow:
        - loop:
            - get:
                url: "/dino"
          count: 100
  - name: Get 1 animal pictures (but with parenthesis)
    flow:
        - loop:
            - get:
                url: "/dino"

          count: 100

and we run yarn artillery run config.yaml --scenario-name="Get 1 animal pictures", we get another error :

Multiple scenarios for Get 1 animal pictures found in script. Make sure you give unique names to your scenarios in your script.

which leads to think the parenthesis (and what they contain) are stripped off the scenario name

I also tried with quoted name in yaml, with no more success :

  - name: "Get 3 animal pictures (but with parenthesis, and quotes)"
    flow:
        - loop:
            - get:
                url: "/dino"
            - get:
                url: "/pony"
            - get:
                url: "/armadillo"
          count: 100

artillery run config.yaml --scenario-name="Get 3 animal pictures (but with parenthesis, and quotes)" gives :

worker error, id: 1 Error: Scenario Get 3 animal pictures (but with parenthesis, and quotes) not found in script. Make sure your chosen scenario matches the one in your script exactly.

Hi @HugoImaios

We'll make this more clear in the documentation, but --scenario-name is using a regex to search. You'll have to escape special characters like ( and ).

I looked for it, is it this line ?

artillery/packages/core/lib/runner.js

Line 340 in 4206f30

const hasScenario = new RegExp(options.scenarioName).test(scenario.name);

I though about it and I don't really see the point of the regex here :
If we could give a real regex pattern as options.scenarioName, lets say with the aim of run only the scenarios that match this regex (with my examples above, it could be --scenario-name="Get 1|3 animal pictures" to get these two scenarios and only these)
But that is not possible, because we would run in the Multiple scenarios for ${options.scenarioName} found in script. Make sure you give unique names to your scenarios in your script error.
So : I think we should either allow multiple scenarios to match and be launched (and make the use of regex explicit in the documentation as you said), or remove the use of RegExp, allowing only the exact string to match the scenario name (which is the more intuitive solution imo) :

- const hasScenario = new RegExp(options.scenarioName).test(scenario.name);
+ const hasScenario = scenario.name === options.scenarioName;

(naive fix, I guess one could get rid of the whole scenarios.filter and use a findIndex instead, but you get it 😉 )

Hi @HugoImaios 👋

We had actually been discussing this same thing earlier in the week. For context, the reason I implemented it as a regex is because most test runners out there tend to have a grep/filter functionality that allows for partial matches. So for example, you could implement some sort of pattern in the name, and then just filter by that pattern.

That being said, I do think that makes more sense when you can run multiple scenarios (which artillery currently doesn't do). We'll discuss this again internally and see what we'll do here!

In the meantime, you should be unblocked for your use case right? Just escaping the special characters should work, i.e.:

artillery run config.yaml --scenario-name="Get 2 animal pictures $but with parenthesis$"

Looks good to me, I'll receive updates from here !

No worries for my use case, I already just removed the parenthesis from the scenario name in the config lol

This is solved in Artillery v2.0.12. You can use exact match or regex now.

Artillery ignores parenthesis in scenarios names