Cannot use tox generated matrix; Azure Pipelines does not accept jobs with '-'

[Alex Graber]

Sample tox envlist:

[tox]
# https://tox.readthedocs.io/en/latest/config.html#generative-envlist
envlist =
    clean,
    check,
    ; py{37,38}-np{18}-pd{0,2}-sci{14,16},  # test conflict btwn numpy, scipy, and c extensions
    ; py{37,38}-np{19,20}-pd{0,2}-sci{15,16},
    py{37,38}-np{18,19,20},
    py39,  # latest version of python, assume 'latest' for dependencies
    report,
    docs

Sample azure pipeline workflow:

resources:
  repositories:
    - repository: tox
      type: github
      endpoint: github
      name: tox-dev/azure-pipelines-template
      ref: refs/tags/0.2

jobs:
  - template: run-tox-env.yml@tox
    parameters:
      tox_version: ""
      jobs:
        check: null
        py37-np18:
        py37-np19:
        py37-np20:
        py38-np18:
        py38-np19:
        py38-np20:
        py39:
        coverage:
          with_toxenv: "report"
          for_envs: [py37-np18, py37-np19, py37-np20, py38-np18, py38-np19, py38-np20, py39]
        docs: null

Azure Pipelines error message:

[Bernát Gábor]

Suggestions to solve this and PR welcomed 👍

[Alex Graber]

Unfortunately, I'm not sure I have the time (or understanding of the underlying code & context) to support a PR.

However, if it helps to provide direction here is my perspective --

1. It does not make sense to me that we have to manually specify the tox jobs in the pipeline. Part of the point of using tox is that it can use the matrix to generate environment names - I don't want to have to go through and manually enumerate them
2. As mentioned above, the azure-pipelines-template requires the job name == tox env name. Tox's matrix requires hyphens; Azure prohibits job names from using hyphens.

Therefore, it seems to me it would be most straightforward if azure-pipelines-template read the tox.ini and auto-generated the job names, perhaps substituting underscores for hyphens when naming the job for Azure. I have no idea if this is possible.

It seems to me that the likely easier way to resolve the second issue would be to provide a with_toxenv argument for each tox env, like you do for coverage, that would allow the separation of the Azure job name from tox env name.

But this discussion has demonstrated a lack of my understanding of how to use tox's azure-pipelines-template?

[Bernát Gábor]

1. It does not make sense to me that we have to manually specify the tox jobs in the pipeline.

This is sadly not possible from a technical point of view 😊 I think one could support by writing an Azure App in TypeScript but that's something that would be complicated for me to take on.