How to define poetry configuration in tox?
fredrikaverpil opened this issue · comments
Hi, this is a very nice initiative! ❤️ 🤘
I was wondering, do you have any ideas on how poetry config entries could be added to this plugin?
I would like to enter something like this in my global tox ini config:
poetry config repositories.repo1 https://...
poetry config repositories.repo2 https://...
poetry config http-basic.repo2 user $pass
Cheers!
Thanks for the question.
My first impression after a quick research is that it seems achievable in a clean manner. It looks like it could be relatively straightforward to translate Poetry's configuration entries related to repositories into Tox's corresponding settings ([1], [2]).
But...
This Tox feature seems to be on the way out. The indexserver setting is deprecated. In the discussion "Clarify which part of the indexserver spec is deprecated" (#1357) on Tox's issue tracker there is at least one alternative solution suggested. I will have to take a closer look at this, and figure out whether or not it is a viable option.
Related:
Allright, thank you 👍
I actually realized that I can get access to my internal PyPi repos by defining pip environment variables and using an additional poetry test env for publishing of packages. I figured I'd share my use case (heavily inspired by @mikenerone ):
[tox]
minversion = 3.20.0
isolated_build = true
requires =
poetry-core>=1.0.0b1
tox-poetry-dev-dependencies>=0.0.1
provision_tox_env = provision
[base]
setenv =
PIP_INDEX_URL = https://$server1
PIP_EXTRA_INDEX_URL = https://$user:$password@$server2
[testenv:pytest]
add_poetry_dev_dependencies = true
setenv = {[base]setenv} # required so that tox/poetry-core can install packages from internal PyPi repos
commands =
pytest
[testenv:{poetry,build_wheel,publish_wheel}]
envdir = {toxworkdir}/.poetry
skip_install = true
deps =
poetry==1.0.10
poetry-dynamic-versioning==0.8.3
setenv = {[base]setenv} # redundant because of the "poetry config" commands below
commands_pre =
poetry --version
poetry config repositories.internal-mirror https://$server1
poetry config repositories.my-repo https://$server2
poetry config http-basic.my-repo $user $pass
commands =
poetry: poetry {posargs:debug info}
build_wheel: poetry build --format wheel
publish_wheel: poetry publish --build --no-interaction --repository my-repoCommands:
tox -e pytest
tox -e poetry -- show --tree
tox -e build_wheel
tox -e publish_wheel
Does that mean you have to duplicate the index URL in Poetry's configuration and in Tox's configuration? If yes, I think it is worth finding a solution to avoid this duplication.
Yes. The index URLs will be specified only in the tox.ini but twice; both in [base] and in [testenv:publish_wheel].
The last one will cause Poetry's config to also hold the index URL.
I think I got it, but in case I might be misunderstanding some things...
In this configuration, the source of truth for the index URL is in the setenv setting of the [base] section of tox.ini. Is that right?
And it is "forwarded" to Poetry via the (incomplete, or redacted?) poetry config repositories.my-repo command of [testenv:publish_wheel]. Meaning that it is set on each run of this environment. Is that right?
Hmm, let me explain.
The index URLs in [base] are for pip only. So this is the way you would configure pip using environment variables. Since poetry is using pip under the hood, this is one way to make poetry/pip look for packages in these PyPi repos. Example:
PIP_INDEX_URL = https://$SERVER/api/pypi/pypi-remote/simple # Public mirror of PyPi.org
PIP_EXTRA_INDEX_URL = https://$USER:$PASSWORD@$SERVER/api/pypi/my-custom-pypi-repo/simple # Internal password-protected custom PyPi repoUsing poetry config you can instead configure credentials and location of additional repositories (without affecting pip). But you can also configure other things: https://python-poetry.org/docs/configuration/
So, you could perhaps see it this way (right now, when your plugin does not handle this):
- Use the pip environment variables when the
poetryexecutable is not available - Use the
poetry configcommands whenever you have thepoetryexecutable available
My example tox.ini in my previous post does not show this very well, but in [test:pytest], the poetry executable is not available. so here I must use the pip environment variables if I want access to the PyPi repos. However in [testenv:poetry], the poetry executable is available and therefore I can enable PyPi repo access using poetry config commands.
What would be cool if you could actually tell your plugin about poetry config commands, so that tox can install packages, defined in pyproject.toml from any internal pypi repos or mirrors.
Sorry. it's late here. I am mixing the envs up... let me re-explain my core issue.
In my pyproject.toml, I have packages which are not hosted on the public pypi.org. Whenever I use add_poetry_dev_dependencies = true I would like to somehow use these internal PyPi repos (and authorize properly with them).
This can be configured through pip environment variables (or poetry config commands).
What complicates things here is that poetry also stores its config in ~/.config/pypoetry even if the poetry config commands are carried out from the .tox/poetry testenv... so I am starting to think I will have to instead have environment variables for pip registered prior to running any tox commands.
Thanks.
I will think about it. See what a good compromise could look like.
Yeah. This is tricky.
The end goal for me is to just have tox as a dependency and specify everything-poetry via pyproject.toml, tox.ini, additional text based files and environment variables.
If we are to add this in the code then I really should have named the project differently! :D
Indeed it goes beyond just the dev-dependencies. I half expected it. Well, that's how it is...
I think I will first try to read from the index URLs from Poetry's configuration and add them to Tox's indexserver setting. If we restrict to tox < 4 that should be good enough for a first step.
Because to be honest I kind of struggle wrapping my head around the whole workflow that would use the PIP_INDEX_URL environment variable. It seems like there are some holes in the logic. What if one redefines the install_command to use something else than pip? How would Conda users do? Should this plugin generate different environment variables then?
A couple of question for you @fredrikaverpil ...
I had a closer look at Poetry's documentation on "Repositories" and it seems to me like the repositories that are set via poetry config are used for uploading (publishing the distributions) exclusively, and not for downloading. Is that right?
What we actually need is to read the [[tool.poetry.source]] sections first. And then, if authentication is needed, we should read the corresponding http-basic configuration values. Is that right?
Indeed it goes beyond just the dev-dependencies. I half expected it. Well, that's how it is...
🙃
... are used for uploading (publishing the distributions) exclusively, and not for downloading. Is that right?
Actually, we have this PyPi server that requires credentials for any access; both uploading and downloading.
What we actually need is to read the [[tool.poetry.source]] sections first. And then, if authentication is needed, we should read the corresponding http-basic configuration values. Is that right?
Yes, I believe this is correct.
I don't know if this helps, but this is what I do:
In a repo with the pyproject.toml file and the poetry.lock file (along with the project's package):
- Create venv, activate it
pip install tox- (optionally
tox -e poetryto set up credentials, as seen in my previously posted code block) tox -e some-test-env
Ideally, it would be cool to have tox.ini basically be able to just handle the rest. But, like I mentioned in the discord chat, maybe this is a little too much to ask. I can prepare environment variables and put the ~/.config/pyproject/* files into place prior to executing tox.
Note, in case it becomes relevant later...
In the tox rewrite there is a mention in a TODO list of the src/new.md file:
# TODO
* index url support for python pip
@fredrikaverpil Version '0.0.2' has been released. It includes a new poetry_use_source_repos, that should help install from alternative repositories. The plugin still doesn't handle credentials set in Poetry's config, but maybe it could already help simplify your tox.ini.
Very nice, thank you! 😊
@fredrikaverpil About the credentials...
Am I understanding it right that your current solution to give pip the credentials for a specific index server, is to place them in the environment variable:
[base]
setenv =
PIP_EXTRA_INDEX_URL = https://$user:$password@$server2
Isn't there a risk of leaking the credentials in the log trace for example? Do you know of any other way to do that? Isn't pip supposed to read credentials from keyring?
And on the side of poetry, are the credentials written somewhere via keyring? I did a quick test and the credentials ended up being written in plain text:
$ poetry config http-basic.global johndoe
[...]
$ cat ~/.config/pypoetry/auth.toml
[http-basic]
[http-basic.global]
username = "johndoe"
password = "password"
Otherwise in principle, I believe it should be possible to implement the handling of the credentials in the plugin. I just want to be sure that I am looking at the problem from the right angle.