pipInstall up-to-date fails when deferring to `pip install -r`
instilled opened this issue · comments
First of all thanks for the great plugin!
We configured pipInstall
to use native requirements.txt
file. This gives devs the opportunity to work pip install -r requirements.txt
on the command line when sourcing the corresponding venv
. Unfortunately this leads to pipInstall
being always up-to-date after the first run. I believe the issue is with this code, i.e. modulesToInstall
is never populated with pip install -r
deferred installs. Also, since a requriements.txt
can include other requirements.txt files a potential solution ideally also deals with this. Maybe one way to solve is by using pip list
to list all installed dependencies when directReqsInstallRequired
is true and use that to populate modulesToInstall
?
python plugin config:
configure(pythonProjects + listOf(rootProject)) {
apply(plugin = "ru.vyarus.use-python")
configure<PythonExtension> {
pythonBinary = "python3.11"
scope = PythonExtension.Scope.VIRTUALENV
installVirtualenv = true
requirements.use = true
requirements.file = "requirements.txt"
requirements.strict = false
envPath = ".venv"
// alwaysInstallModules = true // while not really a fix this solves the problem for the time being.
}
}
and the corresponding requirements.txt
files:
> cat requirement.txt
-r requirements-prod.txt
pytest==7.3.1
pytest-asyncio==0.21.0
pytest-mock==3.10.0
httpx==0.24.1
testcontainers[postgres]==3.7.1
> cat requirements-prod.txt
cachetools==5.3.1
CacheControl==0.13.0
dependency-injector==4.41.0
fastapi==0.95.2
matplotlib==3.7.1
numpy==1.24.3
pandas==2.0.2
scipy==1.10.1
pg8000==1.29.4
Pillow==9.5.0
pyaml-env==1.2.1
pydantic-collections==0.4.0
pydantic-yaml==0.11.2
pydantic==1.10.8
python-multipart==0.0.6
SQLAlchemy==2.0.15
starlette==0.27.0
uvicorn[standard]==0.22.0
yoyo-migrations==8.2.0
google-auth==2.19.1
google-cloud-storage==2.9.0
google-cloud-logging==3.5.0
google-cloud-secret-manager==2.16.1
cloud-sql-python-connector[pg8000]==1.2.3
thanks!@
I'm on vacation now, will look this next week
So sorry for the late answer.. too many work after vacation.
Thank you very much for a detailed description!
pipInstall
would be UP_TO_DATE
either when there is no manual modules to install or due to not changed requirements file.
So, when linked requirements file (requirements-prod.txt) change, root requirements.txt remain unchanged and pipInstall
is not executed (UP_TO_DATE
).
alwaysInstallModules = true
was actually not working properly: it could help only once (due to task input parameter change), but next time it would still be UP_TO_DATE (if there are no manual modules specified).
I fixed alwaysInstallModules now to always trigger requirements installation. But, as you mention in comment, this still not a very good solution.
Maybe one way to solve is by using pip list to list all installed dependencies when directReqsInstallRequired is true and use that to populate modulesToInstall?
Originally, that was done with strict requirements mode: limited requirements files I can parse and use the same way as manual modules. In general, there are too nany variations of what could be specified in requirements files (links to other files (requirements or constants) , direct vcs links), so I'm not able to support all cases in general - and that's non-strict mode (black box).
I would try to support "-r" references for the strict mode: so in strict mode not only direct requirements file would be parsed, but also all "-r " references. Should be enought for your case.
Thanks for taking this up.
I fixed alwaysInstallModules now to always trigger requirements installation. But, as you mention in comment, this still not a very good solution.
This is great to hear.
I would try to support "-r" references for the strict mode: so in strict mode not only direct requirements file would be parsed, but also all "-r " references. Should be enought for your case.
This makes a lot of sense and is the correct way to solve it.
Just a thought: an alternative approach instead of going to parse the requriements.txt
for up-to-date checks maybe to declare them as input files if requirements.use = true
. You could support a configurable regex pattern to include requirement.txt
files as input. In my project I declared it like so which solves the problem nicely:
// kotlin, not groovy syntax
tasks.named("pipInstall", PipInstallTask::class) {
inputs.files(
"requirements.txt",
"requirements-prod.txt"
)
}
(to apply to all modules in a multi-module project declared in a configure(listOf(pyProjectA, pyProjectB, pyProjectC)) { ... }
closure)
Anyways. Thank you for taking this up!