pipInstall up-to-date fails when deferring to `pip install -r`

[Fabio]

First of all thanks for the great plugin!

We configured pipInstall to use native requirements.txt file. This gives devs the opportunity to work pip install -r requirements.txt on the command line when sourcing the corresponding venv. Unfortunately this leads to pipInstall being always up-to-date after the first run. I believe the issue is with this code, i.e. modulesToInstall is never populated with pip install -r deferred installs. Also, since a requriements.txt can include other requirements.txt files a potential solution ideally also deals with this. Maybe one way to solve is by using pip list to list all installed dependencies when directReqsInstallRequired is true and use that to populate modulesToInstall?

python plugin config:

configure(pythonProjects + listOf(rootProject)) {
    apply(plugin = "ru.vyarus.use-python")
    configure<PythonExtension> {
        pythonBinary = "python3.11"
        scope = PythonExtension.Scope.VIRTUALENV
        installVirtualenv = true
        requirements.use = true
        requirements.file = "requirements.txt"
        requirements.strict = false
        envPath = ".venv"
        // alwaysInstallModules = true  // while not really a fix this solves the problem for the time being.
    }
} 

and the corresponding requirements.txt files:

> cat requirement.txt
-r requirements-prod.txt                               
pytest==7.3.1
pytest-asyncio==0.21.0
pytest-mock==3.10.0
httpx==0.24.1
testcontainers[postgres]==3.7.1

> cat requirements-prod.txt                          
cachetools==5.3.1
CacheControl==0.13.0
dependency-injector==4.41.0
fastapi==0.95.2
matplotlib==3.7.1
numpy==1.24.3
pandas==2.0.2
scipy==1.10.1
pg8000==1.29.4
Pillow==9.5.0
pyaml-env==1.2.1
pydantic-collections==0.4.0
pydantic-yaml==0.11.2
pydantic==1.10.8
python-multipart==0.0.6
SQLAlchemy==2.0.15
starlette==0.27.0
uvicorn[standard]==0.22.0
yoyo-migrations==8.2.0

google-auth==2.19.1
google-cloud-storage==2.9.0
google-cloud-logging==3.5.0
google-cloud-secret-manager==2.16.1
cloud-sql-python-connector[pg8000]==1.2.3

thanks!@

[Vyacheslav Rusakov]

I'm on vacation now, will look this next week

[Vyacheslav Rusakov]

So sorry for the late answer.. too many work after vacation.

Thank you very much for a detailed description!

pipInstall would be UP_TO_DATE either when there is no manual modules to install or due to not changed requirements file.
So, when linked requirements file (requirements-prod.txt) change, root requirements.txt remain unchanged and pipInstall is not executed (UP_TO_DATE).

alwaysInstallModules = true was actually not working properly: it could help only once (due to task input parameter change), but next time it would still be UP_TO_DATE (if there are no manual modules specified).

I fixed alwaysInstallModules now to always trigger requirements installation. But, as you mention in comment, this still not a very good solution.

Maybe one way to solve is by using pip list to list all installed dependencies when directReqsInstallRequired is true and use that to populate modulesToInstall?

Originally, that was done with strict requirements mode: limited requirements files I can parse and use the same way as manual modules. In general, there are too nany variations of what could be specified in requirements files (links to other files (requirements or constants) , direct vcs links), so I'm not able to support all cases in general - and that's non-strict mode (black box).

I would try to support "-r" references for the strict mode: so in strict mode not only direct requirements file would be parsed, but also all "-r " references. Should be enought for your case.

[Fabio]

Thanks for taking this up.

I fixed alwaysInstallModules now to always trigger requirements installation. But, as you mention in comment, this still not a very good solution.

This is great to hear.

I would try to support "-r" references for the strict mode: so in strict mode not only direct requirements file would be parsed, but also all "-r " references. Should be enought for your case.

This makes a lot of sense and is the correct way to solve it.

Just a thought: an alternative approach instead of going to parse the requriements.txt for up-to-date checks maybe to declare them as input files if requirements.use = true. You could support a configurable regex pattern to include requirement.txt files as input. In my project I declared it like so which solves the problem nicely:

// kotlin, not groovy syntax
tasks.named("pipInstall", PipInstallTask::class) {
    inputs.files(
        "requirements.txt",
        "requirements-prod.txt"
    )
}

(to apply to all modules in a multi-module project declared in a configure(listOf(pyProjectA, pyProjectB, pyProjectC)) { ... } closure)

Anyways. Thank you for taking this up!

[Vyacheslav Rusakov]

4.0.0 released