Gradle does not recognize Pyenv environment
alexcombessie opened this issue · comments
I am using pyenv to manage Python environments on my Mac.
alexcombessie@Alex-Air-001 giskard % pyenv version
3.7.13 (set by /Users/alexcombessie/Github/giskard/.python-version)
alexcombessie@Alex-Air-001 giskard % python --version
Python 3.7.13
alexcombessie@Alex-Air-001 giskard % which python
/Users/alexcombessie/.pyenv/rosetta/shims/python
Unfortunately, this is not recognized in my gradle build. Gradle never finds which python from pyenv and keeps using the default system package of python, which are of the wrong version.
I tried adding a pythonBinary argument, with different values: "python", "python3.7", or even "/Users/alexcombessie/.pyenv/rosetta/shims/python3.7' but it always failed to build, as the argument was not recognized. The last trial gave me this error message:
Python not found: .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python3.7. This must be a bug of virtualenv support, please report it (https://github.com/xvik/gradle-use-python-plugin/issues). You can disable virtualenv usage with ‘python.scope = USER’.
PS: I do need to set scope = 'VIRTUALENV'
as I need the virtualenv for another part of the app.
FYI @andreybavt
Possibly, this would help:
python {
pythonPath = '/Users/alexcombessie/.pyenv/rosetta/shims/'
// pythonBinary = 'python'
}
I will try to reproduce this with pyenv (never used it before) and try to fix the default behaviour if possible.
Hi @xvik ,
Thanks for your swift reply. I confirm that with the pythonPath
you added, gradle correctly picks up Python from pyenv.
Would it be possible to make this the default behavior? The issue is that I cannot commit this workaround, as it would be only specific to my local dev setup.
Thanks,
Alex
You can also try pythonPath = '~/.pyenv/rosetta/shims/'
- if it would work, it should be acceptable for commit if other devs also use pyenv.
I'll, of course, look, but it might take some time. From what I read on pyenv site, it should already work properly.
The path in your error is confusing: .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python3.7.
. Don't know how .venv/bin/
prefix appeared there. Do you have a venv created in your project? Or, maybe, know what is .venv
dir.
Also, could you please attach full gradle log for failed execution: it should be clear from there if plugin detects some venv and tries to use as virtualenv (instead of using global python for virtulenv creation)
You can also try pythonPath = '~/.pyenv/rosetta/shims/' - if it would work, it should be acceptable for commit if other devs also use pyenv.
Unfortunately that is too brittle for us, we would rather use the default system way of detecting python
from PATH
. Not all of us use Pyenv. But everyone has python in their PATH. When using our shell (sh / zsh / bash) python is recognized. The problem is isolated to gradle.
To reproduce, here is a sample of my build.gradle
python {
pythonBinary = '/Users/alexcombessie/.pyenv/rosetta/shims/python'
envPath = '.venv'
minPythonVersion = '3.7'
scope = 'VIRTUALENV'
installVirtualenv = true
pip 'poetry:1.1.13'
environment "PYTHONPATH", file('generated')
}
task install(type: PythonTask) {
dependsOn python: pipInstall
module = "poetry"
command "install"
}
Output:
lexcombessie@Alex-Air-001 giskard % ./gradlew :giskard-ml-worker:build --info
Initialized native services in: /Users/alexcombessie/.gradle/native
Initialized jansi services in: /Users/alexcombessie/.gradle/native
The client will now receive all logging from the daemon (pid: 23567). The daemon log file: /Users/alexcombessie/.gradle/daemon/7.4/daemon-23567.out.log
Starting 29th build in daemon [uptime: 3 hrs 26 mins 18.575 secs, performance: 100%, non-heap usage: 41% of 256 MiB]
Using 10 worker leases.
Now considering [/Users/alexcombessie/Github/giskard] as hierarchies to watch
Watching the file system is configured to be enabled if available
File system watching is active
Starting Build
Settings evaluated using settings file '/Users/alexcombessie/Github/giskard/settings.gradle'.
Projects loaded. Root project using build file '/Users/alexcombessie/Github/giskard/build.gradle'.
Included projects: [root project 'giskard', project ':giskard-demo-notebook', project ':giskard-frontend', project ':giskard-ml-worker', project ':giskard-server']
> Configure project :
Evaluating root project 'giskard' using build file '/Users/alexcombessie/Github/giskard/build.gradle'.
> Configure project :giskard-demo-notebook
Evaluating project ':giskard-demo-notebook' using build file '/Users/alexcombessie/Github/giskard/giskard-demo-notebook/build.gradle'.
> Configure project :giskard-frontend
Evaluating project ':giskard-frontend' using build file '/Users/alexcombessie/Github/giskard/giskard-frontend/build.gradle'.
> Configure project :giskard-ml-worker
Evaluating project ':giskard-ml-worker' using build file '/Users/alexcombessie/Github/giskard/giskard-ml-worker/build.gradle'.
> Configure project :giskard-server
Evaluating project ':giskard-server' using build file '/Users/alexcombessie/Github/giskard/giskard-server/build.gradle'.
Adding lombok-mapstruct-binding for source set main because DefaultExternalModuleDependency{group='org.mapstruct', name='mapstruct-processor', version='1.4.2.Final', configuration='default'} was found
------------------------------------------------------------------------
Detecting the operating system and CPU architecture
------------------------------------------------------------------------
os.detected.name=osx
os.detected.arch=aarch_64
os.detected.bitness=64
os.detected.version=12.5
os.detected.version.major=12
os.detected.version.minor=5
os.detected.classifier=osx-aarch_64
All projects evaluated.
Selected primary task ':giskard-ml-worker:build' from project :giskard-ml-worker
Tasks to be executed: [task ':giskard-ml-worker:assemble', task ':giskard-ml-worker:check', task ':giskard-ml-worker:checkPython', task ':giskard-ml-worker:pipInstall', task ':giskard-ml-worker:install', task ':giskard-ml-worker:generateProto', task ':giskard-ml-worker:test', task ':giskard-ml-worker:build']
Tasks that were excluded: []
:giskard-ml-worker:assemble (Thread[Execution worker for ':',5,main]) started.
> Task :giskard-ml-worker:assemble UP-TO-DATE
Skipping task ':giskard-ml-worker:assemble' as it has no actions.
:giskard-ml-worker:assemble (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:giskard-ml-worker:check (Thread[Execution worker for ':',5,main]) started.
> Task :giskard-ml-worker:check UP-TO-DATE
Skipping task ':giskard-ml-worker:check' as it has no actions.
:giskard-ml-worker:check (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:giskard-ml-worker:checkPython (Thread[Execution worker for ':',5,main]) started.
> Task :giskard-ml-worker:checkPython FAILED
Caching disabled for task ':giskard-ml-worker:checkPython' because:
Build cache is disabled
Task ':giskard-ml-worker:checkPython' is not up-to-date because:
Task has not declared any outputs despite executing actions.
Python arguments: -c
[python] .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python -c exec("import sys;ver=sys.version_info;print(str(ver.major)+'.'+str(ver.minor)+'.'+str(ver.micro));print(sys.prefix);print(sys.executable)")
Starting process 'command '.venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python''. Working directory: /Users/alexcombessie/Github/giskard/giskard-ml-worker Command: .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python -c exec("import sys;ver=sys.version_info;print(str(ver.major)+'.'+str(ver.minor)+'.'+str(ver.micro));print(sys.prefix);print(sys.executable)")
:giskard-ml-worker:checkPython (Thread[Execution worker for ':',5,main]) completed. Took 0.004 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':giskard-ml-worker:checkPython'.
> Python not found: .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python. This must be a bug of virtualenv support, please report it (https://github.com/xvik/gradle-use-python-plugin/issues). You can disable virtualenv usage with 'python.scope = USER'.
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 535ms
pythonBinary
declares just a name of the binary so it is not correct to declare complete path here (it should be done with pythonPath
).
To better understand how it works:
- When there is no created virtualenv, python would be called as
${pythonPath}/${pythonBinary}
- When virtualenv folder exists then:
${envPath}/bin/${pythonBinary}
(pythonBinary
remains the same in both cases andpythonPath
gets substituted to switch into virtualenv folder)
On non windows host, when pythonBinary
is not declared, plugin would try to use python3
first and then fallback to python
.
So if python3 binary is not recognized then the default should work (will use python
). Overall, the following configuration should work properly:
python {
pythonBinary = 'python'
envPath = '.venv'
minPythonVersion = '3.7'
scope = 'VIRTUALENV'
installVirtualenv = true
pip 'poetry:1.1.13'
environment "PYTHONPATH", file('generated')
}
If it doesn't then, most likley, you have a problem with PATH. For example, when intelliJ IDEA is not launched from shell it will not inherit PATH customizations (from bashrc) and running gradle within it will not find pyenv python.
Running gradle (or idea) directly from shell should work correctly.
You can verify PATH correctness by putting println System.getenv('PATH')
inside your build.gradle.
If you use bash and PATH is modified for pyenv in .bashrc then you can move it into ~/.profile
and log-out/log-in. Probably would help, but can't check now (tip from here).
I don't see other problems in plugin itself: at least I was able to use pyenv python on ubuntu.
Thanks! That gives me enough information for debugging. Since you say it works on Ubuntu, I suspect my issue is that I am on an M1 mac, with rosetta mode, and multiple Python depending on the arch.
I will debug the PATH on my end.
I am running into the same problem! @alexcombessie did you have any luck debugging this?
Yes. It seems there was in issue with the dual ARM/x86 nature of M1 MacBooks with Rosetta emulation.
For some reason, when I launch a gradle command from a shell which is on the x86 arch, it ignores the shell context and uses the default ARM arch.
I found a workaround by hardcording the default python in my zshrc to a x86 binary of python.
I guess gradle always starts a new shell?
In the upcoming version I added manual search for python binary inside PATH: if binary not found, plugin will fail and error message would contain PATH value. This should simplify revealing such problems.
@alexcombessie could you please provide exact sample command for this workaround? I'll put it in readme as a reference for others (I hope it would be useful)
I guess gradle always starts a new shell?
Gradle itself - no. Gradle inherits PATH from the shell launching it. But, in some cases, running shells indeed could differ. For example, on ubuntu, PATH changes from .bashrc might be not visible when gradle launched from IDE, launched from system launcher.
I found interesting stackoverflow answer: the problem might be in jvm - if gradle launched with jvm compiled for x86 - it will run through rosetta. (edit: oh, I missed, you mention problem is different in the previous comment)
@alexcombessie could you please provide exact sample command for this workaround? I'll put it in readme as a reference for others (I hope it would be useful)
My workaround is very dirty, I hardcoded these lines into my .zshrc
:
export PATH="/usr/local/opt/python@3.7/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/python@3.7/lib"
export PKG_CONFIG_PATH="/usr/local/opt/python@3.7/lib/pkgconfig"
alias python=python3
This way, the python
command always goes to the x86 rosetta version of Python 3.7. I installed it with brew.
In the upcoming version I added manual search for python binary inside PATH: if binary not found, plugin will fail and error message would contain PATH value. This should simplify revealing such problems.
That's a good debugging feature. For now, I guess it works for me.
New version 3.0.0 released with auto PATH validation