Gradle does not recognize Pyenv environment

Question

Gradle does not recognize Pyenv environment

alexcombessie opened this issue 2 years ago · comments

I am using pyenv to manage Python environments on my Mac.

alexcombessie@Alex-Air-001 giskard % pyenv version 
3.7.13 (set by /Users/alexcombessie/Github/giskard/.python-version)

alexcombessie@Alex-Air-001 giskard % python --version
Python 3.7.13

alexcombessie@Alex-Air-001 giskard % which python               
/Users/alexcombessie/.pyenv/rosetta/shims/python

Unfortunately, this is not recognized in my gradle build. Gradle never finds which python from pyenv and keeps using the default system package of python, which are of the wrong version.

I tried adding a pythonBinary argument, with different values: "python", "python3.7", or even "/Users/alexcombessie/.pyenv/rosetta/shims/python3.7' but it always failed to build, as the argument was not recognized. The last trial gave me this error message:

Python not found: .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python3.7. This must be a bug of virtualenv support, please report it (https://github.com/xvik/gradle-use-python-plugin/issues). You can disable virtualenv usage with ‘python.scope = USER’.

PS: I do need to set scope = 'VIRTUALENV' as I need the virtualenv for another part of the app.
FYI @andreybavt

Vyacheslav Rusakov · Answer 1 · Wed Aug 10 2022 19:32:37 GMT+0800 (China Standard Time)

Possibly, this would help:

python {
    pythonPath = '/Users/alexcombessie/.pyenv/rosetta/shims/'
    // pythonBinary = 'python'
}

I will try to reproduce this with pyenv (never used it before) and try to fix the default behaviour if possible.

Alex Combessie · Answer 2 · Wed Aug 10 2022 20:13:49 GMT+0800 (China Standard Time)

Hi @xvik ,

Thanks for your swift reply. I confirm that with the pythonPath you added, gradle correctly picks up Python from pyenv.

Would it be possible to make this the default behavior? The issue is that I cannot commit this workaround, as it would be only specific to my local dev setup.

Thanks,

Alex

Vyacheslav Rusakov · Answer 3 · Wed Aug 10 2022 20:37:01 GMT+0800 (China Standard Time)

You can also try pythonPath = '~/.pyenv/rosetta/shims/' - if it would work, it should be acceptable for commit if other devs also use pyenv.

I'll, of course, look, but it might take some time. From what I read on pyenv site, it should already work properly.
The path in your error is confusing: .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python3.7.. Don't know how .venv/bin/ prefix appeared there. Do you have a venv created in your project? Or, maybe, know what is .venv dir.

Also, could you please attach full gradle log for failed execution: it should be clear from there if plugin detects some venv and tries to use as virtualenv (instead of using global python for virtulenv creation)

Alex Combessie · Answer 4 · Wed Aug 10 2022 20:55:14 GMT+0800 (China Standard Time)

You can also try pythonPath = '~/.pyenv/rosetta/shims/' - if it would work, it should be acceptable for commit if other devs also use pyenv.

Unfortunately that is too brittle for us, we would rather use the default system way of detecting python from PATH. Not all of us use Pyenv. But everyone has python in their PATH. When using our shell (sh / zsh / bash) python is recognized. The problem is isolated to gradle.

To reproduce, here is a sample of my build.gradle

python {
    pythonBinary = '/Users/alexcombessie/.pyenv/rosetta/shims/python'
    envPath = '.venv'
    minPythonVersion = '3.7'
    scope = 'VIRTUALENV'
    installVirtualenv = true
    pip 'poetry:1.1.13'
    environment "PYTHONPATH", file('generated')
}

task install(type: PythonTask) {
    dependsOn python: pipInstall

    module = "poetry"
    command "install"

}

Output:

lexcombessie@Alex-Air-001 giskard % ./gradlew :giskard-ml-worker:build --info
Initialized native services in: /Users/alexcombessie/.gradle/native
Initialized jansi services in: /Users/alexcombessie/.gradle/native
The client will now receive all logging from the daemon (pid: 23567). The daemon log file: /Users/alexcombessie/.gradle/daemon/7.4/daemon-23567.out.log
Starting 29th build in daemon [uptime: 3 hrs 26 mins 18.575 secs, performance: 100%, non-heap usage: 41% of 256 MiB]
Using 10 worker leases.
Now considering [/Users/alexcombessie/Github/giskard] as hierarchies to watch
Watching the file system is configured to be enabled if available
File system watching is active
Starting Build
Settings evaluated using settings file '/Users/alexcombessie/Github/giskard/settings.gradle'.
Projects loaded. Root project using build file '/Users/alexcombessie/Github/giskard/build.gradle'.
Included projects: [root project 'giskard', project ':giskard-demo-notebook', project ':giskard-frontend', project ':giskard-ml-worker', project ':giskard-server']

> Configure project :
Evaluating root project 'giskard' using build file '/Users/alexcombessie/Github/giskard/build.gradle'.

> Configure project :giskard-demo-notebook
Evaluating project ':giskard-demo-notebook' using build file '/Users/alexcombessie/Github/giskard/giskard-demo-notebook/build.gradle'.

> Configure project :giskard-frontend
Evaluating project ':giskard-frontend' using build file '/Users/alexcombessie/Github/giskard/giskard-frontend/build.gradle'.

> Configure project :giskard-ml-worker
Evaluating project ':giskard-ml-worker' using build file '/Users/alexcombessie/Github/giskard/giskard-ml-worker/build.gradle'.

> Configure project :giskard-server
Evaluating project ':giskard-server' using build file '/Users/alexcombessie/Github/giskard/giskard-server/build.gradle'.
Adding lombok-mapstruct-binding for source set main because DefaultExternalModuleDependency{group='org.mapstruct', name='mapstruct-processor', version='1.4.2.Final', configuration='default'} was found
------------------------------------------------------------------------
Detecting the operating system and CPU architecture
------------------------------------------------------------------------
os.detected.name=osx
os.detected.arch=aarch_64
os.detected.bitness=64
os.detected.version=12.5
os.detected.version.major=12
os.detected.version.minor=5
os.detected.classifier=osx-aarch_64
All projects evaluated.
Selected primary task ':giskard-ml-worker:build' from project :giskard-ml-worker
Tasks to be executed: [task ':giskard-ml-worker:assemble', task ':giskard-ml-worker:check', task ':giskard-ml-worker:checkPython', task ':giskard-ml-worker:pipInstall', task ':giskard-ml-worker:install', task ':giskard-ml-worker:generateProto', task ':giskard-ml-worker:test', task ':giskard-ml-worker:build']
Tasks that were excluded: []
:giskard-ml-worker:assemble (Thread[Execution worker for ':',5,main]) started.

> Task :giskard-ml-worker:assemble UP-TO-DATE
Skipping task ':giskard-ml-worker:assemble' as it has no actions.
:giskard-ml-worker:assemble (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:giskard-ml-worker:check (Thread[Execution worker for ':',5,main]) started.

> Task :giskard-ml-worker:check UP-TO-DATE
Skipping task ':giskard-ml-worker:check' as it has no actions.
:giskard-ml-worker:check (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:giskard-ml-worker:checkPython (Thread[Execution worker for ':',5,main]) started.

> Task :giskard-ml-worker:checkPython FAILED
Caching disabled for task ':giskard-ml-worker:checkPython' because:
  Build cache is disabled
Task ':giskard-ml-worker:checkPython' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
Python arguments: -c
[python] .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python -c exec("import sys;ver=sys.version_info;print(str(ver.major)+'.'+str(ver.minor)+'.'+str(ver.micro));print(sys.prefix);print(sys.executable)")
Starting process 'command '.venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python''. Working directory: /Users/alexcombessie/Github/giskard/giskard-ml-worker Command: .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python -c exec("import sys;ver=sys.version_info;print(str(ver.major)+'.'+str(ver.minor)+'.'+str(ver.micro));print(sys.prefix);print(sys.executable)")
:giskard-ml-worker:checkPython (Thread[Execution worker for ':',5,main]) completed. Took 0.004 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':giskard-ml-worker:checkPython'.
> Python not found: .venv/bin//Users/alexcombessie/.pyenv/rosetta/shims/python. This must be a bug of virtualenv support, please report it (https://github.com/xvik/gradle-use-python-plugin/issues). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 535ms

Vyacheslav Rusakov · Answer 5 · Thu Aug 11 2022 17:24:57 GMT+0800 (China Standard Time)

pythonBinary declares just a name of the binary so it is not correct to declare complete path here (it should be done with pythonPath).

To better understand how it works:

When there is no created virtualenv, python would be called as ${pythonPath}/${pythonBinary}
When virtualenv folder exists then: ${envPath}/bin/${pythonBinary} (pythonBinary remains the same in both cases and pythonPath gets substituted to switch into virtualenv folder)

On non windows host, when pythonBinary is not declared, plugin would try to use python3 first and then fallback to python.
So if python3 binary is not recognized then the default should work (will use python). Overall, the following configuration should work properly:

python {
    pythonBinary = 'python'
    envPath = '.venv'
    minPythonVersion = '3.7'
    scope = 'VIRTUALENV'
    installVirtualenv = true
    pip 'poetry:1.1.13'
    environment "PYTHONPATH", file('generated')
}

If it doesn't then, most likley, you have a problem with PATH. For example, when intelliJ IDEA is not launched from shell it will not inherit PATH customizations (from bashrc) and running gradle within it will not find pyenv python.
Running gradle (or idea) directly from shell should work correctly.

You can verify PATH correctness by putting println System.getenv('PATH') inside your build.gradle.

If you use bash and PATH is modified for pyenv in .bashrc then you can move it into ~/.profile and log-out/log-in. Probably would help, but can't check now (tip from here).

I don't see other problems in plugin itself: at least I was able to use pyenv python on ubuntu.

Alex Combessie · Answer 6 · Fri Aug 12 2022 16:28:08 GMT+0800 (China Standard Time)

Thanks! That gives me enough information for debugging. Since you say it works on Ubuntu, I suspect my issue is that I am on an M1 mac, with rosetta mode, and multiple Python depending on the arch.
I will debug the PATH on my end.

Nataly Merezhuk · Answer 7 · Wed Aug 31 2022 05:51:44 GMT+0800 (China Standard Time)

I am running into the same problem! @alexcombessie did you have any luck debugging this?

Alex Combessie · Answer 8 · Wed Aug 31 2022 14:51:54 GMT+0800 (China Standard Time)

Yes. It seems there was in issue with the dual ARM/x86 nature of M1 MacBooks with Rosetta emulation.

For some reason, when I launch a gradle command from a shell which is on the x86 arch, it ignores the shell context and uses the default ARM arch.

I found a workaround by hardcording the default python in my zshrc to a x86 binary of python.

I guess gradle always starts a new shell?

Vyacheslav Rusakov · Answer 9 · Wed Aug 31 2022 15:21:13 GMT+0800 (China Standard Time)

In the upcoming version I added manual search for python binary inside PATH: if binary not found, plugin will fail and error message would contain PATH value. This should simplify revealing such problems.

@alexcombessie could you please provide exact sample command for this workaround? I'll put it in readme as a reference for others (I hope it would be useful)

I guess gradle always starts a new shell?

Gradle itself - no. Gradle inherits PATH from the shell launching it. But, in some cases, running shells indeed could differ. For example, on ubuntu, PATH changes from .bashrc might be not visible when gradle launched from IDE, launched from system launcher.

I found interesting stackoverflow answer: the problem might be in jvm - if gradle launched with jvm compiled for x86 - it will run through rosetta. (edit: oh, I missed, you mention problem is different in the previous comment)

Alex Combessie · Answer 10 · Wed Aug 31 2022 19:01:48 GMT+0800 (China Standard Time)

@alexcombessie could you please provide exact sample command for this workaround? I'll put it in readme as a reference for others (I hope it would be useful)

My workaround is very dirty, I hardcoded these lines into my .zshrc:

export PATH="/usr/local/opt/python@3.7/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/python@3.7/lib"
export PKG_CONFIG_PATH="/usr/local/opt/python@3.7/lib/pkgconfig"
alias python=python3

This way, the python command always goes to the x86 rosetta version of Python 3.7. I installed it with brew.

In the upcoming version I added manual search for python binary inside PATH: if binary not found, plugin will fail and error message would contain PATH value. This should simplify revealing such problems.

That's a good debugging feature. For now, I guess it works for me.

Vyacheslav Rusakov · Answer 11 · Sat Oct 22 2022 21:41:27 GMT+0800 (China Standard Time)

New version 3.0.0 released with auto PATH validation