Feature Request: support environments with both Python 2 and 3 present
jspiewak opened this issue · comments
On shinyapps.io we have both Python 2 and 3 present: /usr/bin/python and /usr/bin/python3
Locally, I have both as well, with /usr/local/bin/python3.
Ordinarily with virtualenv I pass --python python3 and it will determine the full path.
There does not appear to be a way today with reticulate to easily specify python3 should be used when calling virtualenv_create.
I managed to create a python3 environment with system2('/usr/local/bin/virtualenv', shQuote(c('--system-site-packages', '--python', '/usr/local/bin/python3', '/Users/jss/.virtualenvs/jssvenv'))). I was then able to use it, and install a package into it. However, I am not able to import that package. I assume that import is stuck on the r-reticulate env, because when I activate the virtualenv from the command line, I am able to import the package.
@kevinushey I noticed that our virtualenv_* functions have the limitation that they will always look for Python 2 first and then only if it's not found look for and use Python 3. This is fine for e.g. the vanilla state of Ubuntu 18.04 (which has no Python 2) but for any system with both it effectively prevents the use of Python 3 with the virutalenv_* functions. How about if we add a python3 = FALSE flag to the signatures of all of these functions and then pass it down to virtualenv_config() (which would then skip the check for python 2 if the flag is set).
If that doesn't sound like the right approach let's certainly discuss. But I think we have a pretty big issue to address here as Python 3 becomes more prevalent (when these functions were originally developed for the TF scenario the DL community was actively discouraging the use of Python 3 so it wasn't really a problem to be so Python 2 centric).
@jjallaire Can you share any updates on this? In 3 months, Python 2.7 will no longer be maintained. Has any progress been made to prevent reticulate from defaulting to Python 2 (or allowing the option to force Python 3 usage in environments where both are present?)
As an example, if I create a Python3 virtualenv like this:
reticulate::virtualenv_create('shiny-app-env', python = 'python3')
reticulate::use_virtualenv('shiny-app-env', required = T)
and deploy to shinyapps.io, I see the following logs:
Creating virtual environment 'shiny-app-env' ...
Using python: python3
New python executable in /home/shiny/.virtualenvs/shiny-app-env/bin/python2
Also creating executable in /home/shiny/.virtualenvs/shiny-app-env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
showing that, although reticulate reports creating a Python 3 virtualenv, it is still using a Python 2 executable.
@kevinushey Are we at a point where we can now have reticulate prefer Python 3 to Python 2 (seems appropriate given the deprecation schedule!)
I think we can start defaulting to Python 3.
That said, this doesn't really answer what's happening in @ranikay's case -- it seems like there is some sort of confusion in what's going on in the shinyapps.io deployment:
> Using python: python3
> New python executable in /home/shiny/.virtualenvs/shiny-app-env/bin/python2
I'm not sure why it would report that python3 is being used, but a python2 virtual environment is being created.
@ranikay I wasn't able to reproduce -- I tried cloning your project and publishing to shinyapps.io, but it looks like a Python 3.5 environment was created and used as expected.
What versions of rsconnect and reticulate do you have installed? Does installing the latest development version from GitHub make a difference? FWIW, in my case I have:
> packageVersion("reticulate")
[1] ‘1.13’
> packageVersion("rsconnect")
[1] ‘0.8.15’
In my case, reticulate is latest from CRAN and rsconnect is the latest development version from GitHub.
Thanks @kevinushey. I have the same versions of rsconnect and reticulate:
> packageVersion("reticulate")
[1] ‘1.13’
> packageVersion("rsconnect")
[1] ‘0.8.15’
Could you have any environment variables set that may be causing the difference? Did you include that exact .Rprofile file from my project when deploying, which set RETICULATE_PYTHON? Or did you set any of the rsconnect options prior to deploying?
Sorry, I spoke too soon. I thought the usage of the python35 in the deployed application implied that Python 3 was indeed being used. In fact, I see the same thing in my deployment log:
2019-10-07T17:35:19.085766+00:00 shinyapps[1295771]:
2019-10-07T17:35:19.085768+00:00 shinyapps[1295771]: Attaching package: ‘DT’
2019-10-07T17:35:19.085769+00:00 shinyapps[1295771]:
2019-10-07T17:35:19.086826+00:00 shinyapps[1295771]: The following objects are masked from ‘package:shiny’:
2019-10-07T17:35:19.086828+00:00 shinyapps[1295771]:
2019-10-07T17:35:19.086830+00:00 shinyapps[1295771]: dataTableOutput, renderDataTable
2019-10-07T17:35:19.086830+00:00 shinyapps[1295771]:
2019-10-07T17:35:22.019186+00:00 shinyapps[1295771]: Creating virtual environment 'python35_env' ...
2019-10-07T17:35:22.019189+00:00 shinyapps[1295771]: Using python: /usr/bin/python3
2019-10-07T17:35:22.381291+00:00 shinyapps[1295771]: New python executable in /home/shiny/.virtualenvs/python35_env/bin/python2
2019-10-07T17:35:22.381293+00:00 shinyapps[1295771]: Also creating executable in /home/shiny/.virtualenvs/python35_env/bin/python
2019-10-07T17:35:25.972368+00:00 shinyapps[1295771]: Installing setuptools, pkg_resources, pip, wheel...done.
2019-10-07T17:35:25.978669+00:00 shinyapps[1295771]: Running virtualenv with interpreter /usr/bin/python2
2019-10-07T17:35:26.468206+00:00 shinyapps[1295771]: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
2019-10-07T17:35:26.609121+00:00 shinyapps[1295771]: Collecting pip
2019-10-07T17:35:26.957170+00:00 shinyapps[1295771]: Using cached https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl
2019-10-07T17:35:27.057430+00:00 shinyapps[1295771]: Collecting wheel
2019-10-07T17:35:27.110122+00:00 shinyapps[1295771]: Collecting setuptools
2019-10-07T17:35:27.102928+00:00 shinyapps[1295771]: Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
2019-10-07T17:35:27.551461+00:00 shinyapps[1295771]: Installing collected packages: pip, wheel, setuptools
2019-10-07T17:35:27.504988+00:00 shinyapps[1295771]: Using cached https://files.pythonhosted.org/packages/6a/9a/50fadfd53ec909e4399b67c74cc7f4e883488035cfcdb90b685758fa8b34/setuptools-41.4.0-py2.py3-none-any.whl
2019-10-07T17:35:28.917629+00:00 shinyapps[1295771]: Using virtual environment 'python35_env' ...
2019-10-07T17:35:28.756865+00:00 shinyapps[1295771]: Successfully installed pip-19.2.3 setuptools-41.4.0 wheel-0.33.6
2019-10-07T17:35:29.935440+00:00 shinyapps[1295771]: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
2019-10-07T17:35:30.017383+00:00 shinyapps[1295771]: Collecting numpy
2019-10-07T17:35:30.412212+00:00 shinyapps[1295771]: Downloading https://files.pythonhosted.org/packages/d7/b1/3367ea1f372957f97a6752ec725b87886e12af1415216feec9067e31df70/numpy-1.16.5-cp27-cp27mu-manylinux1_x86_64.whl (17.0MB)
2019-10-07T17:35:32.377164+00:00 shinyapps[1295771]: Installing collected packages: numpy
2019-10-07T17:35:33.662982+00:00 shinyapps[1295771]: Successfully installed numpy-1.16.5
2019-10-07T17:35:35.656081+00:00 shinyapps[1295771]: Server version: 1.7.8-7
2019-10-07T17:35:35.656083+00:00 shinyapps[1295771]: LANG: en_US.UTF-8
2019-10-07T17:35:35.656127+00:00 shinyapps[1295771]: R version: 3.6.1
2019-10-07T17:35:35.656129+00:00 shinyapps[1295771]: shiny version: 1.3.2
2019-10-07T17:35:35.656748+00:00 shinyapps[1295771]: Using pandoc at /opt/connect/ext/pandoc2
2019-10-07T17:35:35.831733+00:00 shinyapps[1295771]: Using jsonlite for JSON processing
2019-10-07T17:35:35.839791+00:00 shinyapps[1295771]:
2019-10-07T17:35:35.656130+00:00 shinyapps[1295771]: httpuv version: 1.5.2
2019-10-07T17:35:35.866931+00:00 shinyapps[1295771]:
2019-10-07T17:35:35.656130+00:00 shinyapps[1295771]: rmarkdown version: (none)
2019-10-07T17:35:35.656139+00:00 shinyapps[1295771]: knitr version: (none)
2019-10-07T17:35:35.656140+00:00 shinyapps[1295771]: jsonlite version: 1.6
2019-10-07T17:35:35.656167+00:00 shinyapps[1295771]: RJSONIO version: (none)
2019-10-07T17:35:35.656175+00:00 shinyapps[1295771]: htmltools version: 0.4.0
2019-10-07T17:35:35.839793+00:00 shinyapps[1295771]: Starting R with process ID: '117'
2019-10-07T17:35:35.866934+00:00 shinyapps[1295771]: Listening on http://127.0.0.1:43861
2019-10-07T17:35:36.471139+00:00 shinyapps[1295771]: virtualenv: python35_env
2019-10-07T17:35:36.471282+00:00 shinyapps[1295771]: Using virtual environment 'python35_env' ...
2019-10-07T17:35:37.590793+00:00 shinyapps[1295771]: Collecting numpy
2019-10-07T17:35:37.498344+00:00 shinyapps[1295771]: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
2019-10-07T17:35:39.650316+00:00 shinyapps[1295771]: Using cached https://files.pythonhosted.org/packages/d7/b1/3367ea1f372957f97a6752ec725b87886e12af1415216feec9067e31df70/numpy-1.16.5-cp27-cp27mu-manylinux1_x86_64.whl
2019-10-07T17:35:40.205138+00:00 shinyapps[1295771]: Installing collected packages: numpy
2019-10-07T17:35:41.773092+00:00 shinyapps[1295771]: Successfully installed numpy-1.16.5
I'll see if I can learn more.
It looks like this was my misunderstanding -- right now, the virtual environment is created with the equivalent of:
/usr/bin/python3 -m virtualenv /path/to/env
(and this is being done because the venv module is not installed or available.)
However, this does not use the same version of Python to initialize the virtual environment. In fact, on my Ubuntu VM I see:
$ python3 -m virtualenv --help
Usage: virtualenv.py [OPTIONS] DEST_DIR
Options:
< ... >
-p PYTHON_EXE, --python=PYTHON_EXE
The Python interpreter to use, e.g.,
--python=python2.5 will use the python2.5 interpreter
to create the new environment. The default is the
python2 interpreter on your path (e.g.
/usr/bin/python2)
< ... >
So, by default, even if you invoke virtualenv from Python 3, a Python 2 virtual environment is created.
tl;dr: this is a bug in reticulate. I'll fix this soon. Sorry for the trouble!
Fedora rules :-) :-)
$ python3 -m virtualenv --help
Usage: virtualenv.py [OPTIONS] DEST_DIR
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
-p PYTHON_EXE, --python=PYTHON_EXE
The Python interpreter to use, e.g.,
--python=python3.5 will use the python3.5 interpreter
to create the new environment. The default is the
interpreter that virtualenv was installed with
(/usr/bin/python3)
This should be fixed now in the development version of reticulate. Thanks for reporting!
@ranikay Can you include your command for deploying the app to shinyapps.io?
I've tried recreating your example app, but I'm still getting an error:
Error in value[[3L]](cond) :
could not find a Python environment for ~/.virtualenvs/python35_env/bin/python
Definitely. For anyone having difficulties deploying an app to shinyapps.io that uses reticulate and Python 3, I made an end-to-end tutorial. It's a simple demo that shows how to configure virtual environments for use locally and on shinyapps.io, how to deploy a Shiny + reticulate app to shinyapps.io, and how to confirm that your app deployed on shinyapps.io is using the desired version of Python.
If you have any questions related to my demo app, feel free to open an issue in that repo and I'll help if I can!
For Shiny beginners, maybe it is worth mentioning, that one should NOT add the development version of reticulate within the Shiny app. Install the development version of reticulate locally and rsconnect will identify which reticulate to use on shinyapps.io.