Giters

matthewfeickert / cvmfs-venv

Example implementation of getting a Python virtual environment to work with CVMFS LCG views

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add option to work with AnalysisBase

matthewfeickert opened this issue · comments

commented

To be more useful this should also work with ATLAS AnlaysisBase (both release 21 and release 22).

Just the following diff gets things working for AnalysisBase 22.2.61

diff --git a/atlas_setup.sh b/atlas_setup.sh
index 3ef41a7..2908e95 100644
--- a/atlas_setup.sh
+++ b/atlas_setup.sh
@@ -19,8 +19,14 @@ if [ "$(python3 -c 'from platform import platform; print("centos-8" in platform(
     default_LCG_release="LCG_100"
     default_LCG_platform="x86_64-centos8-gcc10-opt"
 fi
-printf "\nlsetup 'views %s %s'\n" "${default_LCG_release}" "${default_LCG_platform}"
-lsetup "views ${default_LCG_release} ${default_LCG_platform}"
+# printf "\nlsetup 'views %s %s'\n" "${default_LCG_release}" "${default_LCG_platform}"
+# lsetup "views ${default_LCG_release} ${default_LCG_platform}"
+
+# _ATLAS_RELEASE="21.2.203"
+_ATLAS_RELEASE="22.2.61"
+printf "\nasetup %s,AnalysisBase\n" "${_ATLAS_RELEASE}"
+asetup "${_ATLAS_RELEASE},AnalysisBase"
+unset _ATLAS_RELEASE
 
 _venv_name="${1:-venv}"
 if [ ! -d "${_venv_name}" ]; then
Example with AnalysisBase 22.2.61: 
(base) feickert@ThinkPad-X1:~$ ssh uchicago 
Last login: Fri Feb 11 15:02:28 2022 from 216.169.6.123
    _ _____ _      _   ___ 
   /_\_   _| |    /_\ / __|
  / _ \| | | |__ / _ \\__ \
 /_/ \_\_| |____/_/ \_\___/
Analysis Facility at UChicago
-----------------------------

-====== DOCUMENTATION ======-
Read the Docs: 
  https://usatlas.readthedocs.io/projects/af-docs/en/latest/

-======== SUPPORT ==========-
Discourse:
  https://atlas-talk.web.cern.ch/c/us-tier-3-user-support/
Admin list:
  atlas-us-chicago-tier3-admins@cern.ch

File system quota report
╒════════╤══════════════╤══════════════╤═════════════╤═════════════╕
│ Path   │   Usage (GB) │   Limit (GB) │   Files (#) │   Quota (%) │
╞════════╪══════════════╪══════════════╪═════════════╪═════════════╡
│ /home  │         5.83 │          100 │       52075 │        5.83 │
├────────┼──────────────┼──────────────┼─────────────┼─────────────┤
│ /data  │         0.2  │         5120 │        1714 │        0    │
╘════════╧══════════════╧══════════════╧═════════════╧═════════════╛

[15:19] login02.af.uchicago.edu:~ $ cd debug/
# Get patched file
[15:20] login02.af.uchicago.edu:~/debug $ . atlas_setup.sh 

asetup 22.2.61,AnalysisBase
Using AnalysisBase/22.2.61 [cmake] with platform x86_64-centos7-gcc11-opt
	at /cvmfs/atlas.cern.ch/repo/sw/software/22.2
Unchanged: COOL_ORA_ENABLE_ADAPTIVE_OPT=Y
# Creating new Python virtual environment 'venv'
(venv) [15:20] login02.af.uchicago.edu:~/debug $ echo $PYTHONPATH
/home/feickert/debug/venv/lib/python3.9/site-packages:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/python:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/lib:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/lib:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/python:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
(venv) [15:20] login02.af.uchicago.edu:~/debug $ command -v python
/home/feickert/debug/venv/bin/python
(venv) [15:20] login02.af.uchicago.edu:~/debug $ python --version
Python 3.9.6
(venv) [15:21] login02.af.uchicago.edu:~/debug $ python -m pip list
Package    Version
---------- -------
Cython     0.29.21
LHAPDF     6.3.0
numpy      1.21.1
pip        22.0.3
setuptools 60.8.2
wheel      0.37.1
xrootd     5.3.1
(venv) [15:21] login02.af.uchicago.edu:~/debug $ python -m pip install awkward
Collecting awkward
  Downloading awkward-1.7.0-cp39-cp39-manylinux2010_x86_64.whl (14.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.5/14.5 MB 260.0 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.13.1 in /cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/numpy-1.21.1-py3.9-linux-x86_64.egg (from awkward) (1.21.1)
Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (from awkward) (60.8.2)
Installing collected packages: awkward
Successfully installed awkward-1.7.0
(venv) [15:22] login02.af.uchicago.edu:~/debug $ python -m pip list
Package    Version
---------- -------
awkward    1.7.0
Cython     0.29.21
LHAPDF     6.3.0
numpy      1.21.1
pip        22.0.3
setuptools 60.8.2
wheel      0.37.1
xrootd     5.3.1
(venv) [15:22] login02.af.uchicago.edu:~/debug $ pip show awkward
Name: awkward
Version: 1.7.0
Summary: Manipulate JSON-like data with NumPy-like idioms.
Home-page: https://github.com/scikit-hep/awkward-1.0
Author: Jim Pivarski
Author-email: pivarski@princeton.edu
License: BSD-3-Clause
Location: /home/feickert/debug/venv/lib/python3.9/site-packages
Requires: numpy, setuptools
Required-by: 
(venv) [15:22] login02.af.uchicago.edu:~/debug $ deactivate 
[15:22] login02.af.uchicago.edu:~/debug $ echo $PYTHONPATH
/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/python:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBase/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/lib:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/lib:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/python:/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
[15:22] login02.af.uchicago.edu:~/debug $ command -v python  
/cvmfs/atlas.cern.ch/repo/sw/software/22.2/AnalysisBaseExternals/22.2.61/InstallArea/x86_64-centos7-gcc11-opt/bin/python
[15:23] login02.af.uchicago.edu:~/debug $ python -m pip list
Package    Version
---------- -------
Cython     0.29.21
LHAPDF     6.3.0
numpy      1.21.1
pip        21.0.1
setuptools 44.1.0
wheel      0.33.4
xrootd     5.3.1

though for AnalysisBase 21 there are problems as things are still Python 2 there. This is a big problem as the venv module wasn't added to Python until Python 3:

$ docker run --rm python:2.7 /bin/bash -c 'python --version --version; python -m venv --help'
Python 2.7.18
/usr/local/bin/python: No module named venv
commented

Hi Matthew,

This looks great! I really like it and it seems to work very well with python3 and R22. As you mention, setting up python2 with Analysisbase is a real pain. We (myself and a postdoc in Manchester) tried to do this with virtualenv rather than venv, and I think I might have managed to get it to work. The problem is that virtualenv provided on lxplus doesn’t work, and pip2 out-of-the-box fails to install it. The only way I could get it to work was to bootstrap install pip2 locally and then install virtualenv from it with the —ignore-installed flag. Then I replace the line

python3 -m venv "${_venv_name}"

with

/afs/cern.ch/user/m/maly/.local/bin/virtualenv "${_venv_name}"

and I had to comment out the check for CENTOS8 since it tries to use python3. Starting from a fresh-shell, this seems to work.

I'm not sure how one would nicely integrate an bootsrap installation of pip2 in your code though, or if there is a way around that.

Thank you very much,
Mo

commented

@MoAly98 yeah for Python 2 virtualenv is pretty much the only way to go. I think the alternative method of acquiring it, compared to what you show, would be do do something similar to manually downloading and installing virtualenv from the PyPA's bootstrap project. Though that seems like it will also need to be revised as the subdirectory of https://bootstrap.pypa.io/ that was kept for Python 2.7 scripts is gone, and the Python 3 version of virtualenv doesn't work with Python 2 (which is a totally reasonable thing. Python 2 has been EOL for over 2 years).

commented

Notes to clean up later:

$ setupATLAS
$ asetup "21.2.203,AnalysisBase"
$ curl -sL --location --output /tmp/virtualenv.pyz https://bootstrap.pypa.io/virtualenv/2.7/virtualenv.pyz
$ command -v python2
/cvmfs/atlas.cern.ch/repo/sw/software/21.2/AnalysisBaseExternals/21.2.203/InstallArea/x86_64-centos7-gcc8-opt/bin/python2
$ python2 --version 
Python 2.7.16
$ python2 /tmp/virtualenv.pyz ~/py2_debug/venv
Traceback (most recent call last):
  File "/cvmfs/atlas.cern.ch/repo/sw/software/21.2/AnalysisBaseExternals/21.2.203/InstallArea/x86_64-centos7-gcc8-opt/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/cvmfs/atlas.cern.ch/repo/sw/software/21.2/AnalysisBaseExternals/21.2.203/InstallArea/x86_64-centos7-gcc8-opt/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/virtualenv.pyz/__main__.py", line 167, in <module>
  File "/tmp/virtualenv.pyz/__main__.py", line 161, in run
  File "/tmp/virtualenv.pyz/virtualenv/__init__.py", line 3, in <module>
  File "/tmp/virtualenv.pyz/virtualenv/run/__init__.py", line 7, in <module>
  File "/tmp/virtualenv.pyz/virtualenv/app_data/__init__.py", line 12, in <module>
  File "/tmp/virtualenv.pyz/virtualenv/app_data/read_only.py", line 3, in <module>
  File "/tmp/virtualenv.pyz/virtualenv/util/lock.py", line 13, in <module>
  File "/tmp/virtualenv.pyz/virtualenv/util/path/__init__.py", line 3, in <module>
  File "/tmp/virtualenv.pyz/virtualenv/util/path/_pathlib/__init__.py", line 14, in <module>
  File "/tmp/virtualenv.pyz/__main__.py", line 153, in load_module
  File "<string>", line 60, in <module>
  File "/tmp/virtualenv.pyz/__main__.py", line 153, in load_module
  File "<string>", line 584, in <module>
AttributeError: 'module' object has no attribute 'DirEntry'

Though python2's virtualenv seems quite broken everywhere

$ docker run --rm -ti python:2.7 /bin/bash
root@d71d9f265349:/# rm -rf /usr/local/bin/virtualenv
root@d71d9f265349:/# rm -rf /usr/local/lib/python2.7/site-packages/virtualenv
root@d71d9f265349:/# command -v virtualenv
root@d71d9f265349:/# find / -type d -iname virtualenv
root@d71d9f265349:/# curl -sL --location --output /tmp/virtualenv.pyz https://bootstrap.pypa.io/virtualenv/2.7/virtualenv.pyz
root@d71d9f265349:/# python2 /tmp/virtualenv.pyz debug/venv
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/virtualenv.pyz/__main__.py", line 167, in <module>
  File "/tmp/virtualenv.pyz/__main__.py", line 163, in run
  File "/tmp/virtualenv.pyz/virtualenv/__main__.py", line 18, in run
  File "/tmp/virtualenv.pyz/virtualenv/run/__init__.py", line 30, in cli_run
  File "/tmp/virtualenv.pyz/virtualenv/run/__init__.py", line 48, in session_via_cli
  File "/tmp/virtualenv.pyz/virtualenv/run/__init__.py", line 75, in build_parser
  File "/tmp/virtualenv.pyz/virtualenv/run/plugin/seeders.py", line 8, in __init__
  File "/tmp/virtualenv.pyz/virtualenv/run/plugin/base.py", line 46, in options
  File "/tmp/virtualenv.pyz/virtualenv/run/plugin/base.py", line 25, in entry_points_for
  File "/usr/local/lib/python2.7/collections.py", line 69, in __init__
    self.__update(*args, **kwds)
  File "/usr/local/lib/python2.7/_abcoll.py", line 571, in update
    for key, value in other:
  File "/tmp/virtualenv.pyz/virtualenv/run/plugin/base.py", line 25, in <genexpr>
  File "<string>", line 105, in load
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named via_app_data.via_app_data
root@d71d9f265349:/#
commented

The problem is that virtualenv provided on lxplus doesn’t work, and pip2 out-of-the-box fails to install it. The only way I could get it to work was to bootstrap install pip2 locally

@MoAly98 can you comment on what you actually did to do a bootstrap install of pip2?

commented

The way I did it was:

curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
python2 get-pip.py --target=./pip

and then got virtualenv

./pip/bin/pip2.7 install --user virtualenv --ignore-installed
commented

@MoAly98 So I had attempted that before but the method you describe in a Release 21 AnalysisBase environment causes the pip2 to be broken.

$ setupATLAS
$ asetup "21.2.203,AnalysisBase"
$ command -v python2
/cvmfs/atlas.cern.ch/repo/sw/software/21.2/AnalysisBaseExternals/21.2.203/InstallArea/x86_64-centos7-gcc8-opt/bin/python2
$ mkdir py2_debug
$ cd py2_debug
$ curl -sLO https://bootstrap.pypa.io/pip/2.7/get-pip.py
$ python2 get-pip.py --target=./pip
$ ./pip/bin/pip2 --help
Traceback (most recent call last):
  File "./pip/bin/pip2", line 5, in <module>
    from pip._internal.cli.main import main
ImportError: No module named main

Similarly, running get-pip.py directly in a python2 -m pip install sort of way will get virtualenv into a user install by default, but then the resulting virtual environment is broken

$ python2 get-pip.py virtualenv
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Defaulting to user installation because normal site-packages is not writeable
...
$ command -v virtualenv
/home/feickert/.local/bin/virtualenv
$ virtualenv venv
created virtual environment CPython2.7.16.final.0-64 in 1130ms
  creator CPython2Posix(dest=/home/feickert/py2_debug/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, setuptools=bundle, via=copy, app_data_dir=/home/feickert/.local/share/virtualenv)
    added seed packages: pip==20.3.4, setuptools==44.1.1, wheel==0.37.1
  activators NushellActivator,PythonActivator,FishActivator,CShellActivator,PowerShellActivator,BashActivator
$ . venv/bin/activate
(venv) $ command -v python
/home/feickert/py2_debug/venv/bin/python
(venv) $ python -m pip --version
Traceback (most recent call last):
  File "/cvmfs/atlas.cern.ch/repo/sw/software/21.2/AnalysisBaseExternals/21.2.203/InstallArea/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/site.py", line 73, in <module>
    __boot()
  File "/cvmfs/atlas.cern.ch/repo/sw/software/21.2/AnalysisBaseExternals/21.2.203/InstallArea/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/site.py", line 3, in __boot
    import os
  File "/home/feickert/py2_debug/venv/lib/python2.7/os.py", line 49, in <module>
    import posixpath as path
ImportError: No module named posixpath
commented

Resolved by PR #18 for AnalysisBase releases that use Python 3 (Release 22 onwards).