aliBuild O2 and pyenv

O²
1

In order to bring back some sanity to my machine (or so I thought…) I decided to rely solely on pyenv to handle all my python versions (i.e. no more python from brew for instance).

So I naively created a virtualenv for Alice (based first on python 3.7.3 and then on 3.6.5 with the same result), and tried aliBuild --defaults o2 build O2.

Unfortunately the build fails in Python-modules with some obscure (to me at least) message about “python not being installed as a framework”

Was the pyenv + virtualenv a bad idea from the start or is the error just a glitch that’s easy to fix ?

log: https://cernbox.cern.ch/index.php/s/KdUkdLjk9ggvnDX

2

Well, I was a bit fast asking for help…

Reading https://matplotlib.org/2.2.3/faq/osx_framework.html I was able to get past the Python-modules compilation.

For the record, I re-installed the python version I use for Alice using :

PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.7.3

3

For the record, there is another problem down the line, now in lhapdf. Investigating…

4

OK, I think I can now tell that :

  • LHAPDF is building fine on a « regular » Mac with python3 from brew
  • LHAPDF is not compiling on a Mac with an isolated python3 env with Pyenv (i.e. a Mac where the build process has only access to python3)

Thing is LHAPDF python extension is built with python2 on the « regular » Mac even though the rest of aliBuild is using python3.

Looking at the Makefile it looks like if Cython is found it should be reran, but for some reason cython is not found even if present.

To get to ball rolling I re-ran cython by hand :

> cd lhapdf/wrappers/python
> cython lhapdf.pyx —cplus

And then the build is going fine.

I guess we could add those lines into the recipe (plus the installation of the cython module…) ?

5

Hi Laurant,

I’m also using virtualenv on my system. For this I had to modify python-modules.sh.
It is mainly removing the --user and all cleanup of the local installation:

diff --git a/python-modules.sh b/python-modules.sh
index 13caefd..c5ded85 100644
--- a/python-modules.sh
+++ b/python-modules.sh
@@ -53,7 +53,10 @@ fi
 for P in "${PIP_REQUIREMENTS[@]}"; do
   echo $P | cut -d' ' -f1
 done > requirements.txt
-env PYTHONUSERBASE="$INSTALLROOT" pip3 install --user -IU -r requirements.txt
+if [ -n "$VIRTUAL_ENV" ]; then
+  env PYTHONUSERBASE="$INSTALLROOT" pip3 install -IU -r requirements.txt
+else
+  env PYTHONUSERBASE="$INSTALLROOT" pip3 install --user -IU -r requirements.txt
 
 # Find the proper Python lib library and export it
 pushd "$INSTALLROOT"
@@ -69,6 +72,7 @@ pushd "$INSTALLROOT"
     rm -f *.deleteme || true
   popd
 popd
+fi
 
 # Check if the modules can be loaded
 ERR_IMPORT=()
@@ -89,7 +93,11 @@ fi
 MATPLOTLIB_TAG="3.0.3"
 if [[ $ARCHITECTURE != slc* ]]; then
   # Simply get it via pip in most cases
+if [ -n "$VIRTUAL_ENV" ]; then
+  env PYTHONUSERBASE=$INSTALLROOT pip3 install "matplotlib==$MATPLOTLIB_TAG"
+else
   env PYTHONUSERBASE=$INSTALLROOT pip3 install --user "matplotlib==$MATPLOTLIB_TAG"
+fi
 else
 
   # We are on a RHEL-compatible OS. We compile it ourselves, and link it to our dependencies
@@ -126,6 +134,7 @@ fi
 env PYTHONPATH="$INSTALLROOT/lib/python/site-packages" python3 -c 'import matplotlib'
 
 # Patch long shebangs (by default max is 128 chars on Linux)
+if [ -z "$VIRTUAL_ENV" ]; then
 pushd "$INSTALLROOT/bin"
   sed -i.deleteme -e '1 s|^#!.*$|#!/usr/bin/env python3|' * || true
   rm -f *.deleteme
@@ -136,6 +145,7 @@ rm -rvf "$INSTALLROOT"/share "$INSTALLROOT"/lib/python*/test
 find "$INSTALLROOT"/lib/python* \
      -mindepth 2 -maxdepth 2 -type d -and \( -name test -or -name tests \) \
      -exec rm -rvf '{}' \;
+fi
 
 # Modulefile
 MODULEDIR="$INSTALLROOT/etc/modulefiles"
6

The situation with LHAPDF is a bit more complex. In a nutshell it’s not python3 ready (yet). So I’ve disabled the building of the python extension if only Python3 is available to the build https://github.com/alisw/alidist/pull/1623

7

Hi Jens,

In my case I did not have to change python-modules. As far as I can tell python-modules.sh installs stuff into its own (user) location, so it should not interfere at all with virtualenvs ?