Very slow CMake O2 installation on macOS

RPATH requires modifying a binary file when you relocate or change the layout of the installation (as we do when deploy on the grid, for example) in order to work. The current approach based on the environment does not need that. Now I agree that constructing such large PATH / LD_LIBRARY_PATH is not a good idea, but I would investigate the concept of “views” rather than hardcode every single install location in our executables and libraries.

Put otherwise, I’ve played in the past with both approaches and ditched the RPATH one as the were too many failure modes for the way we (HEP) deploy stuff. Clearly, if you are a supercomputing centre which has full control on its own installation, that’s a different story.

I agree that modifying binary files is probably not a good thing. But telling users to switch off host-wide security features to get a single application working seems at least equally worrysome. Also Linux has security features that effectively ignore LD_LIBRARY_PATH.

I wonder, if we could just use relative RPATHs from the beginning for package internal reference and across packages? That way one could change the prefix of the sw directory as often as you want. (On mac I can move my .app directory and the app keeps working, I doubt this is accomplished through LD_LIBRARY_PATH mgmt ?!) Only references to libs picked up from the system would stay absolute (edit: not true, if it is a system prefix)

How is the installation layout changed for the grid exactly and why? (If I can learn about this somewhere, I am also happy for a pointer to the appropriate documents, thx)

The problem is that on the grid we would need to use a completely different relative path compared to the one we use on laptops. Again, I think we should have mechanism which creates “views” where all the transitive closure of dependencies for a given package appears in the same location (as for .app) and then we can have RPATH set to use that single entry point completely.

Looks like nix (they also install each package to different directory) solves this by using a custom ld.so.cache. I assume the cache file will accelerate the library lookup significantly. But this is not an option for us, since one cannot add a custom ld.so.cache :frowning:

Let’s patch ld :stuck_out_tongue:

Unless, it is not the system one; Using a custom dynamic loader seems possible:

-Wl,--rpath=/path/to/newglibc -Wl,--dynamic-linker=/path/to/newglibc/ld-linux.so.2

People have already build more sophisticated ld switches even, see: http://bitwagon.com/rtldi/rtldi.html.

p.s.: This is just me brainstorming/exploring ideas :wink:

Small update about this. Other people (including GCC developers) have noticed a regression on anything which does a DYLD_LIBRARY_PATH lookup, to be seen if the issue will actually be considered a bug by Apple.

A workaround I am currently using is:

diff --git a/o2.sh b/o2.sh
index e0bc29f..dbdc809 100644
--- a/o2.sh
+++ b/o2.sh
@@ -96,6 +96,7 @@ if [[ $ALIBUILD_O2_TESTS ]]; then
   CXXFLAGS="${CXXFLAGS} -Werror"
 fi

+unset DYLD_LIBRARY_PATH
 cmake $SOURCEDIR -DCMAKE_INSTALL_PREFIX=$INSTALLROOT                                        \
       ${CMAKE_GENERATOR:+-G "$CMAKE_GENERATOR"}                                             \
       -DCMAKE_MODULE_PATH="$SOURCEDIR/cmake/modules;$FAIRROOT_ROOT/share/fairbase/cmake/modules;$FAIRROOT_ROOT/share/fairbase/cmake/modules_old"  \

not sure if this is acceptable, as I do not know how anything which looks up for a plugin will work in this particular case.

I confirm that the DYLD_LIBRARY_PATH makes the difference.
Compile time for O2 outside of alienv (just by saying make -j4 in the build dir) is 8min while inside alienv (or using aliBuild) it is ~20min.

The same problem applies to anything done within a loaded alienv environment. Try to do make clean after saying alienv enter O2/latest… Likely builds of other packages are affected, too.

So maybe with this understanding the MacOS CI will get faster?

For the record, enabling SIP now works, at least for my stuff, so probably that’s the best think to do.