AliBuild feature request for quick building O2

O²
1

Not sure which is the best way to suggest a feature to AliBuild. Please let me know if your prefer another venue.

  1. Add option on AliBuild to emulate the behavior of the quick rebuild option with “cd <build_directory>” followed by “make install” or “ninja install”. This way we ensure that the environment variables/search paths are exactly the same when building O2 with alibuild or the quick rebuild approach. In some systems, including alidock, ninja install fails.
    • I suppose this could be achieved with a new flag complementary such as --build_only PACKAGE, complementary to the existing --disable PACKAGE.

Alternatively, another way to expedite the build process on CentOS with alibuild:

  1. Add option to disable fetching alicache.
    • I have been using alibuild on Ubuntu and Debian (with --force-unknown-architecture). On those systems an O2 rebuild with AliBuild was very quick, basically fetching locally that no package but O2 needs rebuild. When switching to CentOS I noticed that this fetching takes a long time, about 4 minutes just checking for updates at alicache – at least here in the Americas.

My two cents.
Option 1 is my favorite. :wink:

1 Like
2

Ciao,

regarding option 1., you should probably have a look at direnv which alibuild supports. You will be able to just cd into the folder and type make / ninja. I’d rather not add some extra command. You can even use it to hook to IDEs which support it https://aliceo2group.github.io/advanced/ides.html

–
Ciao,
Giulio

3

Hi Giulio,

Thanks for your suggestion on direnv, which is really nice.

However, this does not solve the issue, but no big deal. On my side alidock (or CentOS7) does not support cd to the O2 build followed by ninja/make. Perhaps an update on the instruction to have make/ninja building O2 will help.

I have been using Ubuntu (containers) since there I am able to get faster rebuilds. I’d rather work on CentOS which is the production environment, but I wonder if others would benefit from a standard way for getting faster rebuilds using alibuild as well. This would ensure uniform behavior in all supported systems and simplify documentation. For instance, on Ubuntu it is not necessary (or possible) to use alienv setenv GCC-Toolchain/latest -c for rebuilding; just a ninja install is good.

Rafael

4

I fully support Rafael’s feature request. In fact I had discussed a similar idea some time ago with Giulio (private communication):

Is there a way to tell alibuild to skip checking the dependencies and just build the last package specified, as in:

alibuild build O2 --defaults o2 --do-not-rebuild-dependencies

This should directly setup the environment of the dependencies and go straight to compilation of O2.

(Note that this is a bit different than building O2 in the build Dir without alibuild).

Indeed, rebuilding on CC7 seems to be extra slow and just the time rechecking the dependencies can be a major overhead. (I sometimes set the remote store to something non-existant to speed it up: --remote-store=foo’ ).

Should we make a JIRA item for this?

1 Like
5

@swenzel: I vaguely remember that at the time I asked you to try hacking it in the alibuild sourcecode by simply keeping the last package of the build list. Did you actually manage to do it that way? If yes we can add it properly, albeit I do think that “cd folder + ninja” is way better, especially if you manage to set up direnv to use one IDE. I think VSCode even supports building inside a container nowadays…

6

Yes. I asked Andrey Erokhin to take a look. The conclusion was that it is not so easy but we should look into it again.

In principle going into the build dir should be fine but there are other bugs such as a tiny changes in CMakeFiles causing complete rebuilds and instabilities in CMake so that I often have to go back to the alibuild level do to fresh builds. In this scenario especially it would be nice not building all the dependencies again.

In this regard, a high level solution on the basis of alibuild would be really nice to have.

7

I can try again, but I also remember something non-trivial happening…

8

That said, we should be able to rerun cmake with just “cmake .” in the correct folder. If that is not the case, that’s also something we should look at.

9

I fully agree that cmake . should work. I guess it mostly does (for O2 at least) as long as we do not have nasty dependencies on environment variables that are there within an aliBuild build but not outside. See e.g. the identified but not yet solved https://alice.its.cern.ch/jira/browse/O2-1044 with dependencies on G4INSTALL and ROOTSYS env. variables (while there’s no fix, there’s a workaround in that ticket).

Just to be clear, it’s ok to need some env. variables when doing the very first cmake configuration stage (for instance to define some cmake options using -D params), but from then on the build directory should be useable whatever the environment is. And that’s because if done correctly the first cmake should have populated the CMakeCache.txt file with all the required information.

And then there is this annoying observation that sometimes a ninja (or make or cmake --build . ) seems to rebuild a very large fraction of O2 for some (yet uncovered) reason. In a private conversation with Sandro I said I’ll tackle the issue but so far I was not able to find the root cause, so no fix there either.

1 Like