Build fails with internal compiler error on CentOS 7

O²
#1

Dear experts,

I have a problem with O2 building failing on a CentOS 7 computer. All dependencies compile fine (and with aliBuild 1.7.2 the dependency checks are a bit faster, thanks!). But the actual compilation fails at 87% in the analysis code. I uploaded the logfile to my CERNbox.

I have been struggling with this build for a few days now, and tried different versions of the dev branch as well as the latest release (21.07, I think). They all crash around 90%, although I’m not sure if it is always the exact same error and I did not keep the logfiles.

Any help would be greatly appreciated. Thanks,

  • Tom.
#2

Here is the relevant section from the logfile (with a bit of extra context):

Scanning dependencies of target O2exe-analysis-cascadefinder
[ 87%] Building CXX object Analysis/Tasks/ALICE3/CMakeFiles/O2exe-analysis-alice3-trackselection.dir/alice3-trackselection.cxx.o
Scanning dependencies of target O2exe-analysis-lambdakzeroanalysis
[ 87%] Building CXX object Analysis/Tasks/PWGLF/CMakeFiles/O2exe-analysis-cascadebuilder.dir/cascadebuilder.cxx.o
[ 87%] Building CXX object Analysis/Tasks/PWGLF/CMakeFiles/O2exe-analysis-cascadefinder.dir/cascadefinder.cxx.o
[ 87%] Building CXX object Analysis/Tasks/PWGLF/CMakeFiles/O2exe-analysis-lambdakzeroanalysis.dir/lambdakzeroanalysis.cxx.o
Scanning dependencies of target O2exe-analysis-track-checks
[ 87%] Building CXX object Analysis/Tasks/PWGLF/CMakeFiles/O2exe-analysis-track-checks.dir/trackchecks.cxx.o
[ 87%] Built target O2exe-analysis-lambdakzerofinder
Scanning dependencies of target O2exe-analysis-nucleispectra-task-skim-analyser
[ 87%] Building CXX object Analysis/Tasks/SkimmingTutorials/CMakeFiles/O2exe-analysis-nucleispectra-task-skim-analyser.dir/spectraNucleiAnalyse
r.cxx.o
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-alice3-trackselection
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See https://gcc.gnu.org/bugs/ for instructions.
gmake[2]: *** [Analysis/Tasks/PWGLF/CMakeFiles/O2exe-analysis-cascadefinder.dir/build.make:82: Analysis/Tasks/PWGLF/CMakeFiles/O2exe-analysis-cascadefinder.dir/cascadefinder.cxx.o] Error 4
gmake[1]: *** [CMakeFiles/Makefile2:40373: Analysis/Tasks/PWGLF/CMakeFiles/O2exe-analysis-cascadefinder.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs…
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-tpcspectra-task-skim-reference
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-tpcspectra-task-skim-analyser
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-nucleispectra-task-skim-analyser
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-upc
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-lambdakzeroanalysis
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-nucleispectra-task-skim-provider
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-lambdakzerobuilder
[ 87%] Built target O2exe-analysis-alice3-trackselection
[ 87%] Built target O2exe-analysis-tpcspectra-task-skim-analyser
[ 87%] Built target O2exe-analysis-nucleispectra-task-skim-provider
[ 87%] Built target O2exe-analysis-tpcspectra-task-skim-reference
[ 87%] Built target O2exe-analysis-nucleispectra-task-skim-analyser
[ 87%] Built target O2exe-analysis-upc
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-nucleispectra-task-skim-reference
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-track-checks
[ 87%] Built target O2exe-analysis-nucleispectra-task-skim-reference
[ 87%] Built target O2exe-analysis-track-checks
[ 87%] Linking CXX executable …/…/…/stage/bin/o2-analysis-cascadebuilder
[ 87%] Built target O2exe-analysis-cascadebuilder
[ 87%] Built target O2exe-analysis-lambdakzeroanalysis
[ 87%] Built target O2exe-analysis-lambdakzerobuilder
gmake: *** [Makefile:182: all] Error 2

#3

Hi Tom,

Just in case you actually do not need the analysis part, note that you can go to the build directory, and change a bit how cmake is configured to disable analysis :

cmake -DBUILD_ANALYSIS=OFF .
cmake --build . 
cmake --install .
#4

Hi Tom,

this looks like you ran out of memory. You could also try to run aliBuild on less cores e.g. -j 1 or -j 2.
See here for the full discussion.

But @laphecet’s solution is probably the way to go.

Cheers,
Thomas

#5

Hello Thomas,

I saw the thread and tried to limit to 1 core, and that did not help either. We have 24 GB of memory, so I did not expect it to be the issue, and the compiler did not report a memory issue.

I am trying @laphecet 's suggestion now. Thanks, Laurent.

Cheers, Tom.

#6

For the record, I reduced the number of parallel compilation jobs for analysis. Could you try again?

#7

@eulisse do I understand correctly that you are limiting the number of parallel compilations for analysis using the JOB_POOL_COMPILE cmake option ? But then, that’s only relevant for Ninja builds, isn’t ? (seems Tom’s build was a make one)

#8

At the moment, I am running a build (using aliBuild and a patched alidist/o2.sh) following Laurent’s suggestion.
I am not sure how to load all the alienv dependencies for building O2, hence the patched alidist.

It takes quite some time for aliBuild to check all the dependencies - maybe because Cape Town is far from some servers at CERN?

I can try with fewer cores afterwards.

#9

I think we use ninja by default now, no?

#10

I’ve done some improvement in aliBuild 1.7.2 which should parallelise the check for dependencies. Could bump your version and see if there is any improvement?

#11

AFAIK we use ninja by default if it’s available on the machine. Or do we have it as a mandatory dep (that we build if not present) and I’ve missed that ?

#12

you are right… Time to change that?

#13

I can confirm that ninja is only used when it is found on the system, and on our one, yum install ninja did not find it.

#14

I have tried the compilation without the analysis code, and this did indeed work. For now, this should give us everything we need, but we will have to come back to this at some point.

#15

I have used aliBuild 1.7.2, and it still seemed quite slow, although it might have been faster than 1.7.1. But I think this is something for a different thread.