O2 build on a test FLP with CentOS 7 fails at Python@v3.6.10

therman · February 19, 2021, 9:56pm

Hello,

I am trying to install O2 on a test FLP setup with CentOS 7 and it fails at Python@v3.6.10.

==> Building Python@v3.6.10
==> Python is being built (use --debug for full output): failed
ERROR: Error while executing /home/flp/alice/sw/SPECS/slc7_x86-64/Python/v3.6.10-1/build.sh on `alimftcom3'.
ERROR: Log can be found in /home/flp/alice/sw/BUILD/Python-latest/log
ERROR: Please upload it to CERNBox/Dropbox if you intend to request support.
ERROR: Build directory is /home/flp/alice/sw/BUILD/Python-latest/Python.

The log is here: CERNBox

I have updated version of both alidist and O2.

The aliDoctor command claims it will pickup Python from the system:

SUCCESS: Required package openmp will be picked up from the system.
WARNING: Package libuv cannot be picked up from the system and will be built by aliBuild.
WARNING: This is due to the fact the following script fails:
WARNING: 
WARNING: brew() { true; }; printf "#include <uv.h>" | c++ -I$(brew --prefix libuv)/include -xc++ - -c -o /dev/null 2>&1
WARNING: 
WARNING: with the following output:
WARNING: 
WARNING: libuv: <stdin>:1:10: fatal error: uv.h: No such file or directory
WARNING: libuv: compilation terminated.
WARNING: libuv: 
WARNING: 
WARNING: Package lz4 cannot be picked up from the system and will be built by aliBuild.
WARNING: This is due to the fact the following script fails:
WARNING: 
WARNING: brew() { true; }; printf "#include <lz4.h>\n" | cc -xc -I$(brew --prefix lz4)/include - -c -M 2>&1
WARNING: 
WARNING: with the following output:
WARNING: 
WARNING: lz4: <stdin>:1:10: fatal error: lz4.h: No such file or directory
WARNING: lz4: compilation terminated.
WARNING: lz4: 
WARNING: 
SUCCESS: Package zlib will be picked up from the system.
WARNING: Package flatbuffers cannot be picked up from the system and will be built by aliBuild.
WARNING: This is due to the fact the following script fails:
WARNING: 
WARNING: brew() { true; }; which flatc && printf "#include \"flatbuffers/flatbuffers.h\"\nint main(){}" | c++ -I$(brew --prefix flatbuffers)/include -xc++ -std=c++11 - -o /dev/null
WARNING: 
WARNING: with the following output:
WARNING: 
WARNING: flatbuffers: which: no flatc in (/opt/rh/devtoolset-7/root/usr/bin:/usr/share/Modules/bin:/opt/alisw/el7/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/home/flp/.local/bin:/home/flp/bin)
WARNING: flatbuffers: 
WARNING: 
WARNING: Package CMake cannot be picked up from the system and will be built by aliBuild.
WARNING: This is due to the fact the following script fails:
WARNING: 
WARNING: brew() { true; }; verge() { [[  "$1" = "`echo -e "$1\n$2" | sort -V | head -n1`" ]]; }
WARNING: type cmake && verge 3.19.2 `cmake --version | sed -e 's/.* //' | cut -d. -f1,2,3`
WARNING: 
WARNING: with the following output:
WARNING: 
WARNING: CMake: /bin/bash: line 1: type: cmake: not found
WARNING: CMake: 
WARNING: 
WARNING: Package yaml-cpp cannot be picked up from the system and will be built by aliBuild.
WARNING: This is due to the fact the following script fails:
WARNING: 
WARNING: brew() { true; }; pkg-config --atleast-version=0.6.2 yaml-cpp && printf "#include \"yaml-cpp/yaml.h\"\n" | c++ -std=c++17 -I`brew --prefix yaml-cpp`/include -I`brew --prefix boost`/include -xc++ - -c -o /dev/null
WARNING: 
WARNING: with the following output:
WARNING: 
WARNING: yaml-cpp: 
WARNING: 
SUCCESS: Package GCC-Toolchain will be picked up from the system.
WARNING: Package ZeroMQ cannot be picked up from the system and will be built by aliBuild.
WARNING: This is due to the fact the following script fails:
WARNING: 
WARNING: brew() { true; }; printf "#include <zmq.h>\n#if(ZMQ_VERSION < 40105)\n#error \"zmq version >= 4.1.5 needed\"\n#endif\n int main(){}" | c++ -I$(brew --prefix zeromq)/include -xc++ - -o /dev/null 2>&1
WARNING: 
WARNING: with the following output:
WARNING: 
WARNING: ZeroMQ: <stdin>:1:10: fatal error: zmq.h: No such file or directory
WARNING: ZeroMQ: compilation terminated.
WARNING: ZeroMQ: 
WARNING: 
SUCCESS: Package OpenSSL will be picked up from the system.
SUCCESS: Package GLFW will be picked up from the system.
SUCCESS: Package FreeType will be picked up from the system.
SUCCESS: Required package system-curl will be picked up from the system.
SUCCESS: Package autotools will be picked up from the system.
SUCCESS: Package libpng will be picked up from the system.
SUCCESS: Package lzma will be picked up from the system.
SUCCESS: Package bz2 will be picked up from the system.
SUCCESS: Package Python will be picked up from the system.
SUCCESS: Required package make will be picked up from the system.
WARNING: Package GSL cannot be picked up from the system and will be built by aliBuild.
WARNING: This is due to the fact the following script fails:
WARNING: 
WARNING: brew() { true; }; printf "#include \"gsl/gsl_version.h\"\n#define GSL_V GSL_MAJOR_VERSION * 100 + GSL_MINOR_VERSION\n# if (GSL_V < 116)\n#error \"Cannot use system's gsl. Notice we only support versions from 1.16 (included)\"\n#endif\nint main(){}" | c++  -I$(brew --prefix gsl)/include -xc++ - -o /dev/null
WARNING: 
WARNING: with the following output:
WARNING: 
WARNING: GSL: <stdin>:1:10: fatal error: gsl/gsl_version.h: No such file or directory
WARNING: GSL: compilation terminated.
WARNING: GSL: 
WARNING: 
SUCCESS: Required package opengl will be picked up from the system.
SUCCESS: Required package Xdevel will be picked up from the system.
SUCCESS: Package libxml2 will be picked up from the system.
WARNING: Package ofi cannot be picked up from the system and will be built by aliBuild.
WARNING: This is due to the fact the following script fails:
WARNING: 
WARNING: brew() { true; }; pkg-config --atleast-version=1.6.0 libfabric 2>&1 && printf "#include \"rdma/fabric.h\"\nint main(){}" | c++ -xc - -o /dev/null
WARNING: 
WARNING: with the following output:
WARNING: 
WARNING: ofi: 
WARNING: 

==> The following packages will be built by aliBuild because
     usage of a system version of it is not allowed or supported, by policy:
    
    - alibuild-recipe-tools 
    - ApMon-CPP 
    - Python-modules-list 
    - capstone 
    - DebugGUI 
    - KFParticle 
    - arrow 
    - boost 
    - fastjet 
    - generators 
    - MCStepLogger 
    - cgal 
    - Alice-GRID-Utils 
    - DDS 
    - FairMQ 
    - fmt 
    - libwebsockets 
    - Monitoring 
    - ROOT 
    - asiofi 
    - GEANT4 
    - O2 
    - libjalienO2 
    - FairLogger 
    - O2-customization 
    - xjalienfs 
    - RapidJSON 
    - MPFR 
    - Clang 
    - JAliEn-ROOT 
    - UUID 
    - lhapdf 
    - ms_gsl 
    - double-conversion 
    - vgm 
    - curl 
    - VecGeom 
    - googletest 
    - AEGIS 
    - hijing 
    - re2 
    - AliEn-CAs 
    - Common-O2 
    - googlebenchmark 
    - simulation 
    - libtirpc 
    - XRootD 
    - Ppconsul 
    - defaults-release 
    - Python-modules 
    - FairRoot 
    - pythia 
    - Vc 
    - json-c 
    - HepMC3 
    - protobuf 
    - pythia6 
    - Configuration 
    - GEANT3 
    - AliEn-Runtime 
    - GEANT4_VMC 
    - GMP 
    - libInfoLogger

==> The following packages will be picked up from the system:
    
    - GLFW
    - autotools
    - Python
    - OpenSSL
    - zlib
    - lzma
    - FreeType
    - bz2
    - GCC-Toolchain
    - libxml2
    - libpng
    
    If this is not you want, you have to uninstall / unload them.

==> The following packages will be built by aliBuild because they couldn't be picked up from the system:
    
    - GSL
    - ofi
    - flatbuffers
    - libuv
    - ZeroMQ
    - CMake
    - lz4
    - yaml-cpp
    
    This is not a real issue, but it might take longer the first time you invoke aliBuild.
    Look at the error messages above to get hints on what packages you need to install separately.

What is it that I am missing?

Thanks.

Cheers,
Tomáš

eulisse · February 19, 2021, 11:09pm

What are you trying to do? If you need to recompile only a single package you might be better off doing:

https://aliceo2group.github.io/quickstart/build.html#developing-o2-on-the-flp--experimental

therman · February 19, 2021, 11:32pm

These are our first tests with QC on a test FLP so I apologize in advance if my idea of how things work is not correct.

The idea is to run on the test FLP the workflow of reading raw data from a file (once we move to the real FLP read directly from the MFT ), run decoding from raw data to digits and run the MFT Digit QC Task. If necessary we can start by making the steps separately but in the end we will need it to be a continuos workflow.

I am not sure what is the best way to set this up but FLP Suite does not contain decoding so we need to have O2 for that. Moreover, the QC in the installed version of the FLP Suite does not contain the Digit QC Task we want. Therefore, we need to compile our own version of QC and plug it in the FLP Suite.

What would be the optimal way to do this kind of testing?

therman · February 23, 2021, 12:55pm

I tried to delete everything, update and install from scratch but I am still getting the same error. AliDoctor does not return any problem:

11:19 [flp@alimftcom3] ~/alice> aliDoctor O2 --defaults o2
    SUCCESS: Required package openmp will be picked up from the system.
    SUCCESS: Package libuv will be picked up from the system.
    SUCCESS: Package lz4 will be picked up from the system.
    SUCCESS: Package zlib will be picked up from the system.
    WARNING: Package flatbuffers cannot be picked up from the system and will be built by aliBuild.
    WARNING: This is due to the fact the following script fails:
    WARNING: 
    WARNING: brew() { true; }; which flatc && printf "#include \"flatbuffers/flatbuffers.h\"\nint main(){}" | c++ -I$(brew --prefix flatbuffers)/include -xc++ -std=c++11 - -o /dev/null
    WARNING: 
    WARNING: with the following output:
    WARNING: 
    WARNING: flatbuffers: which: no flatc in (/opt/rh/devtoolset-7/root/usr/bin:/usr/share/Modules/bin:/opt/alisw/el7/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/home/flp/.local/bin:/home/flp/bin)
    WARNING: flatbuffers: 
    WARNING: 
    WARNING: Package CMake cannot be picked up from the system and will be built by aliBuild.
    WARNING: This is due to the fact the following script fails:
    WARNING: 
    WARNING: brew() { true; }; verge() { [[  "$1" = "`echo -e "$1\n$2" | sort -V | head -n1`" ]]; }
    WARNING: type cmake && verge 3.19.2 `cmake --version | sed -e 's/.* //' | cut -d. -f1,2,3`
    WARNING: 
    WARNING: with the following output:
    WARNING: 
    WARNING: CMake: /bin/bash: line 1: type: cmake: not found
    WARNING: CMake: 
    WARNING: 
    WARNING: Package yaml-cpp cannot be picked up from the system and will be built by aliBuild.
    WARNING: This is due to the fact the following script fails:
    WARNING: 
    WARNING: brew() { true; }; pkg-config --atleast-version=0.6.2 yaml-cpp && printf "#include \"yaml-cpp/yaml.h\"\n" | c++ -std=c++17 -I`brew --prefix yaml-cpp`/include -I`brew --prefix boost`/include -xc++ - -c -o /dev/null
    WARNING: 
    WARNING: with the following output:
    WARNING: 
    WARNING: yaml-cpp: 
    WARNING: 
    SUCCESS: Package GCC-Toolchain will be picked up from the system.
    WARNING: Package ZeroMQ cannot be picked up from the system and will be built by aliBuild.
    WARNING: This is due to the fact the following script fails:
    WARNING: 
    WARNING: brew() { true; }; printf "#include <zmq.h>\n#if(ZMQ_VERSION < 40105)\n#error \"zmq version >= 4.1.5 needed\"\n#endif\n int main(){}" | c++ -I$(brew --prefix zeromq)/include -xc++ - -o /dev/null 2>&1
    WARNING: 
    WARNING: with the following output:
    WARNING: 
    WARNING: ZeroMQ: <stdin>:1:10: fatal error: zmq.h: No such file or directory
    WARNING: ZeroMQ: compilation terminated.
    WARNING: ZeroMQ: 
    WARNING: 
    SUCCESS: Package OpenSSL will be picked up from the system.
    SUCCESS: Package GLFW will be picked up from the system.
    SUCCESS: Package FreeType will be picked up from the system.
    SUCCESS: Required package system-curl will be picked up from the system.
    SUCCESS: Package autotools will be picked up from the system.
    SUCCESS: Package libpng will be picked up from the system.
    SUCCESS: Package lzma will be picked up from the system.
    SUCCESS: Package bz2 will be picked up from the system.
    SUCCESS: Package Python will be picked up from the system.
    SUCCESS: Required package make will be picked up from the system.
    WARNING: Package GSL cannot be picked up from the system and will be built by aliBuild.
    WARNING: This is due to the fact the following script fails:
    WARNING: 
    WARNING: brew() { true; }; printf "#include \"gsl/gsl_version.h\"\n#define GSL_V GSL_MAJOR_VERSION * 100 + GSL_MINOR_VERSION\n# if (GSL_V < 116)\n#error \"Cannot use system's gsl. Notice we only support versions from 1.16 (included)\"\n#endif\nint main(){}" | c++  -I$(brew --prefix gsl)/include -xc++ - -o /dev/null
    WARNING: 
    WARNING: with the following output:
    WARNING: 
    WARNING: GSL: <stdin>:1:10: fatal error: gsl/gsl_version.h: No such file or directory
    WARNING: GSL: compilation terminated.
    WARNING: GSL: 
    WARNING: 
    SUCCESS: Required package opengl will be picked up from the system.
    SUCCESS: Required package Xdevel will be picked up from the system.
    SUCCESS: Package libxml2 will be picked up from the system.
    SUCCESS: Package ofi will be picked up from the system.

==> The following packages will be built by aliBuild because
     usage of a system version of it is not allowed or supported, by policy:
    
    - fmt 
    - Common-O2 
    - O2 
    - simulation 
    - Alice-GRID-Utils 
    - AliEn-CAs 
    - libInfoLogger 
    - FairMQ 
    - Monitoring 
    - pythia 
    - MPFR 
    - UUID 
    - protobuf 
    - curl 
    - googlebenchmark 
    - FairRoot 
    - GEANT4 
    - vgm 
    - arrow 
    - GEANT3 
    - MCStepLogger 
    - cgal 
    - re2 
    - AEGIS 
    - HepMC3 
    - VecGeom 
    - DDS 
    - FairLogger 
    - Vc 
    - XRootD 
    - libtirpc 
    - RapidJSON 
    - boost 
    - googletest 
    - double-conversion 
    - GMP 
    - AliEn-Runtime 
    - GEANT4_VMC 
    - ROOT 
    - DebugGUI 
    - generators 
    - asiofi 
    - pythia6 
    - Configuration 
    - ApMon-CPP 
    - libjalienO2 
    - Python-modules-list 
    - libwebsockets 
    - alibuild-recipe-tools 
    - Clang 
    - json-c 
    - KFParticle 
    - ms_gsl 
    - JAliEn-ROOT 
    - fastjet 
    - xjalienfs 
    - Ppconsul 
    - O2-customization 
    - lhapdf 
    - defaults-release 
    - Python-modules 
    - capstone 
    - hijing

==> The following packages will be picked up from the system:
    
    - ofi
    - zlib
    - libpng
    - bz2
    - lzma
    - Python
    - GCC-Toolchain
    - OpenSSL
    - autotools
    - lz4
    - libuv
    - FreeType
    - GLFW
    - libxml2
    
    If this is not you want, you have to uninstall / unload them.

==> The following packages will be built by aliBuild because they couldn't be picked up from the system:
    
    - GSL
    - CMake
    - ZeroMQ
    - flatbuffers
    - yaml-cpp
    
    This is not a real issue, but it might take longer the first time you invoke aliBuild.
    Look at the error messages above to get hints on what packages you need to install separately.

And the log file still crashes at Python without pointing to a specific problem: CERNBox

bvonhall · February 25, 2021, 9:50am

Hi Tomas,

Do I understand correctly that you are building O2 with aliBuild ?
The software was not installed with RPMs, right ?

If so, which instructions are you following ?

If you have actually installed the machine with the RPMs of the FLP Suite, then it becomes a bit more tricky to mix local development code with the RPMs. Yet, we have explored several options: QualityControl/Advanced.md at master · AliceO2Group/QualityControl · GitHub
The real solution, option 3, is being tested: https://alice.its.cern.ch/jira/browse/O2-1896

Cheers,

therman · February 25, 2021, 10:03am

Hi Barth,

Yes, I was following the instructions for installation of O2 via alibuild in: 🛠 Build the packages · ALICE Analysis Tutorial and I wanted to follow up with instalation of QC via alibuild in: QualityControl/QuickStart.md at master · AliceO2Group/QualityControl · GitHub

Cheers,
Tomáš

bvonhall · February 25, 2021, 10:10am

Ok, thank you for confirming this. What puzzles me is that it should download the precompiled packages and not build them.

@eulisse Any idea why it is not downloading the precompiled binaries ?

therman · March 18, 2021, 9:00pm

@eulisse I tried to build again today with updated versions of alidist, O2.

I follow the guide here: Prerequisites for CentOS 7 · ALICE Analysis Tutorial

And installation instructions here: 🛠 Build the packages · ALICE Analysis Tutorial

Installing via:

aliBuild build O2 --defaults o2

But I still get the same error: CERNBox

Any ideas what is wrong?