Thanks… Looks like the report reports only the first command of a rule and the 8% spent in cd is really spent doing the rootmap rule (which probably starts by cd-ing into some temporary dir.
Can you run once more with:
–profile=report.trace
?
output of :
shake --no-build --profile=report.trace
available at :
CERNBox
Concerning the cd commands, yes, those are most probably the rootmaps (and a pcm copy afterwards as well ?)
e.g.
❯ ninja -t commands libCCDB.dylib | head -1
cd /Users/laurent/tmp/build/O2-shake/CCDB &&
DYLD_LIBRARY_PATH=/Users/laurent/alice/sw/osx_x86-64/ROOT/latest-clion-o2/lib: ROOTSYS=/Users/laurent/alice/sw/osx_x86-64/ROOT/latest-clion-o2
/Users/laurent/alice/sw/osx_x86-64/ROOT/latest-clion-o2/bin/rootcint -f /Users/laurent/tmp/build/O2-shake/CCDB/G__CCDBDict.cxx
-inlineInputHeader -rmf /Users/laurent/tmp/build/O2-shake/lib/libCCDB.rootmap
-rml CCDB.dylib -c -I/Users/laurent/alice/sw/osx_x86-64/FairRoot/latest-clion-o2/include -I/Users/laurent/alice/sw/osx_x86-64/FairRoot/latest-clion-o2/include/fairmq -I/Users/laurent/alice/sw/osx_x86-64/ROOT/latest-clion-o2/include -I/Users/laurent/alice/o2-dev/O2/CCDB -I/Users/laurent/alice/o2-dev/O2/CCDB/include -I/Users/laurent/alice/o2-dev/O2/CCDB/src -I/Users/laurent/alice/sw/osx_x86-64/boost/latest-clion-o2/include -I/Users/laurent/alice/sw/osx_x86-64/protobuf/latest-clion-o2/include include/CCDB/Backend.h include/CCDB/BackendOCDB.h include/CCDB/BackendRiak.h include/CCDB/Condition.h include/CCDB/ConditionId.h include/CCDB/ConditionMetaData.h include/CCDB/FileStorage.h include/CCDB/GridStorage.h include/CCDB/IdPath.h include/CCDB/IdRunRange.h include/CCDB/LocalStorage.h include/CCDB/Manager.h include/CCDB/ObjectHandler.h include/CCDB/Storage.h include/CCDB/XmlHandler.h test/TestClass.h /Users/laurent/alice/o2-dev/O2/CCDB/src/CCDBLinkDef.h &&
/usr/local/Cellar/cmake/3.11.1/bin/cmake -E copy_if_different /Users/laurent/tmp/build/O2-shake/CCDB/G__CCDBDict_rdict.pcm /Users/laurent/tmp/build/O2-shake/lib/G__CCDBDict_rdict.pcm
For the record, I did some further investigation and it looks like a dependency on boost.fusion is creeping in in a few places, generating huge depends.make files… Not sure where it comes from yet, but it’s probably a good idea to set a policy to avoid any boost related matter to creep into the header.
Could you see the same issue also with ninja-build, or is ninja smarter here?
I think ninja does not need that and it’s were it actually simplifies over make. That’s why the incremental builds are faster when using it (of course I could be wrong). However, even if we switch to ninja, that will still be slow in compilation if we have some header which accidentally brings in boost.fusion, as it will have to be parsed for each inclusion.
Well, I guess Ninja does need the same kind of information but is probably a bit smarter in storing it.
Ninja has (as far as I can tell) its include dependencies in a single (binary) .ninja_deps (that can be inspected using the ninja -t deps sub-command), while we get 227 depends.make files…
And the .ninja_deps file is 4.1 MB for the build above while some single depends.make are bigger than that :
find . -name depend.make -exec ls -s {} \; | sort -n
...
4192 ./Framework/Utils/CMakeFiles/test_DPLOutputTest.dir/depend.make
4200 ./Utilities/aliceHLTwrapper/CMakeFiles/aliceHLTwrapper.dir/depend.make
4208 ./Detectors/TPC/calibration/CMakeFiles/TPCCalibration.dir/depend.make
4224 ./Detectors/TPC/base/CMakeFiles/TPCBase.dir/depend.make
5992 ./Framework/Utils/CMakeFiles/test_DPLBroadcasterMerger.dir/depend.make
6152 ./CCDB/CMakeFiles/CCDB.dir/depend.make
6152 ./Framework/Utils/CMakeFiles/DPLUtils.dir/depend.make
8200 ./Detectors/TPC/workflow/CMakeFiles/tpc-reco-workflow.dir/depend.make
10304 ./Utilities/DataFlow/CMakeFiles/DataFlow.dir/depend.make
13512 ./Detectors/MUON/MCH/Mapping/Impl3/CMakeFiles/MCHMappingImpl3.dir/depend.make
22552 ./Framework/Core/CMakeFiles/Framework.dir/depend.make
(ls -s giving sizes in 512-bytes block)
But regardless of the build system, I agree with Giulio that we should anyway be very careful about our dependencies…
So the problem is that a FairMQDevice derives from a FairMQStateMachine which itself derives from a boost::state_machine which basically requires the whole boost to be parsed. Since this happens in the headers it means that anything including FairMQDevice.h will take forever to compile. Moreover, given FairMQDevice is what we use to send messages, anything which sends messages will bring in the whole boost in. I think this is a design issue with FairMQ and they should refactor things so that the state machine internal details are not exposed to the world. The reason why the framework is so slow to compile is merely the fact that we made very easy for people to write devices and send messages so we are simply exposing a scalability issue with the current FairMQDevice.h header.
I will try to follow this up with the FairMQ people.