QC Workflow and Digitizer

szhaozho · February 27, 2019, 11:37pm

So in the QC workflow runBasic.cxx, we have a “Raw data producer”

DataProcessorSpec producer{
“producer”,
Inputs{},
Outputs{
{ “ITS”, “RAWDATA”, 0, Lifetime::Timeframe }
},
AlgorithmSpec{
(AlgorithmSpec::InitCallback) {
std::default_random_engine generator(11);
return (AlgorithmSpec::ProcessCallback) [generator](ProcessingContext& processingContext) mutable {
usleep(100000);
size_t length = generator() % 10000;
auto data = processingContext.outputs().make(Output{ “ITS”, “RAWDATA”, 0, Lifetime::Timeframe },
length);
for (auto&& item : data) {
item = static_cast(generator());
}
};
}
}
};

In principle, can I digitizer this ITS raw data by running the digitizer? If so, should I just set my input for my digitizer to be exactly the output of this producer? Thank you very much!

shahoian · February 28, 2019, 12:52am

Hi,
not sure what you call exactly “digitizer”, usually we call this way a device to convert MC hits to digits (detector response). For the ITS this is done by device O2/Steer/DigitizerWorkflow/src/ITSMFTDigitizerSpec.cxx which sends its output (a vector of digits for the whole simulated TF + vector with some control records to tag separate triggers or RO frames) to writer device managed by ITSMFTDigitWriterSpec.cxx, which writes this vector of digits to a tree.

Not sure this is a vector you need for the input, since you mention raw data.
We have ITS data decoders from MC ( O2/Detectors/ITSMFT/common/reconstruction/include/ITSMFTReconstruction/DigitPixelReader) and raw data (`O2/Detectors/ITSMFT/common/reconstruction/include/ITSMFTReconstruction/RawPixelReader`` which will be completely changed in a few days to read real CRU data rather than previously mocked format).

The DigitPixelReader is wrapped to O2/Detectors/ITSMFT/ITS/workflow/src/DigitReaderSpec.cxx which reads from the MC digits file written by ITSMFTDigitWriterSpec. Once the new RawPixelReader will be ready, it can be also added to the same reader device, but it will still read from file (binary raw data file, real dump from CRU or converted from MC).
So, what kind of input do you need?

bvonhall · February 28, 2019, 7:22am

Hi,

runBasic.cxx, in the QualityControl, is just a mock data flow. The ITS raw data is actually completely fake. You should ignore what is in the QC at this stage, and try to run your workflow in the DPL : MC -> Digit -> Histogramer
You had found a workflow in ITS folder in O2 that had the beginning of the workflow right ? What had to be done was to send out the result instead of storing it in a file and to plug it into your histogrammer.

Cheers,
Barth

szhaozho · February 28, 2019, 8:43am

Hi Ruben,

So basically now I have setup the workflow for digitizing and making the histograms for the MC hits. What I plan to do is to publish my plots to the GUI. So I guess Barth can help me better for this.

Sincerely,
Zhaozhong Shi

szhaozho · February 28, 2019, 8:45am

Hi Barth,

I have setup the workflow already and try to implement it to the QC. So if I implement the histogramer to my QC task, will the QC be able to use the output of my histogramer (root files) to the GUI? Thank you

bvonhall · February 28, 2019, 9:58am

Hi,

As I had told you, I think that the Histogrammer should be the QC task. There is not much sense in sending a histogram to a task that will just forward it to the rest of the QC chain. At a later stage I plan to provide an API for any Device to send prepared histograms to the QC database, but this does not exist yet.

By the way, do you still use files to transfer data from one process to the next ? if so, I would strongly advise to use the DPL to send and receive the data.

I presume that the next step is to move the workflow to the QC repository and make it work as is. Then, move the code of the histogramer into a QC task and plug the Data SAmpling and the QC task into your workflow.

Cheers,
Barth

icali · February 28, 2019, 12:00pm

Hi,
yes, indeed we should move the histogrammer code to the QC task and Zhaozhong is doing this at the moment. The histogrammer was indeed supposed to be only for testing. The intermediate file dependency is as well being removed as it should not be there.

However, we still don’t know how to use the DPL. Probably I’m not looking in the right plance but apart some examples I didn’t find the ITS code. At present we are running the digitizer-workflow on MC data.
Should we create a kind of custom implementation ourself? In principle we can add the first steps in the runBasic.cxx example but not sure it is the way we want to go.

Ciao and that you,
Ivan

bvonhall · February 28, 2019, 12:44pm

Hi Ivan,

You are correct, the first step would be to identify what ITS workflow needs to be ran. I thought that it was the one you were using. For this you should probably see with the ITS people. Maybe Sandro @swenzel might also know ? Or Ruben ?

I would advise to copy the ITS workflow to the QC (and leave the runBasic.cxx) and try to execute it from there. Then you can plug the Data Sampling and the QC task to it as shown here : https://github.com/AliceO2Group/AliceO2/tree/dev/Framework/Core#usage

Cheers,
Barth

shahoian · February 28, 2019, 2:15pm

Hi,
To avoid going in the wrong direction, I think we better re-discuss what exactly we want to do (at ITS WP1/2 meeting next Monday?). In real data processing, we will be decoding ITS data chip by chip and clusterize them on the fly, so, in principle, we will not exchange the digits.
The intermediate data format to keep digits in the clusterizer is not the same what the digitizer (of the digitizer-workflow) or the digitReader (of the its-reco-workflow) provide.

If in the real production the ITS QC will run over the raw data rather than on the output of some upstream device, then you will need to adapt your loop to per chip input from the PixelReader (base class for DigitPixelReader for MC and RawPixelReader for the raw data).
Then you better start with RecoWorkflow -> DigitReaderSpec.cxx -> ClustererSpec.cxx of O2/Detectors/ITSMFT/ITS/workflow/src and modify ClustererSpec for your purposes.

icali · February 28, 2019, 3:27pm

Hi Ruben,
yes, it is a good idea to discuss during the WP1/2 on Monday or any other time convenient for you.
Thank you for the advice and we can adjust the code accordingly. However, we still need to understand how we want to implement the workflow in the QC like environment.
Cheers,
Ivan

szhaozho · February 28, 2019, 3:37pm

Hi Ruben,

I have made a module to read the chip and pixel data and make histograms with the input file itsdigits.root. Is it possible that I can directly read the digit data without going through the tree? I mean, I set the input of my histogrammer to be

				InputSpec{ "digits", "ITS", "DIGITS", 0, Lifetime::Timeframe },

So in principle, can I use DigitPixelReader and ChipPixelData to read digits information from this input. Thank you very much!

Sincerely,
Zhaozhong Shi

shahoian · February 28, 2019, 3:48pm

Hi Zhaozhong,

Yes, this is what I meant suggesting to use the RecoWorkflow -> DigitReaderSpec.cxx -> ClustererSpec.cxx flow: you can add a clone of ClustererSpec, which gets exactly this input and parses it using “DigitPixelReader”.

szhaozho · March 2, 2019, 2:06pm

Hi Barth,

I try to include my workflow to the QC. But when I compile, I found the following problem:

/data/zhaozhong/alice/sw/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-7/bin/…/lib/gcc/x86_64-unknown-linux-gnu/7.3.0/…/…/…/…/x86_64-unknown-linux-gnu/bin/ld: CMakeFiles/qcRunPrint.dir/src/runPrint.cxx.o: undefined reference to symbol ‘_ZN6TChainD1Ev’

/data/zhaozhong/alice/sw/slc7_x86-64/ROOT/v6-14-04-24/lib/libTree.so.6.14: error adding symbols: DSO missing from command line

Do you know why? Is it because of the incompatibility between O2 and QC? Thank you very much!

Sincerely,
Zhaozhong Shi

bvonhall · March 4, 2019, 4:11pm

Hi,
I would guess that it is a problem with the CMake scripts. Some linking is missing when building runPrint.
Do you have the code somewhere ?
Cheers,
Barth

szhaozho · March 4, 2019, 6:55pm

Hi Barth,

It looks like the DigitPixelReader.h and GeometryTGeo.h, both of which from O2, are not loadable when I try to include it in my code. Basically, when I compile my code, it will return undefined reference error to me. Thank you very much!

Sincerely,
Zhaozhong Shi

icali · March 5, 2019, 11:05am

Hi Barth,
thank you for the answer and yes, definitively it is just an include issue. However, during the WP1/2 meeting a couple of subjects come up and they are the following:

O2 and QC are on different repositories. Most probably going ahead with the development the classes of O2 will be needed in QC as well. What is the mechanism foreseen up now to have the cross reference? Do you plan to expose the O2 libraries to QC or is it going to be only via a change of the CMAke?
In the ITS workflow it is at the moment not possible to have a workflow that doesn’t use an intermediate file. As pointed out by Ruben in an earlier message, the only option that we have is to “manually” generate a itsdigis.root and on top add a “DPL reader” that will provide data to the QC. I assume that we would simply need to add the sample functions in the reader and leave the QCTask unchanged but in the previous discussions, it appears that this method is not feasible/acceptable in the QC implementation. Can you advice?

@szhaozho please share in this thread your repository including your latest commits.

Ciao and thank you,
Ivan

szhaozho · March 5, 2019, 1:31pm

Hi Barth,

My latest commit of the SkeletonTask.cxx code is shown as follows:

(https://github.com/MYOMAO/ALICE-ITS-DCS/blob/master/QualityControl/Modules/Skeleton/src/SkeletonTask.cxx)

Can you take a quick look at that? Thank you very much!

Sincerely,
Zhaozhong Shi

bvonhall · March 5, 2019, 2:59pm

Just to be sure that we are all on the same page, the QC will probably run on raw data as well as downstream data elements, i.e. tracks or intermediary data. Anything you want to QA, should be sent to QC, in the synchronous or asynchronous processing.

Good question. The O2 packages are split in a few repositories : AliceO2, QualityControl, Configuration, DataDistribution, … They depend on each other. The dependencies are expressed first in the alibuild recipes (QualityControl depends on O2). They are then specified again in CMake for the build system to know where to look for libraries and headers. For example, we use some O2 libraries within the QC.

In your specific case, we need to make sure that O2 exposes the needed libraries and headers and that QC is able to pick them up.

I am not sure to see the point of using the data sampling and the QC tasks on histograms. If you already have histograms, why do you want to sample them and process them with a task ? I have the feeling there is something that I am missing

I would propose to meet shortly to discuss it. It will be easier than here.

szhaozho · March 11, 2019, 3:22pm

Hi Barth,

It looks like my DIGIReader works but I am not able to sample the digit. I guess there may be some questions with my config file. Here are the output of my DigiReader:

Outputs{

  OutputSpec{ "ITS", "DIGITS", 0, Lifetime::Timeframe },

  OutputSpec{ "ITS", "DIGITSMCTR", 0, Lifetime::Timeframe },

  OutputSpec{ "ITS", "ITSDigitROF", 0, Lifetime::Timeframe },

  OutputSpec{ "ITS", "ITSDigitMC2ROF", 0, Lifetime::Timeframe } },

Here is my data sampling policy:

“dataSamplingPolicies”: [

{

  "id": "raw",

  "active": "true",

  "machines": [],

  "dataHeaders": [

    {

      "binding": "digits",

      "dataOrigin": "ITS",

      "dataDescription": "DIGITS"

    }

  ],

  "subSpec": "0",

  "samplingConditions": [

    {

      "condition": "random",

      "fraction": "1.0",

      "seed": "1234"

    }

  ],

  "blocking": "false"

}

]

Do you know what goes wrong? Why can’t I sampling any output from the Digitreader? Thank you!

bvonhall · March 14, 2019, 8:05am

Hi,
Could you upload or send me the whole file ?
Thank you,
Barth