Sending of empty DPL messages

Dear colleagues,

as far as we could check, it seems that a DPL device that sends two or more messages is dropping all of them whenever one is empty. In particular, we have a raw-to-digits workflow that generates two DPL messages, digits and orbit-info. Whenever the digits message is empty the orbit-info message is dropped.

Is this the expected behavior? If yes, is there an option allowing to send empty messages?

If you are interested, below you can find an explanation of why we need those two messages, and why we need the orbit-info to be sent regardless of the size of the digits messages…

Thanks a lot in advance!

Cheers,
Andrea


In the QC code for MCH we are currently implementing the computation of detector occupancies, which are generally defined as

number-of-digits / size-of-acquisition-window

The detailed math is a bit different in the case of triggered and continuous readout modes. In triggered mode, we open a 1 us time window for each trigger in which the detector signals are collected. Hence we can simply write

occupancy [MHz] = number-of-digits / number-of-triggers

On the other hand, in continuous mode we have to rely on the number of orbits in order to estimate the total length of the acquisition window, hence we have something like

occupancy = number-of-digits / number-of-orbits

Our typical QC chains for occupancy plots will probably look like this:

readout -> o2-mch-raw-to-digits-workflow -> QC
readout -> STFbuilder -> STFsender -> TFbuilder -> o2-mch-raw-to-digits-workflow -> QC

The second chain is the one I would expect to use during normal operation, while the fist one is a simplified workflow in which we process individual HB frames instead of full Time Frames.

The first issue we have is that the information about the number of processed orbits gets lost when going from raw data to digits. We currently solve this by introducing an additional DPL message at the output of the raw-to-digits-workflow, which provides the list of processed orbits for each CRU link.

This works fine when at least one digits is present in the output. However, when no digits are found and the corresponding DPL message is empty, we realized that the DPL output is completely dropped and the message with the list of orbits is also discarded.
This means we are not able to compute the occupancy when no digits are actually sent, which is not OK.

For the moment we “solved” this by injecting a dummy digit into the DPL message when it is empty, but we would like to check is there is a cleaner and more standard way of achieving the same…

Thanks a lot in advance!

Cheers,
Andrea

No, it’s not normal. Probably yet another place where the change of behavior which happened in FairMQ a while ago needs to be addressed. Please open a JIRA ticket and assign it to me.

Ok, I will do it. Would you need a minimal example of the issue?

yes please

After making a simple test program and running some checks, I realized that my initial conclusions were not correct. In fact, DPL itself is passing around empty messages correctly. However, the empty messages are skipped by QC. More precisely, if multiple inputs are connected to a QC tasks, and one of the inputs has a message of size 0, then the monitorData() method is not called.

@bvonhall is this expected, or is it a bug?

For the QC case it is less easy to make a minimal example, but I can try if needed.

@bvonhall as I said this changed in behavior in FairMQ at somepoint. Maybe you should double check all the places where you look for a null payload.

@eulisse Do you mean that the payload pointer of the DataRef is nullptr ? we do indeed skip that.

@pkonopka What do you think of changing line 253 of TaskRunner and accept passing empty payloads ? I don’t see any side effect, as we then simply pass the context to the user’s code.

Sure. I didn’t consider it a valid behaviour before, thus I checked the presence of message payloads. It has to be changed in two places, AFAIK:


https://alice.its.cern.ch/jira/browse/QC-401

If that still fails, you might want to check if Dispatcher forwards correctly messages without payloads. I use InputRecordWalker there - it seems that it accepts empty payloads, but I am not 100% sure.

@bvonhall @pkonopka I will try to test the PR as soon as possible, probably this evening or tomorrow, and let you know the result.

Thanks!

@bvonhall @pkonopka I confirm that PR QC-401 allows to receive empty DPL messages in QC, so it solves our issue with the calculation of detector occupancies.

However I cannot judge myself about possible bad side effects, particularly due to null pointers in input payloads. I guess user code will have to be adapted such that it checks that the payload pointer is valid before using it…

Thanks!

Cheers, Andrea

Thank you for the confirmation !
We should probably add something in our example code and in the doc.