Dear QC experts,
For the MCH monitoring, with @aferrero we plotted the channel-by-channel occupancy of our readout electronics. In continuous readout mode, we defined the occupancy of a channel as a number of hits per orbit, and defined a “mean occupancy” for each detection element as the mean on all the channels.
By taking the number of hits and orbits seen in the last elapsed cycle (which implied keeping track of those numbers at the beginning and at the end of it), we end up with a mean occupancy in the last elapsed cycle.
This value is stored for each detection element in a TH1 (written at the end of each QC cycle), on which we did some trending to see how the values evolved at each cycle as the QC runs in continuous mode.
The only way we found for the moment is to use the hardcoded duration of a QC-cycle. For example, if each cycle is 1 sec long, we want to trigger the TrendingTask every 1 sec as well.
To do so, we keep the TrendingTask always triggered (“updateTrigger” = “always” in .json) and we set the time between trigger checks to 1 sec in the command line (o2-qc-run-postprocessing --config jsonfile.json --name ExampleTrend --period 1).
Other triggers like ones triggering at each end of run, or each time a histogram is updated in the QCDB don’t seem to be usable yet.
Is there any standard way to compute a mean value on the last cycle and do its trending, which would not require to keep filling arrays of old/new values nor matching by hand QC-cycle length and trigger/trigger check times?
Ideally, we may also want to compute the value on any timescale of our choosing, not just one cycle. Do you think this would be possible?
And finally, are there any subtleties we should be aware of about the way we do a TrendingTask (to make sure we stay compatible and in line with the rest of the QC)?
Let’s start from the end of your post.
The post-processing framework is designed to work with longer periods, definitely not on each second basis. The operations on database are relatively costly - one task asking the database to retrieve and store new objects each second might be fine, but a multitude of them would take a significant toll on it.
We should find a way so that you don’t have to perform so many I/O operations.
You should expect that in production all QC tasks will run with at least 60 seconds cycles, unless it is strongly justified to make them shorter. The same goes for post-processing.
Indeed, we are waiting for a proper API to check the run number. Checking if an object in QCDB has been updated should be possible to implement now (before, we also were missing an interface to check that), we just have to do this.
I might not understand you well here, but I am afraid you would have to keep track of time or actually use timestamps in data headers. Please keep in mind that O2 is a message passing system, with message queues which are asynchronous, so data will reach your QC task with some variable latency. Relying on the time when data arrives to you will produce less accurate results
Well, you could start another thread in your QC task which does it on any timescale you want. But please be wary of concurrency problems then.
Otherwise, you would have to compute all missing “microcycles” results at the end of cycle or each time you get new data.
To sum up, post-processing probably should not be used to trend quickly changing observables, unless you can retrieve many of them in one go. I would recommend you to generate such a trend directly in your QC Task.
I hope this is helpful and I understood you post well.
Thanks for the reply, and sorry if not everything was clear.
Regarding the duration of the cycles, it was chosen to be so short for the moment because the data input we use is quickly processed by the QC. As we needed at least multiple cycle to see what the trending was doing, we needed really short cycles… But yes, I imagine that in the end with “full data”, the cycle duration will be chosen so it is much more reasonable, thanks for the heads up !
What I meant by “Is there any standard way to compute a mean value on the last cycle and do its trending, which would not require to keep filling arrays of old/new values” was that:
In order to get the mean value of detector occupancy on the elapsed cycle, we had to keep an array with the number of hits and an array with the number of orbits at the beginning of the cycle. Also, arrays of hits and orbits at the end of the cycle (info we get via DPL where we send the digit with the orbit info, so everything is timestamped and is not impacted by the arrival time of data).
With those arrays we could get the number of hits and orbits to have been seen during the cycle and finally compute the occupancy. I feel that keeping all those arrays and updating them each cycle is quite a “naive” approach which could maybe be improved, and wondered if there was a standard way to do the occupancy calculation. (But it may be more of a coding issue rather than a QC-related one)
Anyway, thanks again for the answer,
Aah, OK, I understand now. Then I don’t have a good idea on how to improve it. Perhaps other detector teams are preparing something similar, they would be welcome to chime in in such case.