CCDB is down

@bvonhall The CCBD data base is currently down. Can you take a look at that? Thank you!

Dear Zhaozhong,

I am not in charge of the CCDB. Calling @grigoras to help with the matter.

Best regards,

@szhaozho Costin wrote to me and it seems that the QC data in the database amounts for 0.7TB. There is no more space and brought down the CCDB. So much data is not normal.

Could you make sure that you are not doing something wrong ? Could you also make sure that you are not updating more than once per minute the histograms ?

Hi Barth,

So now the number of histogram updating is correct. But the size of my histograms are pretty large. I will try to discuss with Ivan and figure out how to make them smaller. Thank you!

Ok, good to know.
Could you give me numbers ?
Number of tasks ? how many plots each ? which size is in average each plot ?

Hi Barth,

I will get back to you soon. Thank you!

Zhaozhong Shi

We have 1 task per minute. 62 plots each and the total size is 5.6 MB so on average ~100 kB each

This is not what I see here :
Each object is 37 MB.

OK I think that is the problem. I am making TH2D histograms with binning 502 x 1024 x 9. So maybe that will lead to some problem to the database. I will discuss with Ivan and see how to will make the histograms to be smaller. Thank you!

Thank you to you for checking this !

Hi Barth,
thank you for looking into this!

so we are now reducing of a factor 4 the histograms size for the time being and also reducing of a factor 5 the update rate. We will need in the next weeks to find a solution on how to display more than 32 k events with pulse activated.

In the workflow that we have now, the logic of looking for new files to be processed is coded inside the QC task (we discussed already in person that we have been creating a monster and we are aware of it) so for what I understand we are also updating the histograms when there are not “new files”. This is clear not the behavior that we want and Zhaozhong is looking into it. Do you know if there is a mechanism possible from inside a QC task to send a command that stops the updating?

Ciao and thank you,

Hi Ivan,
I can also adapt the clean up policy of the CCDB for the ITS tasks in order to remove faster the multiple versions of a given object. For example, instead of a grace period of 1 day during which versions are not touched, we could have just 1 hour, and then we keep 1 version per hour any ways.
Concerning the second point, if you update to the latest version of the QualityControl (v1.13 or master) there is a method “stopPublishing” in ObjectsManager. Maybe you can use that.

Hi Barth,
yes, at least for the time being it would be very useful to reduce to 1 h the grace period for the ITS at least we are sure to don’t saturate the CCDB. With the QC workflow that we are (should) be running now we are publishing 9 MB objects every 2-5 mins. We can review the situation when the workflow is close to final. Zhaozhong is also implementing the “stopPubshing” method now and we hope to have it in production by this afternoon.

Thank again,