CCDB is down

szhaozho · May 23, 2019, 12:16pm

@bvonhall The CCBD data base is currently down. Can you take a look at that? Thank you!

bvonhall · May 23, 2019, 12:43pm

Dear Zhaozhong,

I am not in charge of the CCDB. Calling @grigoras to help with the matter.

Best regards,
Barth

bvonhall · May 23, 2019, 12:48pm

@szhaozho Costin wrote to me and it seems that the QC data in the database amounts for 0.7TB. There is no more space and brought down the CCDB. So much data is not normal.

Could you make sure that you are not doing something wrong ? Could you also make sure that you are not updating more than once per minute the histograms ?

szhaozho · May 23, 2019, 1:12pm

Hi Barth,

So now the number of histogram updating is correct. But the size of my histograms are pretty large. I will try to discuss with Ivan and figure out how to make them smaller. Thank you!

bvonhall · May 23, 2019, 1:16pm

Ok, good to know.
Could you give me numbers ?
Number of tasks ? how many plots each ? which size is in average each plot ?
Cheers,
Barth

szhaozho · May 23, 2019, 1:27pm

Hi Barth,

I will get back to you soon. Thank you!

Sincerely,
Zhaozhong Shi

szhaozho · May 23, 2019, 2:57pm

We have 1 task per minute. 62 plots each and the total size is 5.6 MB so on average ~100 kB each

bvonhall · May 23, 2019, 2:59pm

This is not what I see here : http://ccdb-test.cern.ch:8080/browse/ITSQcTaskTEST?report=true
Each object is 37 MB.

szhaozho · May 23, 2019, 3:05pm

OK I think that is the problem. I am making TH2D histograms with binning 502 x 1024 x 9. So maybe that will lead to some problem to the database. I will discuss with Ivan and see how to will make the histograms to be smaller. Thank you!

bvonhall · May 23, 2019, 3:06pm

Thank you to you for checking this !

icali · May 23, 2019, 4:35pm

Hi Barth,
thank you for looking into this!

so we are now reducing of a factor 4 the histograms size for the time being and also reducing of a factor 5 the update rate. We will need in the next weeks to find a solution on how to display more than 32 k events with pulse activated.

In the workflow that we have now, the logic of looking for new files to be processed is coded inside the QC task (we discussed already in person that we have been creating a monster and we are aware of it) so for what I understand we are also updating the histograms when there are not “new files”. This is clear not the behavior that we want and Zhaozhong is looking into it. Do you know if there is a mechanism possible from inside a QC task to send a command that stops the updating?

Ciao and thank you,
Ivan

bvonhall · May 24, 2019, 6:24am

Hi Ivan,
I can also adapt the clean up policy of the CCDB for the ITS tasks in order to remove faster the multiple versions of a given object. For example, instead of a grace period of 1 day during which versions are not touched, we could have just 1 hour, and then we keep 1 version per hour any ways.
Concerning the second point, if you update to the latest version of the QualityControl (v1.13 or master) there is a method “stopPublishing” in ObjectsManager. Maybe you can use that.
Cheers,
Barth

icali · May 24, 2019, 7:31am

Hi Barth,
yes, at least for the time being it would be very useful to reduce to 1 h the grace period for the ITS at least we are sure to don’t saturate the CCDB. With the QC workflow that we are (should) be running now we are publishing 9 MB objects every 2-5 mins. We can review the situation when the workflow is close to final. Zhaozhong is also implementing the “stopPubshing” method now and we hope to have it in production by this afternoon.

Thank again,
Cheers,
Ivan