We are running QC for ITS inner barrel on our flpits1 and flpits5.
Recently, the QC processing time was dramatically getting longer. For the same run (same data), the processing time increased from 1 hour to 2 hours. We tracked the QC threads and found the pipeline for lz4 decompression and O2 data decoding slowed down the QC processing time.
This problem happened on flpits5 roughly 2 weeks ago and is still there. On flpits1, it happened yesterday afternoon around 4 pm. We tested a standalone lz4 decompression for our data, it looks like the speed is around 4MB/s. The CPU usage is kept around 55% during the decompression. The available disk space is fine on both flps.
I am sure that we did not change our QC and O2. For the other changes on these machines, I am verifying to my colleagues.
We investigated the QC processing sequence, and found that the lz4 decompression and Raw-Pixel-Reader are rather slower than before. Then Qc-Task-Runner have to wait for a long time (but the processing is fine).
We read the data from /data/shifts/runs, which is mounted on /dev/sde1. Actually it’s done via pipes, the pipes (for flpits1) are under /home/its/QC/workdir/infiles
The typical size of a single file is 1.3GB (.lz4 file).
by looking at the YUM logs there is a alisw-lz4 update done on flpits2 on 11 October and flpits1 this morning at 10:18 (therefore too late to match your timeline). This is the only obvious difference I can spot from the system logs. I did not check SW that has been updated outside of YUM.
Otherwise the two machines followed the same upgrade path and the same rebooting episodes. As they both have not been rebooted in the last 32 days, I would exclude an effect induced by a Linux kernel update (for updates both active and pending).