Running multiple instances of o2-qc with pipe fails

O²
1

Hello experts,

when running multiple instances of o2-qc with pipes, the workflows crash. E.g. like this

o2-tpc-reco-workflow --infile tpcdigits.root --output-type tracks | o2-qc --config json:/${QUALITYCONTROL_ROOT}/etc/tpcQCTracks_direct.json | o2-qc --config json:/${QUALITYCONTROL_ROOT}/etc/tpcQCPID_direct.json

Both QC workflows take TPC tracks as input.

The errors:

[576:internal-dpl-injected-dummy-sink]: [09:52:30][ERROR] requested channel has not been configured? check channel names/configuration.
[576:internal-dpl-injected-dummy-sink]: [09:52:30][ERROR] channel: from_QC-TASK-RUNNER-PID_to_internal-dpl-injected-dummy-sink, index: 0
[576:internal-dpl-injected-dummy-sink]: [09:52:30][ERROR] out of range: _Map_base::at
[576:internal-dpl-injected-dummy-sink]: [09:52:30][ERROR] out of range: _Map_base::at
[576:internal-dpl-injected-dummy-sink]: [09:52:30][ERROR] incorrect/incomplete channel configuration?
[576:internal-dpl-injected-dummy-sink]: [09:52:30][ERROR] Unhandled std::exception reached the top of main: _Map_base::at, device shutting down.
[ERROR] websocket_server_callback: Error while reading from websocket
[564:QC-CHECK-RUNNER-QcCheck]: [09:52:32][ERROR] requested channel has not been configured? check channel names/configuration.
[564:QC-CHECK-RUNNER-QcCheck]: [09:52:32][ERROR] channel: from_QC-TASK-RUNNER-Tracks_to_QC-CHECK-RUNNER-QcCheck, index: 0
[564:QC-CHECK-RUNNER-QcCheck]: [09:52:32][ERROR] out of range: _Map_base::at
[564:QC-CHECK-RUNNER-QcCheck]: [09:52:32][ERROR] out of range: _Map_base::at
[564:QC-CHECK-RUNNER-QcCheck]: [09:52:32][ERROR] incorrect/incomplete channel configuration?
[564:QC-CHECK-RUNNER-QcCheck]: [09:52:32][ERROR] Unhandled std::exception reached the top of main: _Map_base::at, device shutting down.
[ERROR] websocket_server_callback: Error while reading from websocket
[555:QC-TASK-RUNNER-Tracks]: [09:52:50][ERROR] requested channel has not been configured? check channel names/configuration.
[555:QC-TASK-RUNNER-Tracks]: [09:52:50][ERROR] channel: from_QC-TASK-RUNNER-Tracks_to_QC-CHECK-RUNNER-QcCheck, index: 0
[555:QC-TASK-RUNNER-Tracks]: [09:52:50][ERROR] out of range: _Map_base::at
[555:QC-TASK-RUNNER-Tracks]: [09:52:50][ERROR] Exception caught: _Map_base::at 
[555:QC-TASK-RUNNER-Tracks]: [09:52:50][ERROR] Unhandled o2::framework::runtime_error reached the top of main, device shutting down.

Here is the full log.

Something I noticed and might be helpful for debugging: When the order of the tasks in the command is switched (PID first, Tracks second), then QC-TASK-RUNNER-PID shows the error and QC-CHECK-RUNNER-QcCheck complains about the channel from_QC-TASK-RUNNER-PID_to_QC-CHECK-RUNNER-QcCheck. I don’t think that the error in internal-dpl-injected-dummy-sink is the real problem here.

Removing the checkers from the QC tasks solves the problem. The checkers, which do nothing at the moment, are configured as follows:

Tracks:

"checks": {
      "QcCheck": {
        "active": "true",
        "className": "o2::quality_control_modules::skeleton::SkeletonCheck",
        "moduleName": "QcSkeleton",
        "policy": "OnAny",
        "detectorName": "TPC",
        "dataSource": [{
          "type": "Task",
          "name": "Tracks",
          "MOs": ["example"]
        }]
      }
    }

PID:

"checks": {
      "QcCheck": {
        "active": "true",
        "className": "o2::quality_control_modules::skeleton::SkeletonCheck",
        "moduleName": "QcSkeleton",
        "policy": "OnAny",
        "detectorName": "TPC",
        "dataSource": [{
          "type": "Task",
          "name": "PID",
          "MOs": ["example"]
        }]
      }
    }

Is there any kind of configuration we need to make, to get the checkers running or is this really a bug?

Thanks a lot for any help!

Cheers,
Thomas

2

Hi, can you try using different check names (I mean the QcCheck strings in config files)? I wouldn’t expect QC to behave well with two different Checks with the same name.

3

Thanks a lot, Piotr! This was an oversight on our side!

Works as expected when I put different names for the checkers.

4

I am happy that it works now :slight_smile:
I noted down to check for duplicates, but I am afraid it won’t be trivial when e.g. Checks come from different workflows: https://alice.its.cern.ch/jira/browse/QC-574

5

This would for sure come in handy but I would say that it was a mistake on our side in this case.