Specify output of o2-tpc-reco-workflow

Dear experts,

we would like to pipe the output (tracks) of the o2-tpc-reco-workflow directly into o2-qc via

o2-tpc-reco-workflow --infile tpcdigits.root --output-type tracks | o2-qc --config json://PATH/TO/.json

I assume what needs to be done is to change the output in a way that it is not written to file anymore but provided as “TPC”, “TRACKS” in some kind of DataProcessorSpec. I think this can be done in O2/Detectors/TPC/workflow/src/RecoWorkflow.cxx, since there the writing-to-file is defined. I naively removed this part and added

DataProcessorSpec trackHandler{
      "tpc-track-handler",
      Inputs{{"inputTracks", {"TPC", "TRACKS"}}},
      Outputs{
      { "TPC", "TRACKS", 0, Lifetime::Timeframe } },
    };

instead.

{"inputTracks", {"TPC", "TRACKS"}}

is the InputSpec that was given to the TreeWriter before and

Outputs{
      { "TPC", "TRACKS", 0, Lifetime::Timeframe } },

comes from our file reader in Modules/TPC/run and there it is working as proper output.
There is for sure a better way to do this but in principle I guess a DataProcessorSpec that takes the tracks, that come from the tpc-tracker and provide it to the devices in the DPL as TPC TRACKS should somehow do the trick.

However, looking at the debug GUI it seems like the connection between the devices in DPL is wrong.
Here you can fin a screenshot of the GUI. I don’t understand why the tpc-tracker would directly go to QC-TASK-RUNNER-PIDtest and also why the tpc-track-handler makes a loop around the QC-TASK-RUNNER-PIDtest.
If I remove the track-handler, everything is the same except that obviously this nonsensical loop is missing.
The errors I get in the log are the following:
For the tpc-tracker:

[3774:tpc-tracker]: [19:42:07][ERROR] Failed to open input file matbud.root
[3774:tpc-tracker]: [19:42:23][ERROR] requested channel has not been configured? check channel names/configuration.
[3774:tpc-tracker]: [19:42:23][ERROR] channel: from_tpc-tracker_to_tpc-track-handler, index: 0
[3774:tpc-tracker]: [19:42:23][ERROR] out of range: _Map_base::at
[3774:tpc-tracker]: [19:42:23][ERROR] Exception caught: _Map_base::at
[3774:tpc-tracker]: [19:42:23][ERROR] requested channel has not been configured? check channel names/configuration.
[3774:tpc-tracker]: [19:42:23][ERROR] channel: from_tpc-tracker_to_tpc-track-handler, index: 0
[3774:tpc-tracker]: [19:42:23][ERROR] out of range: _Map_base::at
[3774:tpc-tracker]: [19:42:23][ERROR] out of range: _Map_base::at
[3774:tpc-tracker]: [19:42:23][ERROR] incorrect/incomplete channel configuration?
[3774:tpc-tracker]: [19:42:23][ERROR] Unhandled exception reached the top of main: _Map_base::at, device shutting down.

and for the tpc-track-handler:

[3792:tpc-track-handler]: [19:42:06][ERROR] requested channel has not been configured? check channel names/configuration.
[3792:tpc-track-handler]: [19:42:06][ERROR] channel: from_tpc-tracker_to_tpc-track-handler, index: 0
[3792:tpc-track-handler]: [19:42:06][ERROR] out of range: _Map_base::at
[3792:tpc-track-handler]: [19:42:06][ERROR] out of range: _Map_base::at
[3792:tpc-track-handler]: [19:42:06][ERROR] incorrect/incomplete channel configuration?
[3792:tpc-track-handler]: [19:42:07][ERROR] Unhandled exception reached the top of main: _Map_base::at, device shutting down.

Clock and file sink have the same error setting up the workflow:

[3810:internal-dpl-clock]: [ERROR] error while setting up workflow: Error while parsing serialised workflow

A extensive log for a run with the handler can be found here.

Removing the tpc-track-handler leads to similar problems.

[15812:tpc-track-writer]: [19:55:13][ERROR] requested channel has not been configured? check channel names/configuration.
[15812:tpc-track-writer]: [19:55:13][ERROR] channel: from_tpc-track-writer_to_QC-TASK-RUNNER-PIDtest, index: 0
[15812:tpc-track-writer]: [19:55:13][ERROR] out of range: _Map_base::at
[15812:tpc-track-writer]: [19:55:13][ERROR] out of range: _Map_base::at
[15812:tpc-track-writer]: [19:55:13][ERROR] incorrect/incomplete channel configuration?
[15812:tpc-track-writer]: [19:55:13][ERROR] Unhandled exception reached the top of main: _Map_base::at, device shutting down.

However, there are no clock and sink errors.
A extensive log for a run without the handler can be found here.

Again a screenshot of the debugGUI. Imho the QC-TASK-RUNNER-PIDtest should not be connected to the tpc-track-writer so it seems to me that the in- and outputs need to be properly defined but I do not really know how.

Any help is greatly appreciated! Thanks a lot for reading.

Cheers,
Thomas

Dear Thomas,

I must have skipped somehow your message. Sorry about that.

I understand that you are trying to plug the qc to your reconstruction workflow but it currently dumps the tracks in a file. Thus it has to be updated to output the tracks (instead or in addition).

The workflow has been written by various people who could probably give you more insightful help (@richterm ?) but let me try as well.

I don’t think that you need a tpc-track-handler. You need tpc-tracker to output the tracks, which it seems to be already doing (see line 534 of CATrackerSpec.cxx).
Therefore, one thing you could try is to configure the qc to get "TPC/TRACKS/0" and just pipe it to the original workflow.

I hope it helps, best regards,
Barth

Dear Barth,

thanks for the reply.

Meanwhile I was in contact with @richterm. You are right, the tpc-track-handler is not needed because the tracks are already provided as TPC/TRACKS from the tracker. It turned out that it works when I remove the tpc-track-writer from RecoWorkflow.cxx. However, I think it should also work having the tpc-track-writer and the QC task running at the same time.
When I try to have both running, there is a connection between the output of the QC-TASK-RUNNER-PID and the input of the tpc-track-writer that I don’t understand. It also gives the following errors:

[9635:tpc-track-writer]: [13:18:49][ERROR] requested channel has not been configured? check channel names/configuration.
[9635:tpc-track-writer]: [13:18:49][ERROR] channel: from_tpc-track-writer_to_QC-TASK-RUNNER-PID, index: 0
[9635:tpc-track-writer]: [13:18:49][ERROR] out of range: _Map_base::at
[9635:tpc-track-writer]: [13:18:49][ERROR] out of range: _Map_base::at
[9635:tpc-track-writer]: [13:18:49][ERROR] incorrect/incomplete channel configuration?
[9635:tpc-track-writer]: [13:18:49][ERROR] Unhandled exception reached the top of main: _Map_base::at, device shutting down.

The run log can be found here.
A screenshot of the debug GUI is here.
The DPL configuration is here.

Is there anything else I have to configure and if so, what and how?
And is there anything obvious that I don’t see, that connects the QC-TASK-RUNNER-PID to the tpc-track-writer?
Maybe you have an idea, I’m happy with anything. I think Matthias is on vacation at the moment.

Cheers,
Thomas

Dear Thomas,

The connection from QC-TASK-RUNNER-PID to tpc-track-writer is probably due to the fact that both needs TPC/TRACKS. The DPL, if I am not mistaken, will not send the data to several devices but will chain them. @eulisse @richterm might be able to clarify this.

The error you pasted is puzzling though. I don’t get why there would be a channel between tpc-track-writer to QC-TASK-RUNNER-PID. @pkonopka any idea ?

Cheers,
Barth

Hello,
I don’t see any channels from tpc-track-writer to QC-TASK-RUNNER-PID indeed. It could be that both QC-TASK-RUNNER-PID and tpc-track-writer think they receive TPC/TRACKS first and they want to send it further to the other guy, which is clearly wrong.

If they both work well separately, both fail in the same workflow, I would look for a problem in DPL and how it merges workflows or connects devices with each other. @eulisse It might be indeed interesting for you.

Best, Piotr

Hi all,
thanks for your comments.

I understand now that in principle the connection from QC-TASK-RUNNER-PID to tpc-track-writer has to be there, since they take the same input, however, if there is a connection between the two devices in one direction then there should not be one in the other direction.

Is this still under investigation? This is important for us for the pre-commissioning of the TPC in mid-March.

Cheers,
Thomas

FYI, this should be reproducible with the more or less latest QualityControl master.

o2-sim -m TPC
o2-sim-digitizer-workflow
o2-tpc-reco-workflow --infile tpcdigits.root --output-type tracks | o2-qc --config json:/${QUALITYCONTROL_ROOT}/etc/tpcQCPID_direct.json

Cheers,
Thomas

Dear experts,

we have a similar problem when chaining the EMCAL reconstruction workflow and the EMCAL QC: Both the reconstruction specs and the QC task runner should subscribe directly (input spec “EMC/DIGITS/0”), however from the debug GUI it looks like the reconstruction workflow components subscribe to the QC task runner. Example here: https://cernbox.cern.ch/index.php/s/AL5xf49oGskSgwC. This results in the following error messages:

[19378:emcal-digit-reader]: [14:58:19][ERROR] requested channel has not been configured? check channel names/configuration.
[19378:emcal-digit-reader]: [14:58:19][ERROR] channel: from_emcal-digit-reader_to_EMCALDigitsPrinter, index: 0
[19378:emcal-digit-reader]: [14:58:19][ERROR] out of range: unordered_map::at: key not found
[19378:emcal-digit-reader]: [14:58:19][ERROR] Exception caught: unordered_map::at: key not found

If I run the EMCAL reconstruction workflow and the EMCAL QC (with a dedicated stand-alone) workflow separately they get properly configured and the error does not appear.

Cheers

Markus

In case this is helpful information:

When I run

o2-tpc-reco-workflow --infile tpcdigits.root --output-type tracks | o2-qc --config json://someconfigfile.json

specifying a task that takes CLUSTERNATIVE instead of TRACKS I have again the same problem that I described earlier, where apparently a connection between the two devices that take the same input from the DPL is generated in both directions. In this case the devices are QC-TASK-RUNNER-Clusters and tpc-tracker.

[4472:tpc-tracker]: [18:36:39][ERROR] requested channel has not been configured? check channel names/configuration.
[4472:tpc-tracker]: [18:36:39][ERROR] channel: from_tpc-tracker_to_QC-TASK-RUNNER-Clusters, index: 0
[4472:tpc-tracker]: [18:36:39][ERROR] out of range: _Map_base::at
[4472:tpc-tracker]: [18:36:39][ERROR] out of range: _Map_base::at
[4472:tpc-tracker]: [18:36:39][ERROR] incorrect/incomplete channel configuration?
[4472:tpc-tracker]: [18:36:39][STATE] Exiting FairMQ state machine
[4472:tpc-tracker]: [18:36:39][ERROR] Unhandled exception reached the top of main: _Map_base::at, device shutting down.

run log
dpl configuration

If I remove the tpc-tracker this error does not occur.

I guess the two errors I described stem from the same underlying problem.

Cheers,
Thomas

Hi, sorry for stepping in late into the discussion. There is indeed a bug in building the routing configuration when workflows are merged. I have opened a PR illustrating the bug


We will work on a solution.

Last Friday I have discussed with Jens to add an option to the TPC reco workflow to skip setting up the writers. This is probably useful anyhow and and can be used as a workaround for this problem

Cheers
Matthias

1 Like

If they both work well separately, both fail in the same workflow, I would look
for a problem in DPL and how it merges workflows or connects devices with each
other. @eulisse It might be indeed interesting for you.

Ok, raising priority on this.

1 Like

For completeness:
this issue was solved by https://github.com/AliceO2Group/AliceO2/pull/3349.