QC task:Error - ServiceDiscovery: Timeout was reached

[sheckel (Stefan Heckel)]

Dear George, dear Barth,

thanks for your replies. So for the publishing of the objects and displaying the most recent ones via the QCG we do not have any isses, this also works without the consul.

Now, for me the questions remains, what about the connection from the reco workflow to the QC task? Up to now, what we do, is piping a QC task directly to the reco workflow. E.g., we first run a TPC O2 simulation, more precisely: the hit creation, digitisation and clusterisation. Afterwards, we start the reco workflow (performing the tracking) with a QC task directly piped to it. In this setup, first the reco workflow will run, finish the tracking, and when it is finished, provide the whole bunch of tracks once to the QC task. The QC task continues to run, but it will not receive further input. Now, what happens, when the reco workflow is running continuously (if I understand corretly, that would mean, in online mode)? In this latter case, we cannot use a simple pipe, because that would wait for the reco workflow to finish, which, however, is continuously running.

To me it is still unclear, how to get this connection from the continuously running reco workflow to the QC task, and whether for this we need the consul.

Cheers,
Stefan

[pkonopka (Piotr Konopka)]

Dear Stefan,
I am not sure I completely understand, I have an impression that you might have a few wrong assumptions about how O2, DPL and QC work. Let’s tackle it one by one.

@sheckel

In this setup, first the reco workflow will run, finish the tracking, and when it is finished, provide the whole bunch of tracks once to the QC task. The QC task continues to run, but it will not receive further input.

If you need it to exit gracefully, you can send EndOfStream signal, as described a few post above.

@sheckel

Now, what happens, when the reco workflow is running continuously (if I understand corretly, that would mean, in online mode)?

QC tasks also run continuously. They can receive new data over and over, each x seconds (10 in debug, 60 in production) they will publish new versions of MonitorObjects.

@sheckel

In this latter case, we cannot use a simple pipe, because that would wait for the reco workflow to finish, which, however, is continuously running.

The fact that you pipe the workflows ( say workflow-a | workflow-b) doesn’t mean, that the first one sends detector data through stdout and the latter receives them in stdin. Anyway, one process doesn’t always have to exit before the following one, after pipe, receives its stdin.

By piping these workflows, we only sum up their configuration, but all the DPL devices are run anyway by the last executable in the chain. Then, they transfer data by message queues. However, this is an internal detail and hopefully you shouldn’t have to worry about it too much.
QC tasks should be fine running continuously and I am pretty sure that other detector teams use them this way.

@sheckel

continuously (if I understand corretly, that would mean, in online mode)

By the way, in production, we won’t use pipes. We will just connect different workflows by matching their channels with the Control software(s).

@sheckel

To me it is still unclear, how to get this connection from the continuously running reco workflow to the QC task…

You don’t have to do anything unusual, I guess. If a reco workflow sends data continuously, QC tasks will continuously receive them. Please tell us if you encounter problems with that.

@sheckel

, and whether for this we need the consul.

You absolutely don’t need consul for that. As for now, it is only needed if you want to see in QCG which Monitor Objects (MOs) are currently published (a.k.a. the online mode). Anyway, you will be able to see the latest published MOs, but they won’t be marked as “online”.

I hope that clarifies things. Don’t hesitate to ask more questions if something is not clear.
Best, Piotr

[sheckel (Stefan Heckel)]

Dear Piotr,

thanks a lot for your detailed clarifications! I think, now I much better understand, how the different things works together. The connection from the reco workflow to the QC task is provided by the DPL and should work also running continously. The consul is only needed for checking, if the QC task has published anything new, and in this case updating the QCG. But this also works without consul, just that one has to update the QCG manually. Now, for the pre-commissioning this is not an issue, so we will not use the consul.

Thanks again!
Cheers, Stefan

[graduta (George Raduta)]

Hi @sheckel,

In order to use Online Mode in
QCG, you will need to configure and use Consul.

QC uses ServiceDiscovery to post the name of the live objects to Consul and then
QCG will monitor Consul to refresh the current objects that are considered to be in live mode.

Hence, without Consul, QCG will not have Online Mode.

I would also like to ask @bvonhall to read your requirement and confirm that it would work from QC side.

Thank you,

George