AliECS workflow and task templates documentation

divia · February 5, 2021, 9:38am

Which InfoBrowser do you use? The Web version or the application?

dstocco · February 5, 2021, 9:40am

Web version I guess (I’ve opened it on firefox at port 8081)

divia · February 5, 2021, 9:49am

OK. There is also an application:

/opt/o2-InfoLogger/bin/infoBrowser

The application as an “Export” feature to save all messages or just the selected messages to a text file.

dstocco · February 5, 2021, 10:22am

Ok, so, if I switch to detector MID, I can launch readout-dataflow w/o QC. If I switch on QC, it complains, most likely because the corresponding QC task is missing:

Error in <TSystem::ExpandFileName>: input: $HOME/.root.mimes, output: $HOME/.root.mimes task="github.com/dstocco/ControlWorkflows/tasks/stfb-qc-dispatcher@dab5bb81d1dd14fd080b5d9678b35024abb6

I want therefore to create a similar workflow, in which, instead of the QC, I put the o2-dpl-raw-parser…which simply dump the read-out data.
This is done with:

github.com

dstocco/ControlWorkflows/blob/mid/workflows/readout-mid.yaml

name: readout-dataflow
defaults:
  roc_ctp_emulator_enabled: "false"
  dd_enabled: "true"
  qcdd_enabled: "true"
  stfb_standalone: "false"
  odc_enabled: "false"
  monitoring_qc_url: "no-op://"
  monitoring_dd_url: "no-op://"
  monitoring_readout_url: "no-op://"
  detector: MID
roles:
  - name: host-{{ it }}
    for:
      range: "{{ hosts }}"
      var: it
    vars:
      readout_cfg_uri_standalone: "consul-ini://{{ consul_endpoint }}/o2/components/readout/readout-standalone-{{ it }}"
      readout_cfg_uri_stfb: "consul-ini://{{ consul_endpoint }}/o2/components/readout/readout-stfb-{{ it }}"
    constraints:

This file has been truncated. show original

and

github.com

dstocco/ControlWorkflows/blob/mid/tasks/mid-checker.yaml

name: mid-checker
defaults:
  feeId-config: "/home/flp/daq_utils/config/feeId_mapper.txt"
  crate-masks: "/home/flp/daq_utils/config/crate_masks.txt"
  user: flp
  fmq_severity: info
control:
  mode: "fairmq"
wants:
  cpu: 0.15
  memory: 128
# command:
#   shell: true
#   value: >-
#     source /etc/profile.d/modules.sh && MODULEPATH=/opt/alisw/el7/modulefiles/ module load O2 && o2-mid-raw-checker-workflow
#   user: "{{ user }}"
#   arguments:
#     - "-b"
#     - "--session"
#     - "default"

This file has been truncated. show original

However, when I try to create the workflow, it stops due to syntax issues:

13 INTERNAL: cannot create new environment: cannot load workflow template: task template load error (template=/var/lib/o2/aliecs/repos/github.com/dstocco/ControlWorkflows/tasks/mid-checker.yaml): yaml: line 33: did not find expected '-' indicator

But I do not quite get what indicator it is expecting. Is it just because I did not specify the bind option or is it a trivial syntax error that I cannot figure out?

dstocco · February 5, 2021, 10:33am

Forget it: it was just a double quote in one of the element of the argument lists…
I have other issues, but I guess I simply need to debug the workflow myself.

dstocco · February 5, 2021, 11:45am

My workflow seems to be invalid:

Invalid workflow: No matching output found for <matcher query: (and origin:MID (and description:RAWDATA (just startTime:$0 )))>. Candidates: task="github.com/dstocco/ControlWorkflows/tasks/mid-checker@ec142a33491d99c8cf26877e135812cb932f9a6

@malexis : is there a way to dump the dpl configuration as a json file on order to understand what’s wrong?

malexis · February 5, 2021, 12:05pm

Hi @dstocco ,

If you want to get the DPL dump, you should obtain it from DPL directly by running your DPL workflow outside AliECS with the --dump-workflow flag. If you want to check the channel configuration you might be able to retrieve it from Mesos sandbox logs. To get the logs you need to visit localhost:5050 and then find the task and click on Sandbox and stdout on the new page

dstocco · February 5, 2021, 2:15pm

Thanks for the info.
After investigation, it seems I’m unable to link with the AlIECS framework the output of stfb-raw-proxy with the input of mid-checker.
The first should output to: from_readout-proxy_to_Dispatcher
So, in the second I write:

connect:
- name: from_readout-proxy_to_Dispatcher
type: pull
rateLogging: “60”

Should I try with bind? Notice that the task is basically a sink, so there is no output message.

malexis · February 5, 2021, 3:13pm

Hi @dstocco ,

Could you try with the bind? Disentangling channel configuration and translating to a ControlWorkflow is a bit hard to do without the DPL dump and a bit of debugging. If you can provide me with the file, I could have a look but probably next week.

Kind Regards,
Miltiadis

dstocco · February 5, 2021, 4:25pm

Hi @malexis ,
I put the files here for reference:

https://cernbox.cern.ch/index.php/s/7TQQ5XNUZHJZhro

But please do not lose time on it. Probably the easiest solution is that we use QC-based tasks, so that we just need to slightly modify the standard readout-dataflow.yaml.
Thanks!
Cheers,
Diego

costaf · February 15, 2021, 5:34pm

Dear @dstocco and @malexis we should progress on this topic and find a solution or we should provide help if needed.
It would be interesting to have the possibility to run the whole data acquisition chain from ALIECS, while the MID group waits for the MID UL to be validated.

I think the main goal would be to run all the needed software/rpoxy from ALIECS.

Please prepare the list of required actions needed … I will take care of pinging the different experts needed.

dstocco · February 15, 2021, 5:55pm

Ciao @costaf ,
I agree. Well, as I said one possibility is to move my checkers to the QC.
However, on a second thought, there are 2 devices involved:

the decoder
the checker

Now, the checker can be moved to the QC. But I need the decoder to run (under O2) as it is needed to build the CTF.
So, I guess that the priority is to change the ControlWorkflow to allow me to run the decoder…

Cheers,
Diego

dstocco · February 16, 2021, 3:01pm

Dear all,
when I launch the AliECS, I see that the readout.exe is launched with the config file:

consul-ini://{{ consul_endpoint }}/o2/components/readout/readout-stfb-{{ it }}

[consul-ini://myhost:8500/o2/components/readout/readout-stfb-myhost]

How can I inspect the configuration file?

Thanks,
cheers,
Diego

malexis · February 16, 2021, 3:10pm

Hi @dstocco and @costaf ,

I am sorry I didn’t find the time to check the workflows.

In order to see the file, you can visit localhost:8500/ui/ . Then select Key/Value

and after that navigate through the files to the o2/components/readout/readout-stfb-localhost

I am using localhost as an example it should be the hostname of the head node and the readout-stfb-{{flp hostname}} is using the hostname of the flp where the task is being executed.

Kind regards,
Miltiadis

dstocco · February 16, 2021, 3:28pm

Hi again,
ok, so let’s try to convert the AliECS problem into a dpl problem.
I’m looking at the infologger, trying to figure out what commands the AliECS is launching.

I’m launching on one window:

source /etc/profile.d/modules.sh && MODULEPATH=/opt/alisw/el7/modulefiles/ module load O2 Control-OCCPlugin && o2-dpl-raw-proxy -b --id readout-proxy --log-color false --dataspec B:MID/RAWDATA --session default --shm-monitor false --severity=info -S $CONTROL_OCCPLUGIN_ROOT/lib/ -P OCClite --color false --control-port 31002

and on the other:

source /etc/profile.d/modules.sh && MODULEPATH=/opt/alisw/el7/modulefiles/ module load O2 Control-OCCPlugin && o2-dpl-raw-parser -b --id MIDChecker --session default --input-spec B:MID/RAWDATA --severity=info --log-level 1 -S $CONTROL_OCCPLUGIN_ROOT/lib/ -P OCClite --color false --control-port 31001

Now, the dpl-proxy starts just fine and waits for inputs from the StfBuilder.
The second command, on the other hand, crashes immediately with:

Invalid workflow: No matching output found for <matcher query: (and origin:MID (and description:RAWDATA (just startTime:$0 )))>. Candidates:

@eulisse : how can I link the dataspec output of the dpl-raw-proxy with the input-spec of the raw-parser?

This is the same kind of error that prevents the creation of the workflow via AliECS, so I guess that, once I solve it, the AliECS will work.

Thanks,
cheers,
Diego

dstocco · February 16, 2021, 3:30pm

Hi @malexis ,
thanks for the info. I can indeed access the cfg…but I see that it a dummy roc emulator is enabled. What’s the proper way to change it? Should I modify the Readout URI field in the AliECS GUI?

malexis · February 16, 2021, 3:51pm

Hi @dstocco ,

The Readout URI in the AliECS GUI is for pointing to another configuration file or consul input.

Since you are able to access the cfg you can edit it inside consul and then hit save at the end of the page.

Andry · April 8, 2021, 9:57am

Dear experts,

I have been going through this thread but I could still not figure out how to run the decoding and digit writing O2 DPLs on the fly and locally on the FLPs, i.e. fed by raw data streaming from readout data flow and controlled by AliECS. Sorry for that. The main difficulty is to edit the templates in http://github.com/AliceO2Group/ControlWorkflows to add the pipes to the decoding and digit writing DPLs: which template to edit? a task or a workflow? Where in the yaml file should I add the pipes to decoding and digit writing to be run locally on the FLP?

Thanks in advance for any further hint on that subject.

Regards,
Andry

pkonopka · April 8, 2021, 2:02pm

Hello Andry,

I understand your confusion, I hope I will clarify it a bit for you and let you advance.

Workflows contain sets of tasks. If you run a DPL command, this is like executing a workflow. Each DPL Data Processor (a process, device) is considered as a task by AliECS. Thus, if you want to add decoding, you effectively want to create a new workflow or extend an existing one and also add the tasks which it should contain.

This is probably a misconception. If you run a DPL command like this:

o2-dpl-first-workflow | o2-dpl-second-workflow

… then the pipe does not transfer the detector data. In reality, the first workflow sends only its configuration to the second one, and that one will execute all the processing.

We have a master workflow for the FLPs - readout-dataflow. This one contains readout, STFB, STBS and some DPL workflows. My recent PR will actually disentangle the DPL workflows from readout-dataflow and they will load a specified DPL workflow from the ones available in the workflows directory. After that, you would still use readout-dataflow, but it would also load your specific DPL sub-workflow as well.

Recently I’ve made some improvements regarding generating the AliECS workflow and task templates. This should allow you to automatically generate your templates if you have a ready DPL command. I’ve just written some documentation on how to do that and I would actually appreciate if you tried following it and let me know your feedback. Please note that you would need the latest O2@dev to use the automatic export feature and this ControlWorkflows branch (which should be merged to the master and flp-suite-v0.17.0 in a few days, I hope).

Let me know if something is still unclear!

Andry · April 9, 2021, 9:02am

Hello Piotr,

Thanks a lot for taking the time to go to his detailed explanation and for providing so many links. They will be precious to go deeper and experiment. I will try as soon as we get the new FLP suite (crossing fingers here on the resolution of the bugs affecting its release).

Cheers,
Andry