O2 analysis tutorials hangs at websocket handshake

O²
1

Hi,

I am new to O2 and am trying to run the analysis tutorials to better understand the framework. I am running Big Sur 11.2.3. O2 is build with aliBuild init O2@dev --defaults o2 and aliBuild build O2 --defaults o2 and builds correctly. However, when trying to run the analysis tutorials, e.g.

o2-analysistutorial-track-iteration --aod-file AO2D.root

I get the following output, where O2 never moves past the handshake:

[INFO] Initialising O2 Data Processing Layer. Driver PID: 1313.
[INFO] O2 Data Processing Layer initialised. We brake for nobody.
[INFO] Optimised build. O2DEBUG / LOG(DEBUG) / LOGF(DEBUG) / assert statement will not be shown.
[INFO] Redeployment of configuration asked.
[INFO] Starting internal-dpl-clock on pid 1314
[INFO] Starting internal-dpl-aod-reader on pid 1315
[INFO] Starting internal-dpl-aod-spawner on pid 1316
[INFO] Starting track-iteration-tutorial_A on pid 1317
[INFO] Starting internal-dpl-injected-dummy-sink on pid 1318
[INFO] Starting track-iteration-tutorial_B on pid 1319
[INFO] Redeployment of configuration done.
[1319:track-iteration-tutorial_B]: [INFO] Spawing new device track-iteration-tutorial_B in process with pid 1319
[1319:track-iteration-tutorial_B]: [07:59:19][INFO]
[1319:track-iteration-tutorial_B]: ______ _ _______ _________
[1319:track-iteration-tutorial_B]: / / ()______ |/ /_ __ \ version 1.4.37
[1319:track-iteration-tutorial_B]: / /_ / __ / / ___/__ /|_/ /_ / / / build RELWITHDEBINFO [1319:track-iteration-tutorial_B]: / __/ / /_/ / / / _ / / / / /_/ / https://github.com/FairRootGroup/FairMQ [1319:track-iteration-tutorial_B]: /_/ \__,_/_/_/ /_/ /_/ \___\_\ LGPL-3.0 © 2012-2021 GSI [1319:track-iteration-tutorial_B]: [1319:track-iteration-tutorial_B]: [07:59:19][STATE] Starting FairMQ state machine --> IDLE [1314:internal-dpl-clock]: [INFO] Spawing new device internal-dpl-clock in process with pid 1314 [1314:internal-dpl-clock]: [07:59:19][INFO] [1314:internal-dpl-clock]: ______ _ _______ _________ [1314:internal-dpl-clock]: / ____/___ _(_)_______ |/ /_ __ \ version 1.4.37 [1314:internal-dpl-clock]: / /_ / __ / / / /|/ /_ / / / build RELWITHDEBINFO
[1314:internal-dpl-clock]: / / / // / / / _ / / / / // / GitHub - FairRootGroup/FairMQ: C++ Message Queuing Library and Framework
[1314:internal-dpl-clock]: // _,/// // /_/ __\ LGPL-3.0 © 2012-2021 GSI
[1314:internal-dpl-clock]:
[1314:internal-dpl-clock]: [07:59:19][STATE] Starting FairMQ state machine → IDLE
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] IDLE —> INITIALIZING DEVICE
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] INITIALIZING DEVICE —> INITIALIZED
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] INITIALIZED —> BINDING
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] BINDING —> BOUND
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] BOUND —> CONNECTING
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] CONNECTING —> DEVICE READY
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] DEVICE READY —> INITIALIZING TASK
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] INITIALIZING TASK —> READY
[1319:track-iteration-tutorial_B]: [07:59:19][STATE] READY —> RUNNING
[1319:track-iteration-tutorial_B]: [07:59:19][INFO] DEVICE: Running…
[1319:track-iteration-tutorial_B]: [07:59:19][INFO] Correctly handshaken websocket connection.
[1314:internal-dpl-clock]: [07:59:19][STATE] IDLE —> INITIALIZING DEVICE
[1314:internal-dpl-clock]: [07:59:19][STATE] INITIALIZING DEVICE —> INITIALIZED
[1314:internal-dpl-clock]: [07:59:19][STATE] INITIALIZED —> BINDING
[1314:internal-dpl-clock]: [07:59:19][STATE] BINDING —> BOUND
[1314:internal-dpl-clock]: [07:59:19][STATE] BOUND —> CONNECTING
[1314:internal-dpl-clock]: [07:59:19][STATE] CONNECTING —> DEVICE READY
[1314:internal-dpl-clock]: [07:59:19][STATE] DEVICE READY —> INITIALIZING TASK
[1314:internal-dpl-clock]: [07:59:19][STATE] INITIALIZING TASK —> READY
[1314:internal-dpl-clock]: [07:59:19][STATE] READY —> RUNNING
[1314:internal-dpl-clock]: [07:59:19][INFO] DEVICE: Running…
[1314:internal-dpl-clock]: [07:59:19][INFO] Correctly handshaken websocket connection.
[1317:track-iteration-tutorial_A]: [INFO] Spawing new device track-iteration-tutorial_A in process with pid 1317
[1317:track-iteration-tutorial_A]: [07:59:19][INFO]
[1317:track-iteration-tutorial_A]: ______ _ _______ _________
[1317:track-iteration-tutorial_A]: / / ()______ |/ /_ __ \ version 1.4.37
[1317:track-iteration-tutorial_A]: / /_ / __ / / ___/__ /|_/ /_ / / / build RELWITHDEBINFO [1317:track-iteration-tutorial_A]: / __/ / /_/ / / / _ / / / / /_/ / https://github.com/FairRootGroup/FairMQ [1317:track-iteration-tutorial_A]: /_/ \__,_/_/_/ /_/ /_/ \___\_\ LGPL-3.0 © 2012-2021 GSI [1317:track-iteration-tutorial_A]: [1317:track-iteration-tutorial_A]: [07:59:19][STATE] Starting FairMQ state machine --> IDLE [1318:internal-dpl-injected-dummy-sink]: [INFO] Spawing new device internal-dpl-injected-dummy-sink in process with pid 1318 [1316:internal-dpl-aod-spawner]: [INFO] Spawing new device internal-dpl-aod-spawner in process with pid 1316 [1316:internal-dpl-aod-spawner]: [07:59:19][INFO] [1316:internal-dpl-aod-spawner]: ______ _ _______ _________ [1316:internal-dpl-aod-spawner]: / ____/___ _(_)_______ |/ /_ __ \ version 1.4.37 [1316:internal-dpl-aod-spawner]: / /_ / __ / / / /|/ /_ / / / build RELWITHDEBINFO
[1316:internal-dpl-aod-spawner]: / / / // / / / _ / / / / // / GitHub - FairRootGroup/FairMQ: C++ Message Queuing Library and Framework
[1316:internal-dpl-aod-spawner]: // _,/// // /_/ __\ LGPL-3.0 © 2012-2021 GSI
[1316:internal-dpl-aod-spawner]:
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] Starting FairMQ state machine → IDLE
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][INFO]
[1318:internal-dpl-injected-dummy-sink]: ______ _ _______ _________
[1318:internal-dpl-injected-dummy-sink]: / / ()______ |/ /_ __ \ version 1.4.37
[1318:internal-dpl-injected-dummy-sink]: / /_ / __ `/ / / /|/ /_ / / / build RELWITHDEBINFO
[1318:internal-dpl-injected-dummy-sink]: / / / // / / / _ / / / / // / GitHub - FairRootGroup/FairMQ: C++ Message Queuing Library and Framework
[1318:internal-dpl-injected-dummy-sink]: // _,/// // /_/ __\ LGPL-3.0 © 2012-2021 GSI
[1318:internal-dpl-injected-dummy-sink]:
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] Starting FairMQ state machine → IDLE
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] IDLE —> INITIALIZING DEVICE
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] INITIALIZING DEVICE —> INITIALIZED
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] INITIALIZED —> BINDING
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] BINDING —> BOUND
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] BOUND —> CONNECTING
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] CONNECTING —> DEVICE READY
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] IDLE —> INITIALIZING DEVICE
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] DEVICE READY —> INITIALIZING TASK
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] INITIALIZING DEVICE —> INITIALIZED
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] INITIALIZED —> BINDING
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] BINDING —> BOUND
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] BOUND —> CONNECTING
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] CONNECTING —> DEVICE READY
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] DEVICE READY —> INITIALIZING TASK
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] INITIALIZING TASK —> READY
[1316:internal-dpl-aod-spawner]: [07:59:19][STATE] READY —> RUNNING
[1316:internal-dpl-aod-spawner]: [07:59:19][INFO] DEVICE: Running…
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] IDLE —> INITIALIZING DEVICE
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] INITIALIZING DEVICE —> INITIALIZED
[1316:internal-dpl-aod-spawner]: [07:59:19][INFO] Correctly handshaken websocket connection.
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] INITIALIZED —> BINDING
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] BINDING —> BOUND
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] BOUND —> CONNECTING
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] CONNECTING —> DEVICE READY
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] DEVICE READY —> INITIALIZING TASK
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] INITIALIZING TASK —> READY
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][STATE] READY —> RUNNING
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][INFO] DEVICE: Running…
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] INITIALIZING TASK —> READY
[1317:track-iteration-tutorial_A]: [07:59:19][STATE] READY —> RUNNING
[1317:track-iteration-tutorial_A]: [07:59:19][INFO] DEVICE: Running…
[1318:internal-dpl-injected-dummy-sink]: [07:59:19][INFO] Correctly handshaken websocket connection.
[1317:track-iteration-tutorial_A]: [07:59:19][INFO] Correctly handshaken websocket connection.

This hangs for at least a couple of hours, and I am assuming that these tutorials should be fairly quick to run. Am I misunderstanding something in how to run these tutorials? The AO2D.root file is downloaded from Run3_Conversion LHC15o test output.

Cheers,
Emil

2

Can you check if adding:

fixes your issue?

3

Thanks for the suggestion. Could you expand on this? Is there some option I have to pass in order to allocate the memory? Using --help I can see there are some different options concerning shared memory and resources, but I am not sure which is appropriate.

4

I just merged it. Simply update your O2 and it should solve your issue.

5

Hi again,

I updated O2 and it did indeed move past the websocket handshake, so thanks for the help.
However, I’ve encountered a new problem, when running the tutorial:

o2-analysistutorial-track-iteration --aod-file AO2D.root -b

I get the following error

…
[27619:track-iteration-tutorial_A]: [11:06:13][STATE] READY —> RUNNING
[27619:track-iteration-tutorial_A]: [11:06:13][INFO] DEVICE: Running…
[27619:track-iteration-tutorial_A]: [11:06:13][INFO] Correctly handshaken websocket connection.
[27617:internal-dpl-aod-reader]: [11:06:14][INFO] Read info: lfn=AO2D.root,size=42733127,total_tf=1,read_tf=1,read_bytes=16187621,read_calls=8,io_time=1.2,wait_time=0.1
[27617:internal-dpl-aod-reader]: [11:06:14][INFO] No input files left to read for reader 0!
[27618:internal-dpl-aod-spawner]:
[27618:internal-dpl-aod-spawner]: *** Break *** illegal instruction

After which the program crashes. I have added the full log of the output as well.
log.txt (78.6 KB)

I appreciate the help.
Cheers,
Emil