[fitdaq@j91 bin]$ ./StfBuilder --id stf_builder-0 --transport shmem --detector FT0 --dpl-channel-name=dpl-chan --channel-config "name=dpl-chan,type=push,method=bind,address=ipc:///tmp/stf-builder-dpl-pipe-0,transport=shmem,rateLogging=1" --channel-config "name=readout,type=pull,method=connect,address=ipc:///tmp/readout-pipe-0,transport=shmem,rateLogging=1"
Warning in <UnknownClass::SetDisplay>: DISPLAY not set, setting it to node202.localdomain:0.0
[2020-03-25 12:17:55.838][I] 
      ______      _    _______  _________ 
     / ____/___ _(_)_______   |/  /_  __ \    version 1.4.12
    / /_  / __ `/ / ___/__  /|_/ /_  / / /    build RELWITHDEBINFO
   / __/ / /_/ / / /    _  /  / / / /_/ /     https://github.com/FairRootGroup/FairMQ
  /_/    \__,_/_/_/     /_/  /_/  \___\_\     LGPL-3.0  © 2012-2019 GSI

[2020-03-25 12:17:55.839][I] [STATE] Starting FairMQ state machine --> IDLE
[2020-03-25 12:17:55.839][D] PID: 9394
[2020-03-25 12:17:55.839][D] Loaded plugin: 'config', version '1.4.12', maintainer 'FairRootGroup <fairroot@gsi.de>', homepage 'https://github.com/FairRootGroup/FairRoot'
[2020-03-25 12:17:55.839][D] Loaded plugin: 'control', version '1.4.12', maintainer 'FairRootGroup <fairroot@gsi.de>', homepage 'https://github.com/FairRootGroup/FairMQ'
[2020-03-25 12:17:55.839][D] Running builtin controller: interactive
[2020-03-25 12:17:55.839][D] Plugin 'control' is setting up signal handling for SIGINT and SIGTERM
[2020-03-25 12:17:55.840][D] Configuration: 
catch-signals               = 1                                                                                                    <int>                 [default] 
channel-config              = name=dpl-chan,type=push,method=bind,address=ipc:///tmp/stf-builder-dpl-pipe-0,transport=shmem,rateLogging=1, name=readout,type=pull,method=connect,address=ipc:///tmp/readout-pipe-0,transport=shmem,rateLogging=1 <vector<string>>      [provided]
color                       = true                                                                                                 <bool>                [default] 
control                     = dynamic                                                                                              <string>              [default] 
data-sink-dir               =                                                                                                      <string>              [default] 
data-sink-enable            = false                                                                                                <bool>                [default] 
data-sink-file-name         = %i                                                                                                   <string>              [default] 
data-sink-max-file-size     = 4096                                                                                                 <unsigned long>       [default] 
data-sink-max-stfs-per-file = 0                                                                                                    <unsigned long>       [default] 
data-sink-sidecar           = false                                                                                                <bool>                [default] 
data-source-dir             =                                                                                                      <string>              [default] 
data-source-enable          = false                                                                                                <bool>                [default] 
data-source-rate            = 44                                                                                                   <unsigned long>       [default] 
data-source-repeat          = false                                                                                                <bool>                [default] 
detector                    = FT0                                                                                                  <string>              [provided]
dpl-channel-name            = dpl-chan                                                                                             <string>              [provided]
file-severity               = debug                                                                                                <string>              [default] 
gui                         = false                                                                                                <bool>                [default] 
id                          = stf_builder-0                                                                                        <string>              [provided]
init-timeout                = 120                                                                                                  <int>                 [default] 
input-channel-name          = readout                                                                                              <string>              [default] 
io-threads                  = 1                                                                                                    <int>                 [default] 
log-to-file                 =                                                                                                      <string>              [default] 
max-buffered-stfs           = -1                                                                                                   <long>                [default] 
max-run-time                = 0                                                                                                    <unsigned long>       [default] 
network-interface           = default                                                                                              <string>              [default] 
ofi-size-hint               = 0                                                                                                    <unsigned long>       [default] 
output-channel-name         = builder-stf-channel                                                                                  <string>              [default] 
rate                        = 0                                                                                                    <float>               [default] 
rdh-data-check              = [unidentified_type]                                                                                  <[unidentified_type]> [default] 
rdh-filter-empty-trigger-v4 = false                                                                                                <bool>                [default] 
session                     = default                                                                                              <string>              [default] 
severity                    = debug                                                                                                <string>              [default] 
shm-monitor                 = true                                                                                                 <bool>                [default] 
shm-segment-size            = 2000000000                                                                                           <unsigned long>       [default] 
stand-alone                 = false                                                                                                <bool>                [default] 
transport                   = shmem                                                                                                <string>              [provided]
verbosity                   = medium                                                                                               <string>              [default] 

[2020-03-25 12:17:55.840][I] [STATE] IDLE ---> INITIALIZING DEVICE
[2020-03-25 12:17:55.840][D] channel-config: Parsing channel configuration
[2020-03-25 12:17:55.842][D] Setting 'shmem' as default transport for the device
[2020-03-25 12:17:55.842][D] Adding 'shmem' transport
[2020-03-25 12:17:55.842][D] Transport: Using ZeroMQ (4.3.2) & boost::interprocess (1.70.0)
[2020-03-25 12:17:55.842][D] no fairmq-shmmonitor found for shared memory id 13488916, starting...
[2020-03-25 12:17:55.856][D] Started fairmq-shmmonitor for shared memory id 13488916
[2020-03-25 12:17:55.857][D] created/opened shared memory segment 'fmq_13488916_main' of 2000000000 bytes. Available are 1999999776 bytes.
[2020-03-25 12:17:55.857][D] device counter found, with value of 1. incrementing.
[2020-03-25 12:17:55.857][D] incremented device counter, now: 2
[2020-03-25 12:17:55.857][D] Initializing transport for channel dpl-chan[0]: shmem
[2020-03-25 12:17:55.857][D] Reusing existing 'shmem' transport
[2020-03-25 12:17:55.857][D] Initializing transport for channel readout[0]: shmem
[2020-03-25 12:17:55.857][D] Reusing existing 'shmem' transport
[2020-03-25 12:17:55.857][I] [STATE] INITIALIZING DEVICE ---> INITIALIZED
[2020-03-25 12:17:55.857][I] [STATE] INITIALIZED ---> BINDING
[2020-03-25 12:17:55.857][D] Validating channel 'dpl-chan[0]'... VALID
[2020-03-25 12:17:55.858][D] Created socket stf_builder-0.dpl-chan[0].push
[2020-03-25 12:17:55.858][D] Attached channel dpl-chan[0] to ipc:///tmp/stf-builder-dpl-pipe-0 (bind) (push)
[2020-03-25 12:17:55.858][I] [STATE] BINDING ---> BOUND
[2020-03-25 12:17:55.858][I] [STATE] BOUND ---> CONNECTING
[2020-03-25 12:17:55.859][D] Validating channel 'readout[0]'... VALID
[2020-03-25 12:17:55.859][D] Created socket stf_builder-0.readout[0].pull
[2020-03-25 12:17:55.859][D] Attached channel readout[0] to ipc:///tmp/readout-pipe-0 (connect) (pull)
[2020-03-25 12:17:55.859][I] [STATE] CONNECTING ---> DEVICE READY
[2020-03-25 12:17:55.859][I] [STATE] DEVICE READY ---> INITIALIZING TASK
[2020-03-25 12:17:55.859][I] Not imposing limits on number of buffered SubTimeFrames. Possibility of creating back-pressure.
[2020-03-25 12:17:55.859][I] (Sub)TimeFrame file sink is disabled.
[2020-03-25 12:17:55.859][I] (Sub)TimeFrame file source is disabled.
[2020-03-25 12:17:55.859][I] DPL Channel name: dpl-chan
[2020-03-25 12:17:55.859][I] Sending data to channel: dpl-chan[0]
[2020-03-25 12:17:55.859][I] [STATE] INITIALIZING TASK ---> READY
[2020-03-25 12:17:55.859][I] [STATE] READY ---> RUNNING
[2020-03-25 12:17:55.860][I] DEVICE: Running...
Following control commands are available:

 [h] help, [c] check current device state,
 [i] init device, [b] bind, [x] connect, [j] init task, [r] run, [s] stop,
 [t] reset task, [d] reset device, [q] end,
 [k] increase log severity, [l] decrease log severity, [n] increase log verbosity, [m] decrease log verbosity

[2020-03-25 12:17:55.860][I] StfOutputThread: sending data to channel: dpl-chan[0]
[2020-03-25 12:17:55.860][D] [stfb-main]region counter found, with value of 1. incrementing.
[2020-03-25 12:17:55.860][D] [stfb-main]incremented region counter, now: 2
[2020-03-25 12:17:55.860][D] [stfb-main]shmem: initialized region queue: fmq_13488916_rgq_2
[2020-03-25 12:17:55.860][D] [stfb-main]shmem: initialized region: fmq_13488916_rg_2
[2020-03-25 12:17:55.915][I] PreRun() done... 
[2020-03-25 12:17:55.915][D] shmem: initialized region queue: fmq_13488916_rgq_1
[2020-03-25 12:17:55.915][D] shmem: initialized region: fmq_13488916_rg_1

 *** Break *** segmentation violation
[2020-03-25 12:18:18.200][I] dpl-chan[0]: in: 0 (0 MB) out: 0 (0 MB)
[2020-03-25 12:18:18.200][I]  readout[0]: in: 1.92488 (8.07357 MB) out: 0 (0 MB)



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================

Thread 15 (Thread 0x7fe45bfff700 (LWP 9473)):
#0  0x00007fe51801cde2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe5173f6b4a in __gthread_cond_timedwait (__abs_timeout=0x7fe45bff9110, __mutex=<optimized out>, __cond=0x7fe444806a20) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:871
#2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7fe444806a20) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:166
#3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7fe444806a20) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:106
#4  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x7fe444806a20) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:138
#5  fair::mq::shmem::Region::SendAcks (this=0x7fe444806930) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/shmem/Region.cxx:140
#6  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x7fe444806b70) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#7  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fe468f78700 (LWP 9472)):
#0  0x00007fe517798c2d in poll () from /lib64/libc.so.6
#1  0x00007fe5143d887d in zmq::signaler_t::wait(int) () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#2  0x00007fe5143bcfdc in zmq::mailbox_t::recv(zmq::command_t*, int) () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#3  0x00007fe5143da242 in zmq::socket_base_t::process_commands(int, bool) () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#4  0x00007fe5143db041 in zmq::socket_base_t::recv(zmq::msg_t*, int) () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#5  0x00007fe5143fb3c9 in s_recvmsg(zmq::socket_base_t*, zmq_msg_t*, int) () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#6  0x00007fe5173d61a8 in fair::mq::shmem::Socket::Receive (this=0x268da80, msgVec=..., timeout=-1) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/shmem/Socket.cxx:287
#7  0x0000000000439a29 in FairMQChannel::Receive (rcvTimeoutInMs=<optimized out>, msgVec=..., this=<optimized out>) at /opt/alisw/el7/FairMQ/v1.4.12-14/include/fairmq/FairMQChannel.h:289
#8  o2::DataDistribution::StfInputInterface::DataHandlerThread (this=0x267eba0, pInputChannelIdx=0) at /home/fitdaq/work/DataDistribution/src/StfBuilder/StfBuilderInput.cxx:100
#9  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x2698350) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#10 0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fe469779700 (LWP 9471)):
#0  0x00007fe51776a4a9 in waitpid () from /lib64/libc.so.6
#1  0x00007fe5176e7f52 in do_system () from /lib64/libc.so.6
#2  0x00007fe5176e8301 in system () from /lib64/libc.so.6
#3  0x00007fe515341fad in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x10048e0) at /build/workarea/sw/20156108/1/SOURCES/ROOT/v6-18-04-alice1/v6-18-04-alice1/core/unix/src/TUnixSystem.cxx:2106
#4  TUnixSystem::StackTrace (this=0x10048e0) at /build/workarea/sw/20156108/1/SOURCES/ROOT/v6-18-04-alice1/v6-18-04-alice1/core/unix/src/TUnixSystem.cxx:2400
#5  0x00007fe515344844 in TUnixSystem::DispatchSignals (this=0x10048e0, sig=kSigSegmentationViolation) at /build/workarea/sw/20156108/1/SOURCES/ROOT/v6-18-04-alice1/v6-18-04-alice1/core/unix/src/TUnixSystem.cxx:3631
#6  <signal handler called>
#7  o2::DataDistribution::ReadoutDataUtils::getSubSpecificationComponents(char const*, unsigned long) (pRdhData=0x2aea38200000 <error: Cannot access memory at address 0x2aea38200000>, len=len
entry=16384) at /home/fitdaq/work/DataDistribution/src/common/ReadoutDataModel.cxx:43
#8  0x0000000000438d7c in o2::DataDistribution::StfInputInterface::StfBuilderThread (this=0x267eba0, pIdx=0) at /home/fitdaq/work/DataDistribution/src/StfBuilder/StfBuilderInput.cxx:272
#9  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x268db50) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#10 0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fe469f7a700 (LWP 9470)):
#0  0x00007fe51801cde2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe5173f6091 in boost::interprocess::ipcdetail::posix_condition::do_timed_wait (mut=..., abs_time=<synthetic pointer>..., this=0x7fe4ec68e058) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/interprocess/sync/posix/condition.hpp:184
#2  boost::interprocess::ipcdetail::posix_condition::timed_wait<boost::interprocess::ipcdetail::internal_mutex_lock<boost::interprocess::scoped_lock<boost::interprocess::interprocess_mutex> > > (abs_time=<synthetic pointer>..., lock=<synthetic pointer>..., this=0x7fe4ec68e058) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/interprocess/sync/posix/condition.hpp:97
#3  boost::interprocess::interprocess_condition::timed_wait<boost::interprocess::scoped_lock<boost::interprocess::interprocess_mutex> > (abs_time=<synthetic pointer>..., lock=..., this=0x7fe4ec68e058) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/interprocess/sync/interprocess_condition.hpp:128
#4  boost::interprocess::message_queue_t<boost::interprocess::offset_ptr<void, long, unsigned long, 0ul> >::do_receive (abs_time=<synthetic pointer>..., priority=<synthetic pointer>: <optimized out>, recvd_size=<synthetic pointer>: <optimized out>, buffer_size=<optimized out>, buffer=<optimized out>, block=boost::interprocess::message_queue_t<boost::interprocess::offset_ptr<void, long, unsigned long, 0> >::timed, this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/interprocess/ipc/message_queue.hpp:887
#5  boost::interprocess::message_queue_t<boost::interprocess::offset_ptr<void, long, unsigned long, 0ul> >::timed_receive (abs_time=<synthetic pointer>..., priority=<synthetic pointer>: <optimized out>, recvd_size=<synthetic pointer>: <optimized out>, buffer_size=<optimized out>, buffer=<optimized out>, this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/interprocess/ipc/message_queue.hpp:841
#6  fair::mq::shmem::Region::ReceiveAcks (this=0x26939f0) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/shmem/Region.cxx:101
#7  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x2693bd0) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#8  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fe46a77b700 (LWP 9469)):
#0  0x00007fe51801cde2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe5173f6b4a in __gthread_cond_timedwait (__abs_timeout=0x7fe46a775110, __mutex=<optimized out>, __cond=0x2693ae0) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:871
#2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x2693ae0) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:166
#3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x2693ae0) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:106
#4  std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (__rtime=..., __lock=..., this=0x2693ae0) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:138
#5  fair::mq::shmem::Region::SendAcks (this=0x26939f0) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/shmem/Region.cxx:140
#6  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x2693ba0) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#7  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fe4ed495700 (LWP 9468)):
#0  0x00007fe51801ca35 in pthread_cond_wait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe517d4632c in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond
entry=0x2681628) at /build/workarea/sw/20156094/1/BUILD/97567abeeaabca84072531b8d220e14e9fdb3677/GCC-Toolchain/build-gcc/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=this
entry=0x2681628, __lock=...) at ../../../../../gcc/libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00000000004214ea in pop (d=<synthetic pointer>..., this=0x2681278) at /home/fitdaq/work/DataDistribution/src/common/include/ConcurrentQueue.h:90
#4  dequeue (pStage=1, this=0x267ead8) at /home/fitdaq/work/DataDistribution/src/common/include/ConcurrentQueue.h:243
#5  StfOutputThread (this=0x267e790) at /home/fitdaq/work/DataDistribution/src/StfBuilder/StfBuilderDevice.cxx:263
#6  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x268b010) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#7  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fe4edc96700 (LWP 9467)):
#0  0x00007fe51801cde2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe51689cb19 in __gthread_cond_timedwait (__abs_timeout=0x7fe4edc8fd00, __mutex=<optimized out>, __cond=0x26803b8) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:871
#2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:166
#3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:106
#4  std::condition_variable::wait_until<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> >, fair::mq::StateMachine::WaitForPendingStateFor(int) const::<lambda()> > (__p=..., __atime=..., __lock=..., this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:129
#5  std::condition_variable::wait_for<long int, std::ratio<1, 1000>, fair::mq::StateMachine::WaitForPendingStateFor(int) const::<lambda()> > (__p=..., __rtime=..., __lock=..., this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/condition_variable:145
#6  fair::mq::StateMachine::WaitForPendingStateFor (this=this
entry=0x267e850, durationInMs=durationInMs
entry=1000) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/StateMachine.cxx:366
#7  0x00007fe517323f75 in FairMQDevice::WaitFor<long, std::ratio<1l, 1l> > (duration=..., this=0x267e790) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/FairMQDevice.h:409
#8  FairMQDevice::LogSocketRates (this=0x267e790) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/FairMQDevice.cxx:811
#9  0x00007fe5173254dd in std::__invoke_impl<void, void (FairMQDevice::*)(), FairMQDevice*> (__t=<optimized out>, __f=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/bits/invoke.h:73
#10 std::__invoke<void (FairMQDevice::*)(), FairMQDevice*> (__fn=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/bits/invoke.h:95
#11 std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >::_M_invoke<0ul, 1ul> (this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/thread:234
#12 std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >::operator() (this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/thread:243
#13 std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::operator() (this=0x7fe4edc90630) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/future:1362
#14 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void> >::_M_invoke(std::_Any_data const&) (__functor=...) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/bits/std_function.h:302
#15 0x00007fe5173258fb in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/bits/std_function.h:706
#16 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (this=0x2696670, __f=<optimized out>, __did_set=0x7fe4edc905d7) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/future:561
#17 0x00007fe51801720b in __pthread_once_slow () from /lib64/libpthread.so.0
#18 0x00007fe517327b48 in __gthread_once (__func=<optimized out>, __once=0x2696688) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:699
#19 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__f=
0x7fe4edc90600: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x7fe5173258e0 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __once=...) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/mutex:684
#20 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (__ignore_failure=false, __res=..., this=0x2696670) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/future:401
#21 std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >&&)::{lambda()#1}::operator()() const (__closure=0x268da38) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/future:1667
#22 std::__invoke_impl<void, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >&&)::{lambda()#1}>(std::__invoke_other, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >&&)::{lambda()#1}&&) (__f=...) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/bits/invoke.h:60
#23 std::__invoke<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >&&)::{lambda()#1}>(std::__invoke_result&&, (std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >&&)::{lambda()#1}&&)...) (__fn=...) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/bits/invoke.h:95
#24 std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x268da38) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/thread:234
#25 std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >&&)::{lambda()#1}> >::operator()() (this=0x268da38) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/thread:243
#26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<void (FairMQDevice::*)(), FairMQDevice*> >&&)::{lambda()#1}> > >::_M_run() (this=0x268da30) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/thread:186
#27 0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x268da30) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#28 0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#29 0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fe4ee497700 (LWP 9466)):
#0  0x00007fe5177a3ea3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe5143b7b0c in zmq::epoll_t::loop() () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#2  0x00007fe5143ef0b8 in thread_routine () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#3  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fe4eec98700 (LWP 9465)):
#0  0x00007fe5177a3ea3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fe5143b7b0c in zmq::epoll_t::loop() () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#2  0x00007fe5143ef0b8 in thread_routine () from /opt/alisw/el7/ZeroMQ/v4.3.2-2/lib/libzmq.so.5
#3  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fe4ef499700 (LWP 9464)):
#0  0x00007fe51801fe9d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fe5173d9a3c in std::this_thread::sleep_for<long, std::ratio<1l, 1000l> > (__rtime=...) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/thread:376
#2  fair::mq::shmem::TransportFactory::SendHeartbeats (this=0x268e580) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/shmem/TransportFactory.cxx:110
#3  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x26934d0) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fe4efc9a700 (LWP 9459)):
#0  0x00007fe51801fe9d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fe5173bd46f in std::this_thread::sleep_for<long, std::ratio<1l, 1000l> > (__rtime=...) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/thread:376
#2  fair::mq::plugins::Control::SignalHandler (this=0x267ee20) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/plugins/Control.cxx:390
#3  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x267f7f0) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fe4f049b700 (LWP 9458)):
#0  0x00007fe517798c2d in poll () from /lib64/libc.so.6
#1  0x00007fe5173bc572 in fair::mq::plugins::Control::InteractiveMode (this=0x267ee20) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/plugins/Control.cxx:165
#2  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x267f900) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fe4f0c9c700 (LWP 9457)):
#0  0x00007fe51801cde2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000046cde3 in spdlog::details::thread_pool::process_next_msg_() ()
#2  0x000000000046d108 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<spdlog::details::thread_pool::thread_pool(unsigned long, unsigned long, std::function<void ()>)::{lambda()#1}> > >::_M_run() ()
#3  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x25b4e10) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fe4f215e700 (LWP 9456)):
#0  0x00007fe51801cde2 in pthread_cond_timedwait

GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000046cde3 in spdlog::details::thread_pool::process_next_msg_() ()
#2  0x000000000046d108 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<spdlog::details::thread_pool::thread_pool(unsigned long, unsigned long, std::function<void ()>)::{lambda()#1}> > >::_M_run() ()
#3  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x25b4b30) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe5177a38cd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fe5183e4fc0 (LWP 9394)):
#0  0x00007fe51801fe9d in nanosleep () from /lib64/libpthread.so.0
#1  0x000000000041d8a5 in sleep_for (__rtime=...) at /opt/alisw/el7/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/thread:376
#2  o2::DataDistribution::StfBuilderDevice::ConditionalRun() (this=<optimized out>) at /home/fitdaq/work/DataDistribution/src/StfBuilder/StfBuilderDevice.cxx:407
#3  0x00007fe517320af5 in FairMQDevice::RunWrapper (this=0x267e790) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/FairMQDevice.cxx:507
#4  0x00007fe517323810 in FairMQDevice::<lambda(fair::mq::State)>::operator()(fair::mq::State) const (__closure=0x2681130, state=fair::mq::State::Running) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/FairMQDevice.cxx:160
#5  0x00007fe5168a0fcd in std::function<void (fair::mq::State)>::operator()(fair::mq::State) const (__args#0=<optimized out>, this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/GCC-Toolchain/v7.3.0-alice1-15/include/c++/7.3.0/bits/std_function.h:706
#6  boost::detail::function::void_function_obj_invoker1<std::function<void (fair::mq::State)>, void, fair::mq::State>::invoke(boost::detail::function::function_buffer&, fair::mq::State) (function_obj_ptr=..., a0=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/function/function_template.hpp:158
#7  0x00007fe5168a9653 in boost::function1<void, fair::mq::State>::operator() (a0=<optimized out>, this=<optimized out>) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/function/function_template.hpp:763
#8  boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (fair::mq::State)>, 0u, fair::mq::State&>(boost::function<void (fair::mq::State)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple<fair::mq::State&> const&, boost::enable_if<boost::is_void<boost::function<void (fair::mq::State)>::result_type>, void>::type*) const (this=<optimized out>, args=..., func=...) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#9  boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::operator()<boost::function<void (fair::mq::State)>, fair::mq::State&, 1ul>(boost::function<void (fair::mq::State)>&, std::tuple<fair::mq::State&> const&, mpl_::size_t<1ul>) const (this=<optimized out>, args=..., func=...) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#10 boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > const&) const (connectionBody=..., this=0x7ffc08eea480) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/signals2/detail/variadic_slot_invoker.hpp:133
#11 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >::dereference() const (this=0x7ffc08eea290) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/signals2/detail/slot_call_iterator.hpp:110
#12 boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > const&) (f=...) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/iterator/iterator_facade.hpp:550
#13 boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type const&, long, false, false>::operator*() const (this=0x7ffc08eea290) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/iterator/iterator_facade.hpp:656
#14 boost::signals2::optional_last_value<void>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >) const (this=<optimized out>, first=..., last=...) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/signals2/optional_last_value.hpp:57
#15 boost::signals2::detail::combiner_invoker<void>::operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value<void>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, fair::mq::State>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (fair::mq::State), boost::function<void (fair::mq::State)> >, boost::signals2::mutex> >) const (this=<optimized out>, combiner=..., first=..., last=...) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/signals2/detail/result_type_wrapper.hpp:64
#16 boost::signals2::detail::signal_impl<void (fair::mq::State), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (fair::mq::State)>, boost::function<void (boost::signals2::connection const&, fair::mq::State)>, boost::signals2::mutex>::operator()(fair::mq::State) (this=0x2680630, args#0=<optimized out>, args#0
entry=fair::mq::State::Running) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/signals2/detail/signal_template.hpp:243
#17 0x00007fe5168b0667 in boost::signals2::signal<void (fair::mq::State), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (fair::mq::State)>, boost::function<void (boost::signals2::connection const&, fair::mq::State)>, boost::signals2::mutex>::operator()(fair::mq::State) const (args#0=fair::mq::State::Running, this=0x2680400) at /build/workarea/sw/20156105/1/slc7_x86-64/boost/v1.70.0-41/include/boost/signals2/detail/signal_template.hpp:726
#18 fair::mq::fsm::Machine_::CallStateHandler (state=fair::mq::State::Running, this=0x2680370) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/StateMachine.cxx:158
#19 fair::mq::fsm::Machine_::ProcessWork (this=0x2680370) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/StateMachine.cxx:206
#20 0x00007fe51689f6ad in fair::mq::StateMachine::ProcessWork (this=0x267e850) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/StateMachine.cxx:375
#21 0x00007fe5172ec040 in FairMQDevice::RunStateMachine (this=<optimized out>) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/FairMQDevice.h:400
#22 fair::mq::DeviceRunner::Run (this=this
entry=0x7ffc08eeb8a0) at /build/workarea/sw/20156105/1/SOURCES/FairMQ/v1.4.12/v1.4.12/fairmq/DeviceRunner.cxx:174
#23 0x000000000041931d in main (argc=<optimized out>, argv=<optimized out>) at /opt/alisw/el7/FairMQ/v1.4.12-14/include/fairmq/runFairMQDevice.h:51
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#7  o2::DataDistribution::ReadoutDataUtils::getSubSpecificationComponents(char const*, unsigned long) (pRdhData=0x2aea38200000 <error: Cannot access memory at address 0x2aea38200000>, len=len
entry=16384) at /home/fitdaq/work/DataDistribution/src/common/ReadoutDataModel.cxx:43
#8  0x0000000000438d7c in o2::DataDistribution::StfInputInterface::StfBuilderThread (this=0x267eba0, pIdx=0) at /home/fitdaq/work/DataDistribution/src/StfBuilder/StfBuilderInput.cxx:272
#9  0x00007fe517d4bf3f in std::execute_native_thread_routine (__p=0x268db50) at ../../../../../gcc/libstdc++-v3/src/c++11/thread.cc:83
#10 0x00007fe518018ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fe5177a38cd in clone () from /lib64/libc.so.6

===========================================================


[12:18:18][DEBUG] send ack worker for fmq_13488916_rg_1 leaving.
[12:18:18][DEBUG] shmem: region queue 'fmq_13488916_rgq_1' is remote, no cleanup necessary
[12:18:19][DEBUG] send ack worker for fmq_13488916_rg_2 leaving.
[12:18:19][DEBUG] receive ack worker for fmq_13488916_rg_2 leaving.
[12:18:19][INFO] dpl-chan[0]: in: 0 (0 MB) out: 0 (0 MB)
[12:18:19][DEBUG] shmem: destroyed region fmq_13488916_rg_2
[12:18:19][INFO]  readout[0]: in: 0 (0 MB) out: 0 (0 MB)
[12:18:19][DEBUG] shmem: removed region queue fmq_13488916_rgq_2
[12:18:19][DEBUG] removed RTDB container factory FairBaseContFact