Problem with running readout.exe

shyuan · September 13, 2019, 1:19pm

Dear O2 experts,

We are trying to run data taking with our setup here.

We are using readout v1.0.6.

When running readout.exe file://readout.cfg, the program stuck and there’s no output.

before running readout.exe, we ran sudo mount -t tmpfs -o size=4G tmpfs /tmp/ramdisk, and mkfifo data.raw under ramdisk.

the readout config file we are using is attached below.

please point us to the solution to our problem.

Many thanks,
Shiming

[readout]
rate=-1
exitTimeout=-1
flushEquipmentTimeout=1
disableAggregatorSlicing=1
#logbookEnabled=
#logbookUrl=
#logbookApiToken=
#logbookUpdateInterval=

[bank-test1]
enabled=1
size=1G
type=MemoryMappedFile
numaNode=-1

[equipment-rorc-1]
enabled=1
equipmentType=rorc
name=CRU_UiB_SETUP
stopOnError=1
#id=3
#idleSleepTime=200
#outputFifoSize=1000
memoryBankName=bank-test1
memoryPoolPageSize=1024k
memoryPoolNumberOfPages=2000
#disableOutput=0
#firstPageOffset=
#blockAlign=
cardId=21:00.0
#channelNumber=0
generatorEnabled=0
#generatorDataSize=8192
#generatorLoopback=INTERNAL
#generatorPattern=INCREMENTAL
#generatorRandomSizeEnabled=0
linkMask=0
#resetLevel=INTERNAL
rdhCheckEnabled=0
rdhDumpEnabled=0
cleanPageBeforeUse=1

[consumer-stats]
enabled=1
consumerType=stats
monitoringEnabled=0
monitoringUpdatePeriod=5
processMonitoringInterval=5
monitoringURI=infologger://

[consumer-rec]
enabled=1
consumerType=fileRecorder
#filesMax=1
fileName=/tmp/ramdisk/data.raw
#bytesMax=0
#dataBlockHeaderEnabled=0

[consumer-processor]
enabled=0
threadInputFifoSize=400
#threadIdleSleepTime=1000
consumerOutput=consumer-rec-lz4
consumerType=processor
numberOfThreads=4
ensurePageOrder=1
libraryPath=/home/shiming/alice/sw/slc7_x86-64/Readout/4a5cf20748-1/lib/libProcessorLZ4Compress.so

[consumer-rec-lz4]
enabled=0
consumerType=fileRecorder
filesMax=1
fileName=/tmp/ramdisk_2/data.lz4

sy-c · September 16, 2019, 7:02am

Hello,
if you have installed the machine with the flp-suite, it is running the infoLogger.
All messages of readout are redirected there.

You can launch the infoBrowser to see live and archived logs:
/opt/o2-InfoLogger/bin/infoBrowser

If you prefer to have the messages in the console where readout is launched, you may set an
environment variable as follows before starting readout.exe:
export INFOLOGGER_MODE=stdout

cheers,
sylvain

shyuan · September 16, 2019, 9:26am

Hi Sylvain,
Thanks for the reply, I added export INFOLOGGER_MODE-stdout in my bashrc file.

Now readout.exe stuck at opening the rec file.

bash-4.2$ readout.exe file://home/shiming/work/rsync/CRU_ITS/software/config/readout_uib.cfg
2019-09-16 11:16:45.412507     Readout process starting, pid 388614
2019-09-16 11:16:45.412611     Optional built features enabled:
2019-09-16 11:16:45.412618     FAIRMQ : yes
2019-09-16 11:16:45.412674     NUMA : yes
2019-09-16 11:16:45.412681     RDMA : no
2019-09-16 11:16:45.412686     OCC : yes
2019-09-16 11:16:45.412691     LOGBOOK : no
2019-09-16 11:16:45.412708     Readout entering standalone state machine
2019-09-16 11:16:45.412716     Readout executing CONFIGURE
2019-09-16 11:16:45.412723     Reading configuration from file://home/shiming/work/rsync/CRU_ITS/software/config/readout_uib.cfg
2019-09-16 11:16:45.413072     Merging selected content of OCC configuration
2019-09-16 11:16:45.413085     No OCC FMQ channels configuration found
2019-09-16 11:16:45.413092     Disabling CPU deep sleep for process
2019-09-16 11:16:45.413104     Error opening /dev/cpu_dma_latency
2019-09-16 11:16:45.413331     Enforcing memory allocated on NUMA node 0
2019-09-16 11:16:45.413341     Creating memory bank bank-test1: type MemoryMappedFile size 1073741824
2019-09-16 11:16:45.413354     Creating shared memory block for bank bank-test1 : size 1073741824 using /var/lib/hugetlbfs/global/pagesize-1GB/readout-bank-test1
2019-09-16 11:16:45.414376     Shared memory block for bank bank-test1 is ready
2019-09-16 11:16:45.577450     Bank bank-test1 added
2019-09-16 11:16:45.577494     Releasing memory NUMA node enforcment
2019-09-16 11:16:45.577541     Configuring consumer consumer-rec: fileRecorder
2019-09-16 11:16:45.577555     Recording path = /tmp/ramdisk/data.raw
2019-09-16 11:16:45.577581     Recording internal data block headers = 0
2019-09-16 11:16:45.577596     Opening file for writing: /tmp/ramdisk/data.raw

I am not familiar with the correct settings of the config file, please point us if there’s anything that should be modified?

###################################
# readout configuration file
#
# comments lines start with #
# inline comments ARE NOT ALLOWED!
# section names are in brackets []
# settings are defined with key=value pairs
# 
# Configuration for UiB CRU FLP
#
# documentation:
# https://github.com/AliceO2Group/Readout/blob/master/doc/configurationParameters.md
# 
################################################

[readout]
rate=-1
exitTimeout=-1
flushEquipmentTimeout=1
disableAggregatorSlicing=1
#logbookEnabled=
#logbookUrl=
#logbookApiToken=
#logbookUpdateInterval=

[bank-test1]
enabled=1
size=1G
type=MemoryMappedFile
numaNode=0

[equipment-rorc-1]
enabled=1
equipmentType=rorc
name=CRU_UiB_SETUP
stopOnError=1
#id=3
#idleSleepTime=200
#outputFifoSize=1000
memoryBankName=bank-test1
memoryPoolPageSize=1024k
memoryPoolNumberOfPages=2000
#disableOutput=0
#firstPageOffset=
#blockAlign=
cardId=21:00.0
#channelNumber=0
generatorEnabled=0
#generatorDataSize=8192
#generatorLoopback=INTERNAL
#generatorPattern=INCREMENTAL
#generatorRandomSizeEnabled=0
linkMask=0
#resetLevel=INTERNAL
rdhCheckEnabled=0 # Why off?
rdhDumpEnabled=0
cleanPageBeforeUse=1

[consumer-stats]
enabled=0
consumerType=stats
monitoringEnabled=0
monitoringUpdatePeriod=5
processMonitoringInterval=5
monitoringURI=infologger://

[consumer-rec]
enabled=1
consumerType=fileRecorder
#filesMax=1
fileName=/tmp/ramdisk/data.raw
#bytesMax=0
#dataBlockHeaderEnabled=0

[consumer-processor]
enabled=0
threadInputFifoSize=400
#threadIdleSleepTime=1000
consumerOutput=consumer-rec-lz4
consumerType=processor
numberOfThreads=4
ensurePageOrder=1
libraryPath=/opt/alisw/el7/Readout/v1.0.6-1/lib/libProcessorLZ4Compress.so

[consumer-rec-lz4]
enabled=0
consumerType=fileRecorder
filesMax=1
fileName=/tmp/ramdisk_2/data.lz4

sy-c · September 16, 2019, 9:38am

why do you think readout is stuck?
what are the logs after you stop readout by pressing CTRL+C ?

shyuan · September 16, 2019, 9:46am

there seems to be an error in memory allocation.
I allocated 1GB for bank-test1 in readout_uib.cfg section [readout], and before running readout.exe, I mounted /tmp/ramdisk with sudo mount -t tmpfs -o size=4G tmpfs /tmp/ramdisk, and mkfifo data.raw under /tmp/ramdisk.

after stopping readout by CTRL+C, the output is:

^C2019-09-16 11:34:32.718940     Received signal 2
*** break ***
2019-09-16 11:34:32.719119 !!! Error - Failed to create file: Interrupted system call
2019-09-16 11:34:32.719138  !  Warning - Recording disabled
2019-09-16 11:34:32.719194     Configuring equipment equipment-rorc-1: rorc
2019-09-16 11:34:32.719417     Equipment CRU_UiB_SETUP: from config [equipment-rorc-1], max rate=-1.000000 Hz, idleSleepTime=200 us, outputFifoSize=2000
2019-09-16 11:34:32.719430     Equipment CRU_UiB_SETUP: requesting memory pool 2000 pages x 1048576 bytes from bank 'bank-test1', block aligned @ 0x200000, 1st page offset @ 0x0
2019-09-16 11:34:32.719449     pageSpaceReserved = 56, aligning 1st page @ 0xFFFC8
2019-09-16 11:34:32.719460 !!! Error - Not enough space left in memory bank 'bank-test1' (need 1023410176 bytes more)
2019-09-16 11:34:32.719492 !!! Error - Failed to create pool of memory pages
2019-09-16 11:34:32.719558 !!! Error - Failed to configure equipment equipment-rorc-1 : error #176
2019-09-16 11:34:32.719569 !!! Fatal - Some equipments failed to initialize, exiting

sy-c · September 16, 2019, 9:56am

Please try to run first without the data recording, and send full log.

shyuan · September 16, 2019, 10:04am

Ok, we disabled [consumer-rec], and readout.exe log is

bash-4.2$ readout.exe file://home/shiming/work/rsync/CRU_ITS/software/config/readout_uib.cfg
2019-09-16 11:51:52.030469     Readout process starting, pid 6984
2019-09-16 11:51:52.030629     Optional built features enabled:
2019-09-16 11:51:52.030635     FAIRMQ : yes
2019-09-16 11:51:52.030679     NUMA : yes
2019-09-16 11:51:52.030686     RDMA : no
2019-09-16 11:51:52.030692     OCC : yes
2019-09-16 11:51:52.030697     LOGBOOK : no
2019-09-16 11:51:52.030705     Readout entering standalone state machine
2019-09-16 11:51:52.030712     Readout executing CONFIGURE
2019-09-16 11:51:52.030719     Reading configuration from file://home/shiming/work/rsync/CRU_ITS/software/config/readout_uib.cfg
2019-09-16 11:51:52.031085     Merging selected content of OCC configuration
2019-09-16 11:51:52.031099     No OCC FMQ channels configuration found
2019-09-16 11:51:52.031104     Disabling CPU deep sleep for process
2019-09-16 11:51:52.031273     Error opening /dev/cpu_dma_latency
2019-09-16 11:51:52.031509     Creating memory bank bank-test1: type MemoryMappedFile size 1073741824
2019-09-16 11:51:52.031530     Creating shared memory block for bank bank-test1 : size 1073741824 using /var/lib/hugetlbfs/global/pagesize-1GB/readout-bank-test1
2019-09-16 11:51:52.032207     Shared memory block for bank bank-test1 is ready
2019-09-16 11:51:52.199584     Bank bank-test1 added
2019-09-16 11:51:52.199641     Configuring equipment equipment-rorc-1: rorc
2019-09-16 11:51:52.199732     Equipment CRU_UiB_SETUP: from config [equipment-rorc-1], max rate=-1.000000 Hz, idleSleepTime=200 us, outputFifoSize=2000
2019-09-16 11:51:52.199740     Equipment CRU_UiB_SETUP: requesting memory pool 2000 pages x 1048576 bytes from bank 'bank-test1', block aligned @ 0x200000, 1st page offset @ 0x0
2019-09-16 11:51:52.199750     pageSpaceReserved = 56, aligning 1st page @ 0xFFFC8
2019-09-16 11:51:52.199759 !!! Error - Not enough space left in memory bank 'bank-test1' (need 1023410176 bytes more)
2019-09-16 11:51:52.199774 !!! Error - Failed to create pool of memory pages
2019-09-16 11:51:52.199800 !!! Error - Failed to configure equipment equipment-rorc-1 : error #176
2019-09-16 11:51:52.199807 !!! Fatal - Some equipments failed to initialize, exiting

sy-c · September 16, 2019, 12:40pm

Hello,

You are trying to allocate 2000x1MB pages in a bank of size 1GB, this does not fit.
You should increase the bank size (e.g. [bank-test1] size=2G) or reduce the number of pages (e.g. [equipment-rorc-1] memoryPoolNumberOfPages=1000)

I would also advise to start from the provided confguration file installed by ansible usually in /home/flp/readout.cfg

cheers,
Sylvain

shyuan · September 16, 2019, 2:40pm

Thanks Sylvain.
The problem with memory allocation seems to have been resolved.