I’d need some clarification about how things are supposed to work on FLPs, as my understanding of what’s the plan, what’s there already vs what’s in dev, is bit fuzzy at the moment.
Let me take MCH as a use case, and please correct me when I’m wrong
The readout program somehow pumps data « from CRU memory » and distribute it to some consumers. One of the possible consumer is a FairMQ device. So I assume that device is the first stage of a FLP DPL workflow.
But what kind of messages are delivered by this device ? What time span ? Are those subtimeframes or smaller chunks ? If smaller chunks who’s building the STF ? (In that case the first stage of a subsystem FLP DPL workflow would not be the readout device but the STF builder ?)
For instance, a processing I’d like to do on MCH FLPs is the pre-clustering of MCH digits. That processing will be part of a DPL workflow. But what kind of input should it expect ? STF ? Other thing ?
Can I already fake such a workflow today with existing software or are some pieces missing ?
The STFBuilder is not inside the DPL, nor is the STFSender, but there will be DPL bridges or proxies around (I don’t know what is the correct name). Gvozden is working on it. I will page him to connect to Discourse. He can then tell us more about the output format of the STFBuilder. They should be subtimeframes.
To be complete, note that the Data Sampling can connect to the Readout and feed QC tasks or can connect to any DPL device, i.e. STFBuilder, to feed QC tasks.
I am myself working on a test chain using the current state of the STFBuilder. If I succeed running it, I will let you know so that you can try it out. I don’t think that there is a fake workflow. If we know the format of the STF, we could have a device producing them…
For the record, thanks to recent developments in DPL, most likely the DPL Bridge will not actually be a device, but a special kind of input any given data processor can declare.
You will get whatever the external source (which does not need to be a DPL device, but will have to attach a DataProcessingHeader with the timestamp) provides you. Currently you would have to do this with a RawFairMQDevice. My idea would be to hide the RawFairMQDevice behind a special InputSpec which will tell the DataProcessingDevice to trigger the fetch for you. At the moment this mechanism is what I use for Timers (which do not have a data counterpart) but it’s flexible enough to handle any non-DPL input.
Sorry for the late reply,
Let me try to clarify on a few points:
STF data model: An STF (object) is just a collection of O2 messages, with the standard DataHeader and payload, for the duration of single timeframe.
StfBuilder collects data payloads received from the Readout and tags them with O2 headers. From this point on, everything conforms to the O2 data format. The DataHeader of these payloads uniquely identifies the producer equipment with the tuple <DataDesc, DataOrigin, SubSpecification>. Equipment can be an input link, a user logic in the FPGA, … depends on the detector. In the end, the StfBuilder collects [0…n] O2 messages per equipment per STF duration. The payload of a single o2 message should contain the data of an individual, or several, complete Heartbeat Frames.
A single STF can have payloads belonging to 1 or more data producers.
Now, for building a fake chain: There is already a fake CRU-StfBuilder-DPL chain in [2]. However, the data payload is fake as well. Currently, the StfBuilder publishes fake readout payloads to DPL as <“FLP”, “RAWDATA”> (see [1]).
If you have a real(-istic) payload data of a specific detector, it should not be too hard to inject it at some step.
