TPC: make IDC/CMV distribute a pure forwarding device + minor improvements

[ehellbar]

---

Stack created with Sapling. Best reviewed with ReviewStack.

• -> TPC: make IDC/CMV distribute a pure forwarding device + minor improvements #15583 (3 commits)
• DPL: add allocator to forward a fair::mq::Message payload by shallow copy #15577 (2 commits)

[ehellbar]

Hello @tubagundem @matthias-kleiner ,

here, there are three things:

1. replacing the two-copy forwarding in the IDC and CMV distribute devices with a shallow(=zero)-copy forwarding

• this requires DPL: add allocator to forward a fair::mq::Message payload by shallow copy #15577, so we’ll have to wait for this one to be reviewed and merged (Giulio is on vacation for 2 weeks)
• the two Framework commits in here are the same as in the prerequisite PR and will be reviewed and merged there. They simply made it here, too, because they were in my same sapling stack. So ignore the Framework part for your review. Once the other PR is merged, a rebase will clean this one up.

2. adjust some of the log message severities according to my experience debugging this at P2, adding some new logs and changing some of the existing log strings, especially matching the printed TF counters in the log messages to the actual ones from DPL

3. adding a CMV test generator which produces dummy data, and provides options to delay or drop data, analogous to the IDC test generator

I tested everything locally with CMVs, but not with IDCs. In any case, @tubagundem @matthias-kleiner please review the changes and test it yourself locally as well to confirm that the output is consistent. I would also like to test it at P2 with replay before merging it, which I still have to do.

Now, this makes distribute a pure forwarding device, so no time is spent there anymore. Not sure if it solves the problems that we observed at P2 during PbPb. It could, since until now, we basically delayed the forwarding by making two copies, so with this implementation it will be faster for sure.

I can imagine, from experiences in my local tests, that we still might run into some issues if we have slow or slightly desynchronized message forwarding (for whatever reason, we observed something like this this year at the beginning of each run for DataDistribution devices) via the direct proxies between the FLPs and the EPN, for which I’ll suggest a solution below.

So, here’s a few points which I would consider to work on to improve distribute for Run 4. Feel free to ignore them, of course, if you disagree or if I have not really thought them through

• since the IDC and CMV distribute do the same thing, with some extra stuff for the CMV case, you could consider to merge the two devices into one and take care of the differences via a command line option e.g. --runcmv, similar as to what is done for the ITS/MFT stf-decoder. I have not evaluated if this is feasible, but for sure it is worth to check
• one minor thing that I have never understood is why you play around with the cru sub spec, starting with cru << 7 and then switching to cru at some point in the chain. Why not send the data from the beginning with sub spec cru?
• I assume with the zero-copy forwarding, the option to have multiple distribute lanes (splitting the CRUs among multiple instances) becomes useless and can be removed? Anyway, you can define the CRUs via the --crus parameter. If at all, I would try to think of a way to make it compatible with the default DPL pipelining, where the TFs are sent round-robin between the different instances. But also this is non-trivial with the special case of n buffered TFs from 144 channels which are sent sporadically. Again, not fully thought through
• revisit missing data check logic
  • in my local tests, I encountered the scenario where the current TF was in the beginning of the second buffers and the if (!(mProcessedTotalData++ % mCheckEveryNData)) check fired, so the first buffer was closed early while many data from the first buffer were still arriving. I don’t think this should happen. I would close the first buffer only if we are at the end of the second one, or when data even newer than the second buffer arrive
  • if the first data newer than the second buffer arrive, you currently skip these new data and finish the first buffer immediately. Instead, you should keep the new data while finishing the first buffer.
• I would make the number of buffers a free parameter. I don’t see a reason why it can only be two. In the end, it depends on the number of aggregate lanes and the compression time of aggregate, so I would tune the number of buffers to that to give the data the maximum time to arrive and not be too aggressive in dropping data due to any disturbance or desynchronization between FLPs and EPN.