Inconsistent TOA output data due to cmake

[andpet1324]

TLDR
The TOA, and consequently, the TOT data is not saved in some instances, where the instance is defined by running make without any changes. Tp triggers, meaning that the TOA trigger works, but data is still not saved. My guess is that the problem lies somewhere in the compiler.

Describe the bug
I'm taking TOA and TOT data on all channels, iterating one at a time. When saving data from link 1, the TOA (and consequently TOT) doesn't get saved in some instances. The instance depends entirely on some process that occurs when running make. If the TOA doesn't get saved, I can run make (without doing ANY changes to pflib), and there's a chance that the TOA will now be saved, but there's also a chance that the effect will be the same.

In detail explanation / to reproduce

First, I run the ROC calibration procedure as explained in ROC-calibration-wiki. NOTE! For inv_vref calibration, I run this app/tool/tasks/inv_vref_scan.cxx, and poke the parameter values produced by the csv. The TOT scan requires this bugfix: TOT-bugfix.

Make sure to SAVE the configurations so that you can set them in another run later!

Another note on the aforementioned calibration. The TOA and TOT procedures requires that TOA and TOT data is saved, which it is not if this bug is in effect. Therefore, you might need to re-run make until you get data. You will know if the data is not being saved in the final steps of the TOT scan: when the TOT trimming parameter is being set on link 1, and all TRIM_TOT values are getting set to 63 (max val), then that is most likely due to this bug. Re-run make and try again.

After successfully calibrating the ROC, I run app/tools/tasks/global_calib_scan.cxx, that scans all channels with charge injections, for later plotting TOA and TOT efficiency curves. It is when doing this scan that the bug is clearly apparent.

In this figure I am showing the results from this scan for all channels on link 0:

We can see that the TOA and TOT triggers nicely. In the expected normal case, the results on link 1 should look almost exactly the same, however when I've scanned with this bug, I get this instead:

where we see that Tp is triggering, but no TOA data, and consequently, no TOT data is saved. ADC data is saved just fine.

Environment:

• ZCU: Lund
• pflib version: v3.9.4

[tomeichlersmith]

I am very doubtful that the lack of TOA/TOT data is related to a bug in the build system because they are so separated. The C++ compilation is deterministic, so - unless you change the code - the compiled program and libraries will be the same. That being said, we are doing some... weird... stuff in order to handle both types of event packets (the constexpr shenanigans and what not), so I'm not totally ruling it out.

My suspicion is that you are leaving pftool in order to re-run make and then re-entering pftool. This "reboot" of pftool resets some aspects of the menu's state which could be undoing some setting that was previously broken. If this hypothesis is correct, you don't need to re-run make at all, you would just exit pftool and re-open it and the problem would go away.

My other suspicion is that some other, lower-level bug is happening that "bubbles up" to trigger this issue. If you don't see TOT/TOA, can you also not see TOT/TOA when running a simple DAQ.CHARGE run? (You'll need to manually ROC.POKE the injection parameters.) What if you don't do "live decoding" and instead write out the binary data? This can separate the decoding as a potential source of the bug.

Having TOT/TOA set to all zero is the behavior we would see if the ROC has its thresholds mis-configured too high. You mention that you do the calibration procedure, but I would encourage you to dump the entire ROC configuration immediately before/after you observe this bug to double check that the ROC has the configuration you expect. (Your comment mentions manually POKEing the parameters, maybe a necessary one is being forgotten?)

The last thought is the ever-present hardware issue, but I don't have any thoughts on that unless we can show that the configuration is not changing and the TOT/TOA values are sometimes present and sometimes not.

[andpet1324]

Thanks for the feedback @tomeichlersmith!

I first considered that the initial setting of pftool was the issue, but after countless iterations (more than 10) of restarting pftool (without running make) and seeing the same behavior, I ruled it out.

It could possibly be a decoding bug. I will give that idea a try!

I have also considered the parameters changing during the scripts. I have tried double-checking pages so that all parameters are correct, and I've included a printout in the script that tells me if the TOA threshold parameters changes in real time, which it doesn't.

On the last paragraph, I do not believe it is a hardware issue. The reason is because we have seen this effect before on other boards when running the TOT alignment. We never seemed to get good values on link 1, which would very well be explained by this bug. I am not ruling out that it is a hardware issue, it just seems unlikely.

[andpet1324]

We've tested more boards, and this issue has only appeared on the board where we found it initially (board 21). It could be that we've just been lucky enough that the bug didn't show up on the new boards, as the bug seems random, but it seems more likely like the bug is connected to board 21.

[andpet1324]

I am closing this issue since we haven't seen this anymore since changing from board 21. When switching back to board 21, everything worked fine. Might have been a connection issue.