Repository navigation
Fix ADC parameter calculation overflow errors - #553
Conversation
|
Thanks Brian. I checked this against current A few requests:
|
4b24a44 to
5e7ec2d
Compare
6ed1cd3 to
39b92f8
Compare
tompollard
left a comment
There was a problem hiding this comment.
Thanks Brian, looks great. Before we merge, could you add a changelog fragment, e.g. changelog.d/adc-float32-overflow.rst? Something like: "Fixed wrsamp failing with an out-of-range error when writing float32 physical signals, by computing ADC parameters in float64."
|
Added the changelog @tompollard , thanks. |
|
Looks good, thanks! |
As discussed in #515, an overflow can occur when calculating ADC parameters from limited-precision physical signal data. The root cause of this overflow is precision loss. When computing min/max values during parameter calculation, insufficient floating-point precision at boundaries (e.g.
int32boundaries) leads to gain/baseline values that cause digital samples to overflow by exactly 1 value.The fix ensures
float64precision is used throughout ADC parameter calculation and conversion:calc_adc_params(): Convert physical signals tofloat64before computing min/maxadc_inplace(): Convert tofloat64before performing gain/baseline arithmeticUsers can now safely provide lower-precision physical data (e.g.,
float32) without validation errors.This PR includes a test for the overflow which uses the data supplied by @Ivorforce in #515. The test checks that:
fmt=32without overflow