Propose pattern for pipelines with multiple/optional correction steps

[SimonHeybrock]

This is a suggestion for a pipeline design pattern, addressing recurring needs for selecting one or more optional correction steps.

The notebook is a draft for gathering feedback on the idea. Will be cleaned up if positive.

See also #192, which I hope is not necessary if the suggested pattern works out.

[jl-wynen]

Nice idea. But this will lead to an exponential growth of functions in the number of corrections. This sounds like a maintenance problem.

We could mitigate this slightly by using something along the lines of

def apply_corrections(corrections: list) -> Callable[...]:
    # just a sketch, doing this requires unpacking the corrections list into separate arguments:
    def apply(raw_data: RawData, *corrections): return ....
    return apply

But this way, it would not be (easily) possible to find the actual function from the graph viz or JSON.

[SimonHeybrock]

Nice idea. But this will lead to an exponential growth of functions in the number of corrections. This sounds like a maintenance problem.

I doubt there can be too many that are all applied at the same stage of a pipeline? What is the largest number you have encountered?

We could mitigate this slightly by using something along the lines of

def apply_corrections(corrections: list) -> Callable[...]:
    # just a sketch, doing this requires unpacking the corrections list into separate arguments:
    def apply(raw_data: RawData, *corrections): return ....
    return apply

But this way, it would not be (easily) possible to find the actual function from the graph viz or JSON.

Not sure I understand how your list approach works.

[jl-wynen]

I doubt there can be too many that are all applied at the same stage of a pipeline? What is the largest number you have encountered?

Just 1 correction with 3 options in powder: the 'run normalisation'. In reflectometry, there are more corrections. @jokasimr How many are there?

We also have the pixel masks with an arbitrary number of masks. Can we also handle them with this approach instead of having a dedicated function that people have to apply to their pipeline?

Not sure I understand how your list approach works.

I was just brainstorming. But essentially:

wf.insert(apply_corrections([apply_a, apply_b]))

where apply_corrections([apply_a, apply_b]) creates and returns a function of the form

def apply(raw_data: RawData, correction_a: CorrectionA, correction_b: CorrectionB) -> CorrectedData:
    return apply_b(apply_a(raw_data, correction_a), correction_b)

[SimonHeybrock]

I doubt there can be too many that are all applied at the same stage of a pipeline? What is the largest number you have encountered?

We also have the pixel masks with an arbitrary number of masks. Can we also handle them with this approach instead of having a dedicated function that people have to apply to their pipeline?

We usually used map/reduce for them? Are there other cases?

Not sure I understand how your list approach works.

I was just brainstorming. But essentially:

wf.insert(apply_corrections([apply_a, apply_b]))

where apply_corrections([apply_a, apply_b]) creates and returns a function of the form

def apply(raw_data: RawData, correction_a: CorrectionA, correction_b: CorrectionB) -> CorrectedData:
    return apply_b(apply_a(raw_data, correction_a), correction_b)

I am not sure you can auto-generate such a function that would work with Sciline?

[jokasimr]

Just 1 correction with 3 options in powder: the 'run normalisation'. In reflectometry, there are more corrections. @jokasimr How many are there?

If we are talking about scalar multiplicative corrections (that is, not polarization correction or background subtraction)
I don't think there are more corrections in reflectometry than for other techniques. For ESTIA there are:

• Correction by virtual slit opening (essentially "footprint correction")
• Correction by monitor (or proton current, or measurement time)
• Correction by supermirror reflectivity (only for reference measurement)

If we are talking about "corrections" to coordinates (not to event weights). There might be more of those.

We also have the pixel masks with an arbitrary number of masks.

There are a number of masks applied, more than the number of corrections.