DM-48896 : Add yaml for DRP processing with sysnthetic source injection for diffim

[BrunoSanchez]

Add a pipeline yaml that takes DRP Diffim stage and adds injected synthetic sources, for transient discovery testing.

[leeskelvin]

Are these actually used in this YAML?

[BrunoSanchez]

I am thinking now that maybe they are not being used. I left them so they are clear, and explicit for the user to see.
Do you think it might be better without these?

[leeskelvin]

Clear for what purpose though? If we change the task default from injected_ to post_injection_ for example, the injected_prefix here would still read injected_. Someone might look at this thinking your injected outputs will still be injected_.

[leeskelvin]

If these aren't being used, my recommendation would be to remove them. While we do have examples of code used for demonstration purposes in the stack(!), I've never been a fan of it myself, and I think it ultimately leads to more confusion and more unnecessary maintenance downstream.

[BrunoSanchez]

That's fine then. Since this yaml defines inputs and outputs explicitly and no dataset type templates are actually in use, I will remove these.

[leeskelvin]

The alternative is that you do use these parameters in your config over-rides below.

[leeskelvin]

Can I ask why you've decided to hard-code the VisitInjectTask here, rather than dynamically create it as we do with other injection pipelines? Putting this in would be the second instance in the stack (after pipelines/_ingredients/ApPipeWithFakes.yaml).

[BrunoSanchez]

Because dynamically creating this has a key shortcomings for our case, as it doesn't include the matching tasks, with the associated metrics tasks to verify that we have found our injections

[leeskelvin]

From make_injection_pipeline --help:

  -a, --additional-pipelines FILE [FILE ...]
      Location(s) of additional input pipeline definition YAML file(s).Tasks
      from these additional pipelines will be added to the output injection
      pipeline.

This allows you to define the additional task stubs you need in their own stand-along YAML, and then build a complete SI YAML when drp_pipe gets constructed.

[leeskelvin]

We faced the same issue 5 years ago in our DRP fakes pipelines. Setting them up as static and stand-alone is easier out of the gate, but the problem of having to maintain them as other aspects of the pipeline change quickly becomes onerous. It's almost certainly better for you to set up the AP synthetic pipelines as dynamic now to guard against future maintenance overhead.

[leeskelvin]

Trying to build this pipeline for me fails with:

ImportError: Could not get attribute 'MatchPVIInjectedTask' from 'lsst.ap.pipe.matchSourceInjected' when importing 'lsst.ap.pipe.matchSourceInjected.MatchPVIInjectedTask
' (No module named 'lsst.ap.pipe.matchSourceInjected.MatchPVIInjectedTask'; 'lsst.ap.pipe.matchSourceInjected' is not a package)

Should this be MatchInitialPVIInjectedTask instead?

[leeskelvin]

Perhaps not, as making that switch returns:

AttributeError: lsst.pipe.base.config.MatchInitialPVIInjectedConfigConnections has no attribute injectedPVICat

I'm on main in drp_pipe and ap_pipe - do I need to set up something else?

[BrunoSanchez]

I made a mistake, and I was using a branch class from ap_pipe. I think I can update this to use the current main function, as a replacement matching task is still in flux.

[BrunoSanchez]

It should be fixed on new version.