Multiple branches / halting execution of current pipeline

[hubertlepicki]

I suspect my problems are originating in that I am refactoring existing code that is abusing with clause, and not writing code from scratch, but I stumbled upon some issues when translating the legacy code to the code that uses Opus pipeline.

One feature that would help me is a way to halt the pipeline after I do conditional link. What I currently have is code like this:

with :ok <- check_should_run_something?(),
        {:something, something} <- do_something(),
        :ok <- check_should_run_something1?(),
        {:something1, something1} <- do_something1() do
           :ok
else
  {:error, :expected_error} ->
     run_entirely_different_scenario()
  {:error, :expected_error1} ->
     run_other_different_scenario()
end

I could fairly easily translate that to Opus minus the possibility to halt execution of current pipeline when linking.

For example, the code I would write would be:

link EntirelyDifferentScenario, unless: :should_run_something?
step :run_something
link OtherDifferentScenario, unless: :should_run_something1?
step :run_something1

But the problem here is that link doesn't halt execution of the current pipeline. I want the other pipeline to basically take over, ignoring any further steps, checks and link in current pipeline.

I can simulate the behavior adding a flag that's being passed between steps that says if the pipeline is halted or not, and provide unless: :halted? option to all steps.

But maybe that's an useful improvement that I could add to Opus?

The API I came up with would be something like:

link EntirelyDifferentScenario, unless: :should_run_something?, halt: true
step :run_something
link OtherDifferentScenario, unless: :should_run_something1?, halt: true
step :run_something1

i.e. if conditional link was being executed, and it's got halt: true option we stop execution of the current pipeline after the link returns.

What do you think?

[zorbash]

Thank you for clearly expressing your idea and problem you're trying to solve.

There is a way to implement the flow control you're describing with the skip/2 macro.
The caveat is that the condition is checked twice but imho having an instruction to halt the pipeline as a separate stage makes it more clear and easier to visualise in opus_graph.

link EntirelyDifferentScenario, unless: :should_run_something?
skip :halt_for_different_scenario, unless: :should_run_something?

step :run_something

link OtherDifferentScenario, unless: :should_run_something1?
skip :halt_for_other_different_scenario, unless: :should_run_something1?

step :run_something1

Is this something which can work in your project?

[hubertlepicki]

Oh yes, that's perfect. I only glanced over existence of "skip" and didn't catch that it's halting the whole pipeline. I assumed it's used to skip over one step, but that makes total sense now.

Yes, the need to run check twice is suboptimal but I don't think it's a biggie, and in fact it can improve the code readability if I extract the data fetching needed to perform the check into own step that'll execute before link and skip, and the condition checking function will be pure function without any side effects.

I believe that's perfect, thank you so much for your help @zorbash

[zorbash]

I'm glad that it works for you 🌈

For the record, initiallyskip/2 was added with a slightly different use-case in mind.
It was so that the logic for whether the input is compatible with the pipeline can be contained in the pipeline.

Think of the following example, multiple pipelines are linked but only the applicable ones run without the parent pipeline knowing about their required parameters. The applicability can be checked against a function, a struct or even a static schema (I've used ex_json_schema for that in a project).

defmodule ProcessPayload do
  use Opus.Pipeline
  
  link :validate_input
  link ProcessCommandA
  link ProcessCommandB
  link ProcessCommandC
  link :format_result
end

defmodule ProcessCommandA
  use Opus.Pipeline

  skip :not_applicable, unless: &match?(%{command: "command_a"}, &1)

  step :step_one
  step :step_two
end

[hubertlepicki]

So @zorbash I wasn't actually able to make it work the way you described in post here #28 (comment)

but it works as it was intended to be working.

So I think halting the pipeline if skip is a first stage, but doesn't halt it if the pipeline is already in progress...

EDIT: it actually does work but in my case it neede done more skip with condition reversed to skip original pipeline. All good!