CI: can filter jobs to run depending on modified files

[mockersf]

Objective

• Allow Bevy CI jobs to be filtered depending on what needs to be tested
• But not in a way that is too dependent on GitHub actions
• But still be compatible with branch protection rules

Solution

• Add a command cargo run -p ci -- what-to-run --trigger pull_request --head main --rust-version stable that list commands to be run
• Add a CI job that runs the above command and outputs the results
• Add a CI matrix job that runs all the jobs from the output
• Add a CI job that has a single status for all matrix jobs, and can be used for branch protection rules

For now, this is just the shape of it to open discussion, no filtering is done

[mockersf]

@NicoZweifel @janhohenheim you were interested in this

[NicoZweifel]

Nice, it's apparent that you put a lot of thought into this already, I think this looks pretty good for a PoC (maybe it could be converted to a draft for now). I do have a few thoughts about it:

A minor maintenance issue in ci.rs is that adding a Command requires editing multiple match statements and locations, which either don't compile or fail silently, but that overhead might be a trade-off that has to be made somewhere.

A job runner is always provisioned and the tools are always built but I think that is an okay trade-off to make.
I noticed that the ci crate is very lean, which is nice. Is it faster to fully re-compile instead of using a cache in the prepare stage as well or is there another reason?

Maybe you already thought about this too, but the WhatToRunCommand will become a huge function with lots of responsibilities, so personally I would improve the WhatToRunCommand to work with workflows/strategies, e.g.:

pub struct Context {
    pub trigger: Trigger,
    pub changed_files: Vec<String>,
}

pub trait Workflow {
    fn suite(&self, context: &Context) -> Vec<Commands>;
}

This also simplifies some of the existing code for running everything since you can just return a Full strategy that runs everything (reduces the maintenance issue touched earlier), e.g. here:

    fn prepare<'a>(&self, sh: &'a xshell::Shell) -> Vec<PreparedCommand<'a>> {
        let args = self.into();
        match &self.command {
            Some(command) => command.prepare(sh, args),
            None => {
                // Note that we are running the subcommands directly rather than using any aliases
                let mut cmds = vec![];
                cmds.append(&mut commands::FormatCommand::default().prepare(sh, args));
                cmds.append(&mut commands::ClippyCommand::default().prepare(sh, args));
                cmds.append(&mut commands::TestCommand::default().prepare(sh, args));
                cmds.append(&mut commands::TestCheckCommand::default().prepare(sh, args));
                cmds.append(&mut commands::IntegrationTestCommand::default().prepare(sh, args));
                cmds.append(
                    &mut commands::IntegrationTestCheckCommand::default().prepare(sh, args),
                );
                cmds.append(
                    &mut commands::IntegrationTestCleanCommand::default().prepare(sh, args),
                );
                cmds.append(&mut commands::DocCheckCommand::default().prepare(sh, args));
                cmds.append(&mut commands::DocTestCommand::default().prepare(sh, args));
                cmds.append(&mut commands::CompileCheckCommand::default().prepare(sh, args));
                cmds.append(&mut commands::CompileFailCommand::default().prepare(sh, args));
                cmds.append(&mut commands::BenchCheckCommand::default().prepare(sh, args));
                cmds.append(&mut commands::ExampleCheckCommand::default().prepare(sh, args));

                cmds
            }
        }
    }

[NicoZweifel]

This seems a bit odd, maybe it's just temporary?

[NicoZweifel]

Maybe we could differentiate between shell and rust commands?

[NicoZweifel]

This is a bit fragile. I assume this is so that there is no dependency on github ($GITHUB_OUTPUT) here?

[NicoZweifel]

I think I would prefer to pass an env variable as argument that the script should set, following the same logic/pattern (prepare/run) that the rest of the commands use.

The command could prepare: set $ENV_VARIABLE = [workflow] , which then resolves to set $GITHUB_OUTPUT = [cargo test, etc...].

I would also recommend something like the Workflow trait I mentioned earlier to help with organizing the workflows. Happy to help with any of this if you'd like.