Can't merge promotion branch if stable branch was updated

[achave11-ucsc]

Slack conv.

Initially, promotion PR #6828, was opened against the prod branch as commonly done when it's time to do a promotion. This PR had already built its status checks and was ready to be merged, as part of the promotion, but an additional change was introduced to the prod branch, as part of a hotfix PR (#6825). In other words, the branch prod moved ahead, after the promotion feature branch had been created and its status checks were OK. We suspect this might have caused the missing test status check, since the promotion branch had not been updated to build prod's most recent base commit (the hotfix changes), when the merge commit in prod was pushed to GitHub.

❯ git push github prod
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 10 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 312 bytes | 312.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
remote: error: GH006: Protected branch update failed for refs/heads/prod.
remote: 
remote: - Required status check "test" is expected.
To https://github.com/DataBiosphere/azul.git
 ! [remote rejected]     prod -> prod (protected branch hook declined)
error: failed to push some refs to 'https://github.com/DataBiosphere/azul.git'

[hannes-ucsc]

is what I think causes it. But we should wait until this reoccurs so that we can disable the requirement and see if that fixes it.

[hannes-ucsc]

We ran into a similar situation with the anvilprod promotion today. The backport of a hotfix on anvilprod wasn't part of that promotion so the anvilprod branch was ahead by one commit (and behind by several commits, as is typical for a promotion). The PR showed the "PR branch isn't up-to-date with target branch" message at the bottom" and the push of the merge commit was declined. Disabling the branch requirement that I highlighted above made the push succeed. I re-enabled it afterwards. In lieu of disabling the requirement, two other remedies exist: either the backport would have to have been included in the promotion—not an option this time because there were other, immature commits before the backport merge commit, so including the backport would have forced us to include the destabilizing commits—or the promotion PR author (operator) would have needed to include a merge commit from anvilprod into the promotion branch (a so called back merge).

We should leave the branch requirement enabled and add a checklist item to the PR templates for all types of PRs that are not rebased. The CL item would instruct the PR author to perform a backmerge from the target branch into the PR branch if needed. The need can be determined by looking for the "out-of-date" warning at the bottom of the PR.