Update nf-azure: Add Azure Compute Gallery image and verification options by LennyBEL · Pull Request #7338 · nextflow-io/nextflow

LennyBEL · 2026-07-15T13:58:45Z

Split from #7321

Overview

Azure Compute Gallery
Last year we had a major outage when an update to the Microsoft Marketplace Ubuntu 22.04 HPC image flipped its verification state to unverified. Because Nextflow only targets verified images, new pools could no longer be created. Our pipeline tooling did not support Ubuntu 24.04 yet, so we had to scramble to restore service.

In order to avoid such an event again, Microsoft suggested that we switch to Azure Compute Gallery images. Within Azure Compute Gallery, we have full control over the images. However, Nextflow did not support the usage of them yet.

A new option virtualMachineImageId has been introduced to support usage of the Azure Compute Gallery.

Image verification state
For non-production and testing purposes we occasionally need to use unverified Marketplace images, but Nextflow always forced verified.

This PR adds azure.batch.pools.<name>.verification with values verified (default), unverified, or any - where any ignores the verification state entirely. This setting is ignored when virtualMachineImageId is set.

Note: Windows support was left out compared to the original PR. OsType is no longer configurable and remains fixed on Linux.

Expected Impact

None, as the default behavior does not change at all.

We have been running these changes as a custom nf-azure fork (based on the upstream plugin) in production for a while, so we are confident they behave as intended.

Tests

Unit tests added/updated in nf-azure:

AzPoolOptsTest
- Compute Gallery image configuration (virtualMachineImageId + sku).
- Parsing/validation of verification (verified/unverified/any, case-insensitive), with an error on invalid values.
- Cache-key (funnel) sensitivity: the pool hash changes when the image config changes.
AzBatchServiceTest
- Updated the auto-pool id hash to reflect the new pool options now included in the cache key.

Docs updated: docs/azure.mdx and docs/reference/config.mdx.

Add support for provisioning Azure Batch pool nodes from a custom VM image in an Azure Compute Gallery via the new 'azure.batch.pools.<name>.virtualMachineImageId' option, and add the 'azure.batch.pools.<name>.verification' option to choose the image verification type ('verified', 'unverified' or 'any'). Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

netlify · 2026-07-15T13:58:50Z

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	`caaecfb`
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a68c08d13f5090008b61ff1

Signed-off-by: Lenny <vdwlenny@outlook.be>

adamrtalbot

This would also help when Azure deprecates a Batch image without a replacement image lined up. It would improve the supply chain security of our Azure users. So worth it overall.

adamrtalbot

(Claude review here, but raised some good points)

Focused, sensible change — the verification == null = "any" handling in getImage, the gallery/marketplace branch in poolVmConfig, and the verified default all look correct. A few points below, most-important first.

Cache-key note (whole-PR): the pool funnel now hashes verification, which was a field but was previously not hashed. On upgrade every existing auto-pool id changes (hence the test hash change 42f3635f…→7483c5b1…), so existing pools won't be reused and will idle out. Worth calling out in the description — "Expected Impact: None" isn't quite accurate for users relying on pool reuse across the upgrade.

christopher-hakkaart

I've added some editorial suggestions. I'll defer to Adam for the exact location of the more advanced content.

LennyBEL · 2026-07-24T13:06:52Z

@adamrtalbot Would you like me to incorporate your suggested changes, or leave it as-is? You approved the PR so I'm a bit confused ;)

adamrtalbot · 2026-07-24T13:16:51Z

@adamrtalbot Would you like me to incorporate your suggested changes, or leave it as-is? You approved the PR so I'm a bit confused ;)

Oh, sorry.

I think this works fine which is why I approved, but I think verified should be included in unverified. This would simplify the code as well with a simple boolean (unverified = true). The last thing I want is to be rejected from a verified image because I have set unverified = true

I'm not a codeowner (not a Nextflow developer), so we need someone with higher powers to approve.

Add unverified = true, remove the enum-like string.
unverified = true also allows verified.
make @christopher-hakkaart's changes on the docs.
Wait for a codeowner review

…eReference to comply with SDK changes in beta.6

…s' boolean. Move to advanced features in documentation

Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

…s compatibility with existing pools pre-upgrade Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

…geId is set Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io> Signed-off-by: Lenny <vdwlenny@outlook.be>

Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

…ow into nf-azure-compute-gallery Merge

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Lenny <vdwlenny@outlook.be>

adamrtalbot

Nice, this looks really good.

Just minor docs and comments to add.

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Lenny <vdwlenny@outlook.be>

…ts.groovy Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Lenny <vdwlenny@outlook.be>

…ow into nf-azure-compute-gallery Merge

LennyBEL requested a review from a team as a code owner July 15, 2026 13:58

LennyBEL mentioned this pull request Jul 15, 2026

Update nf-azure: node communication mode setting, allow Azure Compute Gallery and allow configuration of verification state #7321

Closed

LennyBEL and others added 2 commits July 15, 2026 16:04

Merge branch 'master' into nf-azure-compute-gallery

c284277

Signed-off-by: Lenny <vdwlenny@outlook.be>

Merge branch 'master' into nf-azure-compute-gallery

890ca8e

adamrtalbot approved these changes Jul 15, 2026

View reviewed changes

adamrtalbot reviewed Jul 15, 2026

View reviewed changes

christopher-hakkaart reviewed Jul 15, 2026

View reviewed changes

Comment thread docs/reference/config.mdx Outdated

Comment thread docs/reference/config.mdx Outdated

Comment thread docs/azure.mdx

Comment thread docs/azure.mdx Outdated

Comment thread docs/azure.mdx Outdated

bentsherman added the executor/azure-batch label Jul 18, 2026

LennyBEL and others added 11 commits July 27, 2026 10:08

Remove stale MetadataItem and replace ImageReference with BatchVmImag…

8fd38fd

…eReference to comply with SDK changes in beta.6

Remove 'verification' property and replace with 'allowUnverifiedImage…

5a8fbe2

…s' boolean. Move to advanced features in documentation

Order opts more logically

93ec481

Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

Place guard around hashing 'allowUnverifiedImages' to ensure backward…

ad21c26

…s compatibility with existing pools pre-upgrade Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

Add check on sku and virtualMachineImageId requirement properly

9ff93ba

Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

Ignore allowunverifiedimages property in hashing if virtualMachineIma…

ca37100

…geId is set Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

Update docs/reference/config.mdx

fb30feb

Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io> Signed-off-by: Lenny <vdwlenny@outlook.be>

Update docs/azure.mdx

48f230c

Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io> Signed-off-by: Lenny <vdwlenny@outlook.be>

Update docs/azure.mdx

7cd6de0

Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io> Signed-off-by: Lenny <vdwlenny@outlook.be>

Tidy up documentation

8a3471d

Signed-off-by: Lenny Van de Winkel <vdwlenny@outlook.be>

Merge branch 'nf-azure-compute-gallery' of github.com:LennyBEL/nextfl…

a81216b

…ow into nf-azure-compute-gallery Merge

adamrtalbot reviewed Jul 27, 2026

View reviewed changes

Comment thread docs/azure.mdx Outdated

Update docs/azure.mdx

0896c7c

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Lenny <vdwlenny@outlook.be>

adamrtalbot reviewed Jul 27, 2026

View reviewed changes

Comment thread docs/azure.mdx

Comment thread docs/azure.mdx Outdated

Comment thread docs/reference/config.mdx

Comment thread docs/reference/config.mdx

Comment thread plugins/nf-azure/src/main/nextflow/cloud/azure/config/AzPoolOpts.groovy Outdated

LennyBEL and others added 2 commits July 27, 2026 14:18

Update docs/azure.mdx

f5e6f3c

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Lenny <vdwlenny@outlook.be>

Update plugins/nf-azure/src/main/nextflow/cloud/azure/config/AzPoolOp…

06119f9

…ts.groovy Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Lenny <vdwlenny@outlook.be>

adamrtalbot approved these changes Jul 27, 2026

View reviewed changes

Comment thread docs/reference/config.mdx

Add AddedInVersion tags for relevant configs

444039a

LennyBEL and others added 4 commits July 27, 2026 16:29

Merge branch 'nf-azure-compute-gallery' of github.com:LennyBEL/nextfl…

b9db562

…ow into nf-azure-compute-gallery Merge

Merge branch 'master' into nf-azure-compute-gallery

eb5d971

Merge branch 'master' into nf-azure-compute-gallery

8b8a3f4

Merge branch 'master' into nf-azure-compute-gallery

caaecfb

Uh oh!

Conversation

LennyBEL commented Jul 15, 2026

Overview

Expected Impact

Tests

Uh oh!

netlify Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for nextflow-docs canceled.

Uh oh!

adamrtalbot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamrtalbot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christopher-hakkaart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LennyBEL commented Jul 24, 2026

Uh oh!

adamrtalbot commented Jul 24, 2026

Uh oh!

Uh oh!

adamrtalbot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

netlify Bot commented Jul 15, 2026 •

edited

Loading

adamrtalbot left a comment •

edited

Loading