Fix plugin secrets in config

[bentsherman]

Fix #5494 #5943

This PR fixes support for plugin secrets (i.e. AWS secrets) in the config by loading the config before resolving plugins. If the config attempts to use secrets, it will be reloaded with the complete set of secrets providers.

This is possible because plugin config must be static, unlike other config which can be dynamic and reference secrets. Therefore, we can simply load the config to extract the list of plugins (and other config settings like tower.enabled and process.executor which can trigger additional core plugins.

To minimize the impact of loading the config twice, we only reload the config if we detect that secrets were used by the config on the first load. This way, the double config load should only affect users who want to use secrets in the config.

In summary:

1. Load config with "shim" secrets provider which only tracks whether secrets are used by config
2. Load plugins
3. Load secrets provider
4. Reload config if secrets were used in config

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	0ba2507
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/6894e99e3a39640008aec646
😎 Deploy Preview	https://deploy-preview-6249--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[bentsherman]

Here is how I've been testing:

1. Build with make pack
2. Upload dist binary to s3
3. Launch this pipeline in platform: https://github.com/bentsherman/nf-tests/tree/plugin-secrets
4. Download dist binary from s3 and replace default nextflow in pre-run script

However, it's not loading the xpack-amzn plugin:

Jul-03 19:06:49.448 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[[email protected], [email protected], [email protected], [email protected]]

It's at least loading nf-amazon since the workDir is s3, but I can't remember how to trigger the xpack plugin

[bentsherman]

I wonder if the issue is that I'm using the dist release instead of the standard release

[pditommaso]

This looks smart. Moving to draft until the problem with the plugins is addressed. Also unit test and integration test should be added (it may be convenience setup a test GH repo for this)

[bentsherman]

It turns out the issue was that I was using the single-VM CE, which doesn't support AWS secrets. It works with AWS Batch.

Regarding testing, I don't think we'll need a git repo unless you want to do an e2e test with platform. I'll add a local e2e test that just tests the detection of config secrets -> config reloading.

[bentsherman]

One minor issue with this double-loading approach is that you can't use secrets in a config include (#5312). On the first load, the secret will be null and the include will likely fail, which means the run will fail

However, you should be able to work around this using a dynamic include:

includeConfig secrets.GITHUB_TOKEN
    ? "https://raw.githubusercontent.com/markpanganiban/nf-test/master/ubuntu.config?token=${secrets.GITHUB_TOKEN}"
    : '/dev/null'

On the first load, it will access the secret, triggering a reload, but won't attempt to include the secured config. On the second load it will include the config.

I will add this to the docs on config secrets

[bentsherman]

The e2e test that I added is failing in GitHub, but not sure why because it works for me locally

[christopher-hakkaart]

Some very minor suggestions. Feel free to ignore if the meaning changes.

[christopher-hakkaart]

Suggested change

	Secrets can be used with the local executor and the grid executors (e.g., Slurm or Grid Engine). Secrets can be used with the AWS Batch executor when launched from [Seqera Platform](https://seqera.io/blog/pipeline-secrets-secure-handling-of-sensitive-information-in-tower/).
	Secrets can be used with the local executor and grid executors (e.g., Slurm or Grid Engine). Secrets can be used with the AWS Batch executor when launched from [Seqera Platform](https://seqera.io/blog/pipeline-secrets-secure-handling-of-sensitive-information-in-tower/).

[christopher-hakkaart]

Suggested change

	Secrets should not be assigned to pipeline parameters, as they can be easily leaked by the pipeline.
	Secrets should not be assigned to pipeline parameters, as they can be leaked by the pipeline.

Highly optional suggestion