feature: fast initializer for conda environments #2226
Conversation
| 'DISABLE_TRACING=True python -m metaflow.plugins.pypi.bootstrap "%s" %s "%s" linux-64' | ||
| % (self.flow.name, id_, self.datastore_type), | ||
| 'DISABLE_TRACING=True python -m metaflow.plugins.pypi.%s "%s" %s "%s" linux-64' | ||
| % (bootstrap_module_name, self.flow.name, id_, self.datastore_type), |
There was a problem hiding this comment.
nit - can we maintain the singular - metaflow.plugins.pypi.bootstrap module and change the bootstrapping logic within it? easier to benchmark changes that way.
| @@ -88,6 +58,9 @@ def install_fast_initializer(architecture): | |||
| return fast_initializer_path | |||
|
|
|||
| # TODO: take architecture into account | |||
| url = CONDA_FAST_INIT_BIN_URL | |||
There was a problem hiding this comment.
With this change, does the TODO above still make sense? Or are we going to handle the architecture at a different level, by setting CONDA_FAST_INIT_BIN_URL to an architecture-specific url?
There was a problem hiding this comment.
Fair point. The client needs to have knowledge of the target architecture either way as it is the one doing the solving for packages. The client also provides the architecture input for the bootstrap script, so we might as well keep the URL defining logic completely client-side.
| @@ -433,6 +433,12 @@ | |||
| # should result in an appreciable speedup in flow environment initialization. | |||
| CONDA_DEPENDENCY_RESOLVER = from_conf("CONDA_DEPENDENCY_RESOLVER", "conda") | |||
|
|
|||
| # Conda Fast init binary url | |||
| CONDA_FAST_INIT_BIN_URL = from_conf("CONDA_FAST_INIT_BINARY_URL", None) | |||
There was a problem hiding this comment.
can you remove this config and introduce the path to actual binary in the install_fast_initializer method instead? also, if you can note some latency improvements in the PR, that will be great!
adds a feature flag to switch between regular micromamba and a custom binary for bootstrapping of environments when executing remotely.
also adds deferring of micromamba install until the binary location is actually requested.