Implemented multi-fidelity bayesian search for the auto-tuner #1019
Conversation
|
Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
meta-cla
bot
added
the
CLA Signed
|
kicked off the CI, but you'll probably need to update the requirements file |
|
@oulgen do you mind taking a look whenever time permits? |
|
Hey @FranciscoThiesen thank you for implementing a new autotuning algorithm. Could you share some results? Perhaps you can compare to PatternSearch in terms of
also please make sure the tests and the lint are passing |
|
also please update your new unit test file to be the same style as rest of the test using a class |
|
@oulgen do you have any available GPUs that could be used for convergence analysis + best perf comparison? I definitely agree that having this is a must to assess how good is MFBO versus the current hill-climbing approach. I think this will really shine in terms of reducing the total time/resources that the auto-tuner takes, while still finding good solutions. I see that the CI runs a few of the tests on GPUs and unfortunately I don't have a personal one that I can use for my OS contributions. |
I can give it a try, but no promises on when. |
FranciscoThiesen
changed the title
|
@oulgen got GPUs here for running the convergence analysis. Will run it whenever time permits and then share the results. |
There was a problem hiding this comment.
This looks algorithm specific, let's move all the related files to a subfolder.
|
|
||
| from typing import TYPE_CHECKING | ||
|
|
||
| import numpy as np |
There was a problem hiding this comment.
Let's use torch to avoid adding a numpy dependency.
| Args: | ||
| config: The configuration to benchmark. | ||
| fn: A precompiled version of config. | ||
| fidelity: Number of repetitions for benchmarking (default: 50). |
There was a problem hiding this comment.
Let's rename to the repeat or samples.
| ) | ||
| elif enc_type == "enum": | ||
| # One-hot encoding | ||
| if hasattr(spec, "choices"): |
| from .config_generation import FlatConfig | ||
|
|
||
|
|
||
| class ConfigEncoder: |
There was a problem hiding this comment.
Could this share more code with ConfigGeneration?
| if category in { | ||
| Category.BLOCK_SIZE, | ||
| Category.NUM_WARPS, | ||
| }: |
There was a problem hiding this comment.
I think it would be better if you used the ConfigFragement directly (add a method for this type of encoding) rather than switching based on category.
| except (ValueError, IndexError): | ||
| # Default to first choice if value not found | ||
| encoded[enc_start] = 1.0 | ||
|
|
There was a problem hiding this comment.
| else: | |
| raise and error |
| def get_bounds(self) -> list[tuple[float, float]]: | ||
| """ | ||
| Get bounds for each encoded dimension. | ||
| Returns: | ||
| List of (min, max) tuples for each dimension. | ||
| """ | ||
| bounds: list[tuple[float, float]] = [] | ||
|
|
||
| for flat_idx, spec in enumerate(self.flat_spec): | ||
| category = spec.category() | ||
| enc_start, enc_end, enc_type = self.encoding_map[flat_idx] | ||
|
|
||
| if enc_type == "numerical": | ||
| if category in {Category.BLOCK_SIZE, Category.NUM_WARPS}: | ||
| # Power-of-2: log2 bounds | ||
| min_val = math.log2(float(spec.low)) # type: ignore[attr-defined] | ||
| max_val = math.log2(float(spec.high)) # type: ignore[attr-defined] | ||
| bounds.append((min_val, max_val)) | ||
| else: | ||
| # Other numerical bounds | ||
| bounds.append( | ||
| (float(spec.low), float(spec.high)) # type: ignore[attr-defined] | ||
| ) | ||
| elif enc_type == "enum": | ||
| # One-hot: each dimension is 0 or 1 | ||
| num_choices = enc_end - enc_start | ||
| bounds.extend([(0.0, 1.0)] * num_choices) |
There was a problem hiding this comment.
The ConfigSpecFragment already has methods for this.
| import numpy as np | ||
| from sklearn.gaussian_process import GaussianProcessRegressor | ||
| from sklearn.gaussian_process.kernels import ConstantKernel | ||
| from sklearn.gaussian_process.kernels import Matern |
There was a problem hiding this comment.
These should be optional deps.
3 participants
No description provided.